Azure Monitor Logging and Configuration – Complete Guide for DP-203
Why Azure Monitor Logging and Configuration Matters
In any enterprise data engineering environment, having comprehensive visibility into the health, performance, and security of your data pipelines, storage accounts, and compute resources is absolutely critical. Azure Monitor is Microsoft's unified monitoring platform that provides a centralized way to collect, analyze, and act on telemetry data from your Azure resources. For the DP-203 (Data Engineering on Microsoft Azure) exam, understanding how to configure and leverage Azure Monitor logging is essential because it directly ties into the Secure, Monitor, and Optimize Data Storage and Data Processing domain.
Without proper monitoring and logging, data engineers would be unable to detect pipeline failures, identify performance bottlenecks, track unauthorized access attempts, or meet compliance and auditing requirements. Azure Monitor ensures operational excellence and data reliability across your entire data platform.
What is Azure Monitor?
Azure Monitor is a comprehensive monitoring service in Azure that collects, analyzes, and acts on telemetry from cloud and on-premises environments. It helps you maximize the availability and performance of your applications and services by delivering insights through:
• Metrics – Numerical time-series data that describes some aspect of a system at a particular point in time.
• Logs – Structured and semi-structured records of events that occurred within the system, stored in Azure Monitor Logs (Log Analytics workspaces).
• Alerts – Proactive notifications triggered when specific conditions are met in metrics or logs.
• Dashboards and Workbooks – Visual representations of monitoring data for analysis and reporting.
Key Components of Azure Monitor
1. Log Analytics Workspace
This is the central repository for log data in Azure Monitor. All logs from various sources (Azure resources, applications, operating systems) are stored here. You query data using Kusto Query Language (KQL). A single workspace can collect data from multiple subscriptions and resources.
2. Diagnostic Settings
Diagnostic settings define where resource logs and metrics are sent. You can route them to:
• A Log Analytics workspace (for querying and analysis)
• An Azure Storage account (for long-term archival)
• Azure Event Hubs (for streaming to external systems like SIEM tools)
3. Azure Monitor Metrics
These are lightweight, near-real-time numerical values. They are stored in a time-series database and are ideal for alerting and fast detection of issues. Examples include CPU utilization, DTU consumption, and pipeline run durations.
4. Azure Monitor Logs
Logs contain detailed, rich data about operations and events. They are stored in Log Analytics workspaces and queried with KQL. Examples include activity logs, resource logs, and custom application logs.
5. Activity Log
A platform log that records subscription-level events such as resource creation, modification, and deletion. It provides insight into who did what and when at the management plane level.
6. Alerts and Action Groups
Alerts are rules that trigger when conditions are met. Action groups define what happens when an alert fires (e.g., send email, call a webhook, trigger an Azure Function, create an ITSM ticket).
How Azure Monitor Logging Works
The flow of monitoring data in Azure follows these steps:
Step 1: Data Collection
Data is collected from multiple sources:
• Platform metrics and logs – Automatically generated by Azure resources (e.g., Azure Data Factory pipeline runs, Synapse SQL pool query metrics, Storage account transactions).
• Resource logs – Require diagnostic settings to be enabled. These contain detailed operational data specific to each resource type.
• Activity logs – Automatically collected for all subscription-level operations.
• Custom logs – Ingested via the Data Collection API or Log Analytics agent.
Step 2: Data Storage
• Metrics are stored in the Azure Monitor metrics database (retained for 93 days by default).
• Logs are stored in a Log Analytics workspace (default retention is 30 days, configurable up to 730 days, or archived for longer).
• Data can also be exported to Storage accounts or Event Hubs.
Step 3: Analysis and Querying
• Use KQL (Kusto Query Language) in Log Analytics to query log data.
• Use Metrics Explorer for visualizing metric data.
• Use Azure Workbooks for creating rich, interactive reports.
Step 4: Alerting and Response
• Configure alert rules based on metrics (metric alerts) or log queries (log alerts).
• Alert rules evaluate conditions at defined frequencies.
• When conditions are met, action groups are triggered to notify or remediate.
Step 5: Visualization and Integration
• Azure Dashboards, Workbooks, and Power BI can visualize monitoring data.
• Integration with Azure Logic Apps, Azure Functions, and third-party tools enables automated workflows.
Configuring Azure Monitor for Key DP-203 Services
Azure Data Factory (ADF)
• Enable diagnostic settings to send pipeline run logs, trigger run logs, and activity run logs to a Log Analytics workspace.
• Use the ADFPipelineRun, ADFTriggerRun, and ADFActivityRun tables in KQL queries.
• Monitor pipeline durations, failure rates, and data movement throughput.
• ADF also has a built-in monitoring hub, but for historical analysis and alerting, Log Analytics is essential.
Azure Synapse Analytics
• Enable diagnostic settings on Synapse workspaces to capture SQL request logs, built-in pool activities, and Spark application events.
• Key log categories include SynapseBuiltinSqlPoolRequestsEnded, SynapseSqlPoolExecRequests, and SynapseBigDataPoolApplicationsEnded.
• Monitor query performance, resource utilization, and DWU/cDWU consumption.
Azure Data Lake Storage Gen2
• Enable diagnostic settings to log read, write, and delete operations.
• Use StorageBlobLogs table in Log Analytics to audit data access patterns.
• Monitor latency, throughput, and availability metrics.
Azure Stream Analytics
• Enable diagnostic logs to capture input/output events, runtime errors, and watermark delays.
• Monitor SU (Streaming Unit) utilization and backlogged events.
Azure Event Hubs / IoT Hub
• Enable diagnostic logs for operational and archive logs.
• Monitor throughput, throttled requests, and consumer lag.
Key KQL Query Examples for DP-203
Querying failed ADF pipeline runs:
ADFPipelineRun | where Status == "Failed" | project PipelineName, Start, End, Status, ErrorMessage | order by Start desc
Querying Synapse SQL pool long-running queries:
SynapseSqlPoolExecRequests | where DurationMs > 60000 | project QueryText, DurationMs, StartTime, EndTime | order by DurationMs desc
Querying storage account access patterns:
StorageBlobLogs | where OperationName == "GetBlob" | summarize count() by CallerIpAddress | order by count_ desc
Diagnostic Settings Configuration Best Practices
• Always enable diagnostic settings for production resources – they are NOT enabled by default for resource logs.
• Use Azure Policy to enforce diagnostic settings across all resources in a subscription or management group.
• Send logs to a centralized Log Analytics workspace for cross-resource querying and correlation.
• For compliance and long-term retention, also send logs to a Storage account with appropriate retention policies.
• For real-time streaming to third-party SIEM solutions, use Event Hubs as a destination.
• Configure multiple destinations simultaneously when needed (e.g., Log Analytics + Storage account).
Log Retention and Archival
• Default retention in Log Analytics is 30 days (free tier) and can be extended up to 730 days (2 years).
• For longer retention, configure data export rules to send data to Azure Storage or use the Archive feature in Log Analytics.
• Archived data can be searched using Search Jobs or Restore operations.
• Storage account retention can be managed with lifecycle management policies.
Role-Based Access Control (RBAC) for Monitoring
• Monitoring Reader – Can read all monitoring data (metrics, logs, alerts).
• Monitoring Contributor – Can read monitoring data and modify monitoring settings.
• Log Analytics Reader – Can read log data from a workspace.
• Log Analytics Contributor – Can read and manage Log Analytics resources.
• Use resource-context or workspace-context access control modes on Log Analytics workspaces to control data access granularity.
Alerts Configuration
• Metric Alerts – Evaluate metric values at regular intervals. Best for resource utilization thresholds (e.g., CPU > 80%, DTU > 90%).
• Log Alerts – Run KQL queries at defined intervals and alert based on results. Best for complex conditions (e.g., more than 5 failed pipeline runs in the last hour).
• Activity Log Alerts – Trigger on specific activity log events (e.g., when a resource is deleted).
• Smart Detection – Uses machine learning to automatically detect anomalies (Application Insights).
• Alerts can have severity levels from 0 (Critical) to 4 (Verbose).
Azure Monitor vs. Other Monitoring Tools
• Azure Monitor – Platform-level monitoring for all Azure resources.
• Application Insights – Part of Azure Monitor, focused on application performance monitoring (APM).
• Azure Advisor – Provides recommendations for cost, performance, security, and reliability (not real-time monitoring).
• Microsoft Defender for Cloud – Security monitoring and threat protection.
• Azure Service Health – Monitors Azure platform health and planned maintenance.
========================================
Exam Tips: Answering Questions on Azure Monitor Logging and Configuration
========================================
1. Remember that diagnostic settings must be explicitly configured.
Resource logs are NOT collected automatically. If a question asks how to capture detailed operational logs for a resource like ADF, Synapse, or ADLS, the answer involves configuring diagnostic settings. Activity logs, however, are collected automatically.
2. Know the three destinations for diagnostic settings.
Questions often test whether you know that diagnostic data can be sent to: (a) Log Analytics workspace, (b) Azure Storage account, or (c) Event Hubs. If the scenario requires real-time streaming to a third-party tool, choose Event Hubs. For long-term archival, choose Storage. For querying and alerting, choose Log Analytics.
3. Understand KQL basics.
You may see questions that include KQL snippets or ask you to choose the correct KQL query. Know basic operators like where, summarize, project, order by, extend, join, and render. You do not need to be an expert, but familiarity is important.
4. Differentiate between metrics and logs.
Metrics are numerical, lightweight, and near-real-time – ideal for quick alerting. Logs are detailed, rich records – ideal for deep analysis and troubleshooting. If a question asks about near-real-time alerting on resource utilization, think metrics. If it asks about detailed audit trails or complex querying, think logs.
5. Know the default retention periods.
Log Analytics default is 30 days (configurable up to 730 days). Metrics are retained for 93 days. If a question mentions needing data beyond these periods, look for answers involving Storage account archival or the Archive feature.
6. Use Azure Policy for governance at scale.
If a question asks how to ensure all resources in a subscription have diagnostic settings enabled, the answer is Azure Policy with a DeployIfNotExists policy effect, not manual configuration.
7. Know which table names correspond to which services.
For ADF: ADFPipelineRun, ADFActivityRun, ADFTriggerRun. For Synapse: SynapseSqlPoolExecRequests, SynapseBigDataPoolApplicationsEnded. For Storage: StorageBlobLogs. Recognizing these table names in KQL queries can help you quickly identify the correct answer.
8. Understand alert types and when to use each.
Metric alerts for threshold-based numeric conditions. Log alerts for complex conditions requiring KQL queries. Activity log alerts for management plane events. If a scenario describes alerting when a specific resource configuration changes, think Activity Log alert.
9. Remember the role of Action Groups.
Action groups are reusable across multiple alert rules. They define notification and automation actions (email, SMS, webhook, Azure Function, Logic App, ITSM). Questions about automated remediation will often involve action groups triggering runbooks or Logic Apps.
10. Distinguish Azure Monitor from Azure Advisor.
Azure Monitor provides real-time and historical monitoring data. Azure Advisor provides recommendations for improvement. If a question asks about getting performance recommendations, the answer is Advisor. If it asks about monitoring and alerting, the answer is Azure Monitor.
11. Watch for questions about cross-resource queries.
Log Analytics supports cross-workspace and cross-resource queries using the workspace() and resource() functions. If a question describes querying data from multiple workspaces, this feature is key.
12. Pay attention to cost optimization.
Questions may test your knowledge of reducing monitoring costs. Key strategies include: adjusting log retention periods, using resource-specific log tables instead of Azure Diagnostics mode, filtering which log categories to collect, and using commitment tier pricing for Log Analytics workspaces with high data volumes.
13. Know the difference between Azure Diagnostics mode and Resource-specific mode.
Azure Diagnostics mode sends all logs to a single AzureDiagnostics table. Resource-specific mode sends logs to dedicated tables per resource type. Resource-specific mode is recommended for better query performance and schema clarity. Many newer services default to resource-specific mode.
14. Remember that Application Insights is part of Azure Monitor.
If questions reference monitoring application-level telemetry (e.g., custom events, dependencies, exceptions in Spark applications or Azure Functions used in pipelines), Application Insights is the correct component.
15. Practice reading scenarios carefully.
Many exam questions provide a scenario with specific requirements (e.g., minimize cost, ensure compliance, enable real-time alerting). The correct answer depends on matching the requirement to the right Azure Monitor feature. Always identify the key requirement before selecting your answer.
