Logging and Monitoring with CloudWatch

Logging and Monitoring with CloudWatch – Complete Guide for AWS Data Engineer Associate

Why Logging and Monitoring with CloudWatch Is Important

In any data engineering environment, visibility into the health, performance, and behavior of your systems is non-negotiable. Amazon CloudWatch serves as the central nervous system for observability on AWS. Without proper logging and monitoring, you risk undetected failures in data pipelines, silent data loss, performance degradation, cost overruns, and security blind spots. For the AWS Data Engineer Associate exam, CloudWatch is a foundational service that intersects with nearly every data-related AWS offering — from Glue jobs to Kinesis streams to Redshift clusters.

What Is Amazon CloudWatch?

Amazon CloudWatch is a monitoring and observability service that collects and tracks metrics, collects and monitors log files, sets alarms, and automatically reacts to changes in your AWS resources. It provides a unified view of operational health across AWS services and on-premises resources.

CloudWatch is composed of several key components:

1. CloudWatch Metrics
Metrics are time-ordered data points published by AWS services or custom applications. Every AWS service automatically publishes metrics to CloudWatch. Examples include:
- AWS Glue: glue.driver.aggregate.bytesRead, glue.driver.aggregate.recordsRead, glue.driver.jvm.heap.usage
- Amazon Kinesis Data Streams: IncomingRecords, GetRecords.IteratorAgeMilliseconds, ReadProvisionedThroughputExceeded
- Amazon Redshift: CPUUtilization, DatabaseConnections, PercentageDiskSpaceUsed
- Amazon S3: BucketSizeBytes, NumberOfObjects, AllRequests
- AWS Lambda: Invocations, Duration, Errors, Throttles, ConcurrentExecutions

Metrics have a namespace, metric name, and up to 30 dimensions (key-value pairs that further identify the metric). The default resolution is 1-minute intervals, but detailed monitoring (available for some services) provides data at 1-second or 5-second intervals. Custom metrics can be published using the PutMetricData API with standard (60-second) or high-resolution (1-second) granularity.

2. CloudWatch Logs
CloudWatch Logs enables you to centralize logs from all your systems, applications, and AWS services. Key concepts include:

- Log Groups: A collection of log streams that share the same retention, monitoring, and access control settings. Each AWS service typically creates its own log group (e.g., /aws/glue/jobs/output, /aws/lambda/function-name).
- Log Streams: A sequence of log events from the same source (e.g., a specific Glue job run or Lambda invocation).
- Log Events: Individual records containing a timestamp and raw event message.
- Retention Settings: You configure how long logs are retained (1 day to 10 years, or indefinitely). This is critical for cost management.
- Metric Filters: Allow you to extract metric observations from ingested log data and create CloudWatch metrics from patterns found in logs. For example, you can create a filter to count ERROR occurrences in Glue job logs.
- Log Insights: A fully integrated, interactive, and pay-per-query log analytics service. It uses a purpose-built query language to search and analyze log data. This is powerful for ad-hoc troubleshooting of data pipelines.
- Subscription Filters: Allow you to stream log data in real time to other services such as Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, or AWS Lambda for further processing, analysis, or loading into other systems.
- Log Export to S3: You can export log data to Amazon S3 for long-term archival and batch analysis using services like Athena.

3. CloudWatch Alarms
Alarms watch a single metric over a specified time period and perform one or more actions based on the metric value relative to a threshold. Alarm states include:
- OK: The metric is within the defined threshold.
- ALARM: The metric has breached the threshold.
- INSUFFICIENT_DATA: Not enough data to determine the state.

Actions can include:
- Sending notifications via Amazon SNS (which can trigger emails, SMS, Lambda, or other integrations)
- Executing Auto Scaling policies
- Triggering EC2 actions (stop, terminate, reboot, recover)
- Creating OpsItems in Systems Manager or incidents in Incident Manager

Composite Alarms combine multiple alarms using AND/OR logic to reduce alarm noise and create more sophisticated alerting strategies.

4. CloudWatch Dashboards
Custom dashboards provide a global, visual overview of your resources and metrics. Dashboards can include graphs, text widgets, and alarm status widgets. They are cross-region and cross-account capable, making them ideal for centralized monitoring of distributed data platforms.

5. CloudWatch Events / Amazon EventBridge
While Amazon EventBridge has largely superseded CloudWatch Events, the underlying system allows you to respond to state changes in AWS resources. For data engineering, common patterns include:
- Triggering a Lambda function when a Glue job changes state (SUCCEEDED, FAILED)
- Starting a Step Functions workflow when new data arrives in S3
- Sending alerts when a Kinesis stream encounters throttling

6. CloudWatch Contributor Insights
Analyzes log data to create time series that display contributor data, helping identify the top-N contributors (e.g., top IP addresses, most called API endpoints). This can be useful for identifying hotkeys in DynamoDB or noisy tenants in multi-tenant data systems.

7. CloudWatch Anomaly Detection
Uses machine learning to continuously analyze metric data and determine normal baselines. It creates an anomaly detection band, and alarms can be set to trigger when a metric falls outside this band. This is particularly useful for data pipelines where normal throughput varies by time of day or day of week.

How CloudWatch Works in a Data Engineering Context

AWS Glue Monitoring:
- Glue jobs automatically send metrics to CloudWatch including bytes read/written, records processed, JVM heap usage, and executor counts.
- Glue job logs (driver and executor) are written to CloudWatch Logs under /aws/glue/jobs/logs-v2, /aws/glue/jobs/output, and /aws/glue/jobs/error.
- You can enable continuous logging in Glue for real-time log delivery (as opposed to logs appearing only after job completion).
- Spark UI logs can be sent to S3 for deeper profiling with the Spark History Server.
- Set alarms on glue.driver.aggregate.elapsedTime to detect jobs running longer than expected.

Amazon Kinesis Monitoring:
- Kinesis Data Streams publishes shard-level and stream-level metrics.
- GetRecords.IteratorAgeMilliseconds is a critical metric — it tells you how far behind your consumer is from the tip of the stream. A growing iterator age indicates consumer lag.
- ReadProvisionedThroughputExceeded and WriteProvisionedThroughputExceeded indicate you need more shards or are experiencing hot shards.
- Enhanced shard-level monitoring can be enabled for per-shard metrics at additional cost.

Amazon Redshift Monitoring:
- CloudWatch metrics include CPUUtilization, PercentageDiskSpaceUsed, DatabaseConnections, HealthStatus, MaintenanceMode, ReadIOPS, WriteIOPS, ReadLatency, WriteLatency, ReadThroughput, WriteThroughput, and more.
- Query-level monitoring requires Redshift system tables (STL_QUERY, SVL_QLOG) or Redshift Query Monitoring Rules (QMR), not just CloudWatch.
- Redshift audit logs (connection logs, user activity logs, user logs) can be exported to S3 or CloudWatch Logs.

Amazon S3 Monitoring:
- S3 CloudWatch request metrics (must be enabled in bucket configuration) provide per-bucket metrics like AllRequests, GetRequests, PutRequests, 4xxErrors, 5xxErrors, FirstByteLatency, TotalRequestLatency.
- S3 storage metrics (BucketSizeBytes, NumberOfObjects) are free and reported once daily.
- S3 access logging and AWS CloudTrail provide additional audit-level visibility.

AWS Lambda Monitoring:
- Lambda automatically publishes Invocations, Duration, Errors, Throttles, ConcurrentExecutions, IteratorAge (for stream-based invocations).
- Lambda function logs are sent to CloudWatch Logs automatically.
- Duration metrics help identify cold starts and performance bottlenecks in data transformation functions.

Amazon EMR Monitoring:
- EMR publishes cluster-level metrics (IsIdle, CoreNodesRunning, AppsRunning, etc.).
- YARN, HDFS, and Spark metrics are forwarded to CloudWatch.
- CloudWatch alarms can trigger cluster scaling policies.

Step Functions Monitoring:
- Metrics include ExecutionsStarted, ExecutionsFailed, ExecutionsSucceeded, ExecutionsTimedOut.
- Execution history and state machine logs can be sent to CloudWatch Logs for debugging complex data orchestration workflows.

How to Set Up Effective Monitoring for Data Pipelines

1. Define key metrics for each stage of your pipeline (ingestion, transformation, loading, serving).
2. Set CloudWatch Alarms on critical thresholds: Glue job failures, Kinesis iterator age > threshold, Redshift disk > 80%, Lambda error rate > 1%.
3. Use metric filters on logs to extract business-relevant KPIs (e.g., records processed, transformation errors).
4. Create dashboards that provide end-to-end pipeline visibility.
5. Configure subscription filters to stream important logs to a centralized analytics platform.
6. Set retention policies on log groups to balance cost with compliance requirements.
7. Use CloudWatch Logs Insights for troubleshooting query patterns across services.
8. Leverage EventBridge for event-driven automation — auto-remediation of failed jobs, notifications, and workflow triggering.

CloudWatch vs. Other Monitoring Services

- CloudWatch vs. CloudTrail: CloudWatch monitors performance and operational metrics. CloudTrail records API calls for auditing and governance. They are complementary, not competing services.
- CloudWatch vs. X-Ray: X-Ray provides distributed tracing for request-level analysis. CloudWatch provides aggregate metrics and logs. Use X-Ray for tracing individual requests through microservices or Lambda-based data pipelines.
- CloudWatch vs. AWS Config: Config tracks resource configuration changes over time. CloudWatch monitors runtime behavior. Config tells you what changed; CloudWatch tells you how it's performing.

Cost Considerations

- Free tier includes 10 custom metrics, 10 alarms, 1 million API requests, 5 GB of log ingestion, 5 GB of log storage, and 3 dashboards (up to 50 metrics each) per month.
- Log ingestion is charged per GB. Logs storage is charged per GB per month. Set appropriate retention periods.
- High-resolution custom metrics and detailed monitoring cost more than standard metrics.
- CloudWatch Logs Insights charges per GB of data scanned.
- Cross-account and cross-region dashboard access is free, but underlying data transfer costs may apply.

---

Exam Tips: Answering Questions on Logging and Monitoring with CloudWatch

Tip 1: Know Which Service Uses Which Log Group
Exam questions often present a scenario where you need to find logs for a specific service. Remember the log group naming conventions: /aws/lambda/function-name for Lambda, /aws/glue/jobs/ for Glue, /aws/redshift/ for Redshift, etc.

Tip 2: Understand the Difference Between Metrics, Logs, and Events
If a question asks about tracking numerical performance over time, the answer is CloudWatch Metrics. If it asks about debugging specific errors or viewing output, it's CloudWatch Logs. If it asks about reacting to state changes, it's CloudWatch Events/EventBridge.

Tip 3: Iterator Age Is the Key Kinesis Metric
When a question describes a Kinesis consumer falling behind or data processing delays, the metric to monitor is GetRecords.IteratorAgeMilliseconds. This is a very commonly tested concept.

Tip 4: Metric Filters for Custom Metrics from Logs
If a question asks how to create alarms based on patterns in log files (e.g., count errors, track specific keywords), the answer is CloudWatch Metric Filters. These extract numerical data from log events and publish them as custom metrics that you can alarm on.

Tip 5: Subscription Filters for Real-Time Log Processing
When the scenario involves streaming logs in real time to another service (Kinesis, Lambda, Elasticsearch/OpenSearch), the answer is Subscription Filters. If the question mentions batch export or long-term archival, the answer is export to S3.

Tip 6: CloudWatch Logs Insights for Ad-Hoc Queries
If a question asks about querying or searching log data interactively, CloudWatch Logs Insights is the answer. It does not require any pre-configuration — just write a query. Remember it uses its own query syntax, not SQL.

Tip 7: Composite Alarms Reduce Alarm Noise
If the question describes a scenario with too many alarms firing independently and the need to alert only when multiple conditions are true simultaneously, the answer is Composite Alarms.

Tip 8: Anomaly Detection for Variable Workloads
When a question describes workloads with predictable patterns (higher on weekdays, lower on weekends) and asks how to set dynamic thresholds, the answer is CloudWatch Anomaly Detection.

Tip 9: Glue Continuous Logging
If a question asks how to view Glue job logs in real time (while the job is still running), the answer is to enable continuous logging. Without it, logs only appear after the job completes or fails.

Tip 10: CloudWatch vs. CloudTrail — Don't Confuse Them
This is one of the most common exam traps. If the question is about who made an API call, what resource was modified, or auditing access, the answer is CloudTrail, not CloudWatch. If it's about performance monitoring, resource utilization, or operational health, the answer is CloudWatch.

Tip 11: Retention and Cost Optimization
Questions about reducing CloudWatch costs will likely involve adjusting log retention periods, using S3 export for long-term storage (cheaper than CloudWatch log storage), or being selective about which metrics are published at high resolution.

Tip 12: Cross-Account and Cross-Region Monitoring
If a question describes a multi-account or multi-region data platform needing centralized monitoring, know that CloudWatch supports cross-account observability through sharing configurations and that dashboards can display metrics from multiple accounts and regions.

Tip 13: Embedded Metric Format (EMF)
If a question asks about efficiently publishing custom metrics from Lambda or container-based applications alongside log data, the answer is the CloudWatch Embedded Metric Format. EMF allows you to embed metric data within structured log events, and CloudWatch automatically extracts the metrics without needing separate PutMetricData API calls.

Tip 14: Remember Default vs. Detailed Monitoring
EC2 and some services default to 5-minute metric intervals. Detailed monitoring provides 1-minute intervals at extra cost. For services like Lambda and Glue, metrics are published at 1-minute intervals by default.

Tip 15: Always Consider the End-to-End Pipeline
Exam questions may present complex multi-service data pipeline scenarios. Think about monitoring at each stage: ingestion (Kinesis metrics), processing (Glue/Lambda metrics and logs), storage (S3 metrics, Redshift metrics), and orchestration (Step Functions execution metrics). The best monitoring strategy covers all stages with appropriate alarms and dashboards.