Amazon CloudWatch Logs and Metrics: Complete Guide for AWS Security Specialty
Why Amazon CloudWatch Logs and Metrics Matter
Amazon CloudWatch Logs and Metrics form the backbone of security monitoring and observability in AWS. For the AWS Security Specialty exam, this topic is critical because security professionals must be able to detect anomalies, investigate incidents, and maintain continuous compliance across AWS environments. Without robust logging and metrics, organizations are effectively blind to threats, misconfigurations, and unauthorized access. CloudWatch serves as the centralized nervous system for collecting, analyzing, and responding to security-relevant data across your entire AWS infrastructure.
What Are Amazon CloudWatch Logs and Metrics?
CloudWatch Metrics are time-ordered sets of data points that represent the behavior of your AWS resources and applications. Metrics are published to CloudWatch by AWS services automatically (basic monitoring) or can be custom-published by your applications. Each metric is defined by a namespace, metric name, and zero or more dimensions.
CloudWatch Logs is a service that enables you to centralize, store, monitor, and analyze log data from AWS services, on-premises servers, and applications. Logs are organized into log groups (logical groupings) and log streams (sequences of log events from a single source).
Key components include:
• Log Groups: Containers for log streams that share the same retention, monitoring, and access control settings.
• Log Streams: Sequences of log events from the same source (e.g., a specific EC2 instance or Lambda function invocation).
• Metric Filters: Patterns applied to log data that extract metric observations from ingested log events and transform them into CloudWatch metrics.
• CloudWatch Alarms: Watch a single metric or the result of a math expression and perform actions based on threshold breaches.
• CloudWatch Logs Insights: An interactive, purpose-built query language for searching and analyzing log data in near real-time.
• Subscription Filters: Enable real-time streaming of log data to other services like Amazon Kinesis Data Streams, Kinesis Data Firehose, or AWS Lambda for further processing.
• Custom Metrics: User-defined metrics published via the PutMetricData API, enabling monitoring of application-specific security indicators.
How Amazon CloudWatch Logs and Metrics Work
1. Data Collection and Ingestion
AWS services automatically send metrics to CloudWatch. For example, EC2 sends CPU utilization metrics, and VPC Flow Logs can be directed to CloudWatch Logs. The CloudWatch Agent (unified agent) can be installed on EC2 instances and on-premises servers to collect system-level metrics (memory, disk usage) and log files. The agent supports both Linux and Windows and is configured via a JSON configuration file.
Key AWS services that integrate with CloudWatch Logs for security:
• AWS CloudTrail: API call logs can be sent to CloudWatch Logs for real-time monitoring of management and data events.
• VPC Flow Logs: Network traffic metadata can be captured and sent to CloudWatch Logs for traffic analysis and anomaly detection.
• Amazon GuardDuty: Findings can trigger CloudWatch Events (EventBridge) rules.
• AWS Config: Configuration changes generate CloudWatch Events.
• Route 53 DNS Query Logs: DNS queries can be logged to CloudWatch Logs.
• S3 Server Access Logs, ELB Access Logs: While these go to S3, they can be processed and sent to CloudWatch.
• Lambda Function Logs: Automatically sent to CloudWatch Logs.
• RDS and Aurora Logs: Database logs can be published to CloudWatch Logs.
2. Metric Filters and Pattern Matching
Metric filters are essential for security monitoring. You define a filter pattern that matches specific text in log events, and when a match is found, CloudWatch increments a custom metric. For example:
• Filter for unauthorized API calls in CloudTrail logs: { ($.errorCode = "*UnauthorizedAccess") || ($.errorCode = "AccessDenied*") }
• Filter for root account usage: { $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }
• Filter for console sign-in failures: { ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }
• Filter for security group changes: { ($.eventName = AuthorizeSecurityGroupIngress) || ($.eventName = RevokeSecurityGroupIngress) }
These metric filters are fundamental to implementing the CIS AWS Foundations Benchmark monitoring recommendations.
3. CloudWatch Alarms
Once metric filters create metrics, alarms can be configured to trigger notifications and automated responses. Alarms have three states: OK, ALARM, and INSUFFICIENT_DATA. Actions can include:
• Sending notifications via Amazon SNS (email, SMS, HTTP endpoints, Lambda)
• Triggering Auto Scaling actions
• Executing EC2 actions (stop, terminate, reboot, recover)
• Creating OpsItems in Systems Manager OpsCenter
• Triggering EventBridge rules for complex automated remediation workflows
4. CloudWatch Logs Insights
Logs Insights provides a powerful query language for ad-hoc security investigations. You can query across multiple log groups simultaneously, which is invaluable during incident response. Example queries include finding the top 20 source IPs generating rejected VPC flow log entries, or identifying unusual API call patterns in CloudTrail logs.
5. Subscription Filters and Cross-Account Log Aggregation
Subscription filters allow real-time processing of log data by streaming to:
• Kinesis Data Streams: For real-time processing with custom consumers
• Kinesis Data Firehose: For delivery to S3, Redshift, Elasticsearch, or Splunk
• Lambda: For custom processing and transformation
For multi-account architectures, you can use cross-account log subscriptions with a destination in a centralized security account. This involves creating a destination in the receiving account and granting permissions via a resource policy.
CloudWatch Logs can also export data to S3 using the CreateExportTask API for long-term archival, but this is not real-time — it can take up to 12 hours. For near real-time delivery to S3, use Subscription Filters with Kinesis Data Firehose.
6. Log Retention and Encryption
By default, CloudWatch Logs are retained indefinitely (never expire). You can configure retention policies per log group ranging from 1 day to 10 years, or keep them indefinitely. Logs are encrypted at rest by default using AWS-managed keys, but you can use AWS KMS customer-managed keys (CMKs) for additional control via the associate-kms-key or create-log-group API calls. The KMS key policy must grant CloudWatch Logs service principal permission to use the key.
7. CloudWatch Anomaly Detection
CloudWatch Anomaly Detection applies machine learning algorithms to continuously analyze metrics and determine normal baselines. It creates a model of expected behavior and generates anomaly detection bands. This is particularly useful for security because it can detect unusual patterns such as unexpected spikes in API calls, network traffic, or error rates without requiring you to manually set static thresholds.
8. CloudWatch Events / Amazon EventBridge
While technically a separate service now (Amazon EventBridge), CloudWatch Events rules are closely tied to monitoring. They enable event-driven security automation by matching events from AWS services and routing them to targets like Lambda, SNS, SQS, Step Functions, and more. Security use cases include:
• Responding to CloudTrail events (e.g., someone disabling CloudTrail)
• Reacting to GuardDuty findings
• Triggering remediation when Config rules become non-compliant
• Detecting IAM policy changes
Security-Specific CloudWatch Capabilities
• CloudWatch Contributor Insights: Analyzes log data and creates time series displaying contributor data, such as top-N contributors to network traffic or error patterns. Useful for identifying bad actors or misconfigured resources.
• CloudWatch Synthetics (Canaries): Scripts that monitor endpoints and APIs, useful for detecting availability issues that could indicate a DDoS attack or compromise.
• Embedded Metric Format (EMF): Allows applications to embed custom metrics within structured log events, enabling both detailed logs and aggregated metrics from a single source.
• Cross-Account Observability: CloudWatch supports cross-account dashboards and metrics sharing, enabling centralized security monitoring across an AWS Organization.
IAM Permissions for CloudWatch
Access to CloudWatch Logs and Metrics is controlled through IAM policies. Key actions include:
• logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents — for writing logs
• logs:GetLogEvents, logs:FilterLogEvents, logs:StartQuery — for reading logs
• logs:PutMetricFilter, logs:PutSubscriptionFilter — for configuring filters
• cloudwatch:PutMetricData, cloudwatch:GetMetricData — for metrics
• cloudwatch:PutMetricAlarm, cloudwatch:DescribeAlarms — for alarms
Resource-level permissions and condition keys can further restrict access to specific log groups, ensuring least privilege for security-sensitive log data.
Exam Tips: Answering Questions on Amazon CloudWatch Logs and Metrics
• Real-time vs. Batch Processing: If a question asks about real-time log processing, the answer is Subscription Filters (to Kinesis or Lambda), NOT the S3 export task (CreateExportTask), which can take up to 12 hours and is near-real-time at best.
• CIS Benchmark Monitoring: Many questions reference implementing CIS AWS Foundations Benchmark controls. The correct approach is CloudTrail → CloudWatch Logs → Metric Filters → CloudWatch Alarms → SNS notifications. Know the common metric filter patterns for root usage, unauthorized API calls, console sign-in failures, IAM policy changes, security group changes, NACL changes, and CloudTrail configuration changes.
• Cross-Account Log Aggregation: For multi-account scenarios, remember that CloudWatch Logs supports cross-account subscription filters using a destination resource with a resource-based policy. The destination can be in a centralized security/log archive account.
• Encryption: CloudWatch Logs supports encryption with KMS CMKs. The KMS key policy must allow the logs.region.amazonaws.com service principal to perform cryptographic operations. This is associated at the log group level.
• Retention: Default retention is indefinite. If a question mentions cost optimization or compliance-driven retention, configure retention settings per log group. For long-term archival at lower cost, export to S3 or stream via Kinesis Data Firehose to S3.
• Agent vs. API: The unified CloudWatch Agent collects both logs and custom system metrics (memory, disk). It must be installed and configured on EC2 instances. An IAM role with appropriate permissions must be attached to the instance.
• VPC Flow Logs Destination: VPC Flow Logs can go to CloudWatch Logs, S3, or Kinesis Data Firehose. If the question involves real-time alerting on network anomalies, CloudWatch Logs with metric filters and alarms is the right choice. If it involves long-term storage and analysis with Athena, S3 is better.
• CloudWatch vs. CloudTrail: Do not confuse them. CloudTrail records API activity (who did what). CloudWatch monitors operational metrics and logs (what is happening). CloudTrail logs can be sent to CloudWatch Logs for real-time analysis.
• Alarms and Actions: CloudWatch Alarms can directly trigger SNS, Auto Scaling, and EC2 actions. For more complex remediation (e.g., revoking a security group rule), the pattern is Alarm → SNS → Lambda or Alarm → EventBridge → Lambda/Step Functions.
• Logs Insights vs. Metric Filters: Logs Insights is for ad-hoc interactive queries and incident investigation. Metric Filters are for continuous, automated monitoring that feeds into alarms. Both are important but serve different purposes.
• Anomaly Detection: If a question asks about detecting unusual patterns without defining static thresholds, CloudWatch Anomaly Detection is the answer. It uses machine learning to establish baselines.
• Embedded Metric Format: If a question asks about generating custom metrics from Lambda functions or applications without making separate PutMetricData API calls, EMF is the answer — it embeds metrics in log output.
• Log Group Resource Policy: For services like Route 53, Elasticsearch, or API Gateway to write logs to CloudWatch, a resource-based policy on the log group may be required. This is different from IAM roles.
• Watch for Distractors: Questions may include AWS X-Ray, AWS Config, or third-party tools as answer choices. Stay focused on what CloudWatch natively provides versus what requires additional services. For API-level auditing, the answer involves CloudTrail + CloudWatch. For resource configuration compliance, the answer involves AWS Config (potentially with CloudWatch Events for notifications).
• Cost Considerations: Be aware that CloudWatch Logs ingestion and storage incur costs. High-volume logs (like VPC Flow Logs) can be expensive in CloudWatch. Questions about cost-effective solutions may point toward sending logs to S3 instead and using Athena for queries.
• Key Integration Pattern to Remember: The most commonly tested security monitoring architecture is: AWS Service → CloudTrail → CloudWatch Logs → Metric Filter → CloudWatch Alarm → SNS Topic → Lambda Function (automated remediation). This end-to-end pattern covers detection, alerting, and response.
