Centralized Logging Architecture – AWS Security Specialty Guide
Why Centralized Logging Architecture Matters
In any enterprise cloud environment, logs are generated from dozens or even hundreds of sources: VPC Flow Logs, AWS CloudTrail, Amazon CloudWatch Logs, AWS Config, Application Load Balancer access logs, S3 access logs, Route 53 DNS query logs, GuardDuty findings, and application-level logs. Without a centralized logging strategy, these logs remain siloed across accounts, regions, and services. This makes it nearly impossible to perform effective security analysis, incident response, forensic investigation, or compliance auditing.
Centralized logging is a foundational pillar of the AWS Well-Architected Framework's Security Pillar. It enables organizations to detect threats faster, maintain a single source of truth for audit evidence, correlate events across accounts and services, and meet regulatory requirements such as PCI-DSS, HIPAA, SOC 2, and FedRAMP.
What Is Centralized Logging Architecture?
Centralized logging architecture is a design pattern where log data from multiple AWS accounts, regions, and services is aggregated into a single, secure, and durable location for storage, analysis, and alerting. The architecture typically involves:
• Log Sources: CloudTrail, VPC Flow Logs, CloudWatch Logs, Config, S3 access logs, ELB logs, GuardDuty findings, Security Hub findings, WAF logs, and custom application logs.
• Log Aggregation Layer: Mechanisms to collect and forward logs from member accounts to a central account, such as CloudWatch Logs cross-account subscriptions, S3 bucket replication, or Kinesis Data Firehose streams.
• Central Log Storage: A dedicated logging account (often called the Log Archive account in AWS Organizations) with an S3 bucket as the primary log repository. This account is tightly controlled with minimal access.
• Analysis and Search Layer: Tools like Amazon OpenSearch Service (formerly Elasticsearch Service), Amazon Athena, or third-party SIEM solutions for querying and visualizing log data.
• Alerting and Response: Amazon CloudWatch Alarms, Amazon SNS, AWS Lambda, and AWS Security Hub for real-time detection and automated response.
How Centralized Logging Architecture Works
Step 1: Enable Logging Across All Accounts
Use AWS Organizations with Service Control Policies (SCPs) to ensure that CloudTrail, Config, and other logging services cannot be disabled by member accounts. Create an organization trail in CloudTrail that automatically logs API activity across all accounts in the organization. Enable VPC Flow Logs for all VPCs, and enable S3 access logging and ELB access logging as needed.
Step 2: Designate a Central Logging Account
In an AWS Organizations multi-account strategy, designate a dedicated Log Archive account. This account should be separate from the security tooling account and production accounts. Apply strict access controls: only a small number of security and compliance team members should have read access. No one should have delete permissions on the log S3 bucket.
Step 3: Configure Cross-Account Log Delivery
There are several mechanisms for centralizing logs:
• CloudTrail Organization Trail: Automatically delivers CloudTrail logs from all organization member accounts to a central S3 bucket in the Log Archive account. This is the simplest and most recommended approach for CloudTrail.
• CloudWatch Logs Cross-Account Subscriptions: Use subscription filters in member accounts to stream CloudWatch Logs to a Kinesis Data Stream or Kinesis Data Firehose in the central account, which then delivers to S3 or OpenSearch.
• S3 Cross-Account Bucket Policies: Configure bucket policies on the central S3 bucket to allow member accounts to write logs directly (used by services like VPC Flow Logs, Config, and ELB access logs).
• Amazon Kinesis Data Firehose: Acts as a managed delivery pipeline that can batch, compress, encrypt, and deliver log data to S3, OpenSearch, or Redshift.
• AWS Config Aggregator: Aggregates Config data from multiple accounts and regions into a single dashboard in the central account.
Step 4: Secure the Central Log Repository
Security of the log archive is paramount:
• Enable S3 Object Lock (Compliance mode) to make logs immutable — even the root user cannot delete them during the retention period.
• Enable S3 Versioning to protect against accidental overwrites.
• Apply S3 Bucket Policies that deny deletion and restrict write access to only the expected log-producing services and accounts.
• Enable SSE-KMS encryption with a customer-managed KMS key. Control access to the key using KMS key policies and IAM policies.
• Enable S3 access logging on the log bucket itself (meta-logging) to detect unauthorized access attempts.
• Use SCPs to prevent anyone in the organization from disabling CloudTrail or modifying the log archive bucket.
• Enable MFA Delete on the S3 bucket as an additional safeguard.
• Apply lifecycle policies to transition logs to S3 Glacier or S3 Glacier Deep Archive for cost-effective long-term retention.
Step 5: Enable Analysis and Querying
• Amazon Athena: Use Athena with AWS Glue to catalog and query CloudTrail logs, VPC Flow Logs, and other log data stored in S3 using standard SQL. This is serverless and cost-effective for ad-hoc analysis.
• Amazon OpenSearch Service: Use Kinesis Data Firehose to stream logs into OpenSearch for near-real-time search, dashboards, and alerting. OpenSearch Dashboards (Kibana) provides visual analytics.
• Amazon CloudWatch Logs Insights: For logs stored in CloudWatch Logs, use Logs Insights for interactive querying.
• AWS Security Hub: Aggregates findings from GuardDuty, Inspector, Macie, IAM Access Analyzer, and Firewall Manager into a centralized security dashboard. Supports cross-account and cross-region aggregation.
• Amazon Detective: Uses log data (CloudTrail, VPC Flow Logs, GuardDuty findings) to build a graph model for security investigation and root cause analysis.
Step 6: Implement Real-Time Alerting
• Create CloudWatch Metric Filters on CloudTrail logs to detect specific API calls (e.g., unauthorized API calls, root account usage, changes to security groups, IAM policy changes).
• Use CloudWatch Alarms to trigger SNS notifications or Lambda functions for automated remediation.
• Use EventBridge rules to capture specific events (e.g., GuardDuty findings, Config compliance changes) and route them to targets like Lambda, SNS, or Step Functions.
• Integrate with AWS Security Hub for consolidated findings and automated response workflows.
Key Architecture Patterns to Know
Pattern 1: Organization Trail + Central S3 Bucket
CloudTrail organization trail → Central S3 bucket in Log Archive account → Athena for querying → CloudWatch Alarms for alerting.
Pattern 2: Real-Time Streaming Pipeline
CloudWatch Logs (member accounts) → Subscription Filters → Kinesis Data Stream (central account) → Kinesis Data Firehose → Amazon OpenSearch Service → OpenSearch Dashboards for visualization and alerting.
Pattern 3: Security Hub Centralization
GuardDuty, Config, Inspector, Macie (all accounts) → Security Hub delegated administrator account → EventBridge rules → Lambda for automated remediation.
Pattern 4: Cross-Region Aggregation
Enable CloudTrail in all regions and aggregate to a single S3 bucket. Use S3 replication to a secondary region for disaster recovery. Use Config Aggregator to view compliance status across all regions and accounts.
Common Services in Centralized Logging Scenarios
• AWS CloudTrail: API activity logging — management events, data events, and Insights events.
• Amazon CloudWatch Logs: Operational and application logs with metric filters and alarms.
• VPC Flow Logs: Network traffic metadata (source/destination IP, port, protocol, action).
• AWS Config: Configuration change history and compliance evaluation.
• Amazon S3 Access Logs / CloudTrail Data Events: Object-level access logging for S3.
• Amazon GuardDuty: Intelligent threat detection using ML on CloudTrail, VPC Flow Logs, and DNS logs.
• AWS Security Hub: Centralized security findings aggregation and compliance checks.
• Amazon Kinesis Data Firehose: Managed log delivery pipeline.
• Amazon Athena: Serverless SQL querying of log data in S3.
• Amazon OpenSearch Service: Near-real-time log search and visualization.
Exam Tips: Answering Questions on Centralized Logging Architecture
1. Always think multi-account first: The AWS Security Specialty exam assumes a multi-account environment managed by AWS Organizations. If a question mentions multiple accounts, your answer should involve a dedicated Log Archive account with cross-account access via bucket policies or organization trails.
2. Organization Trail is the answer for CloudTrail centralization: When asked how to ensure CloudTrail logs are collected from all accounts in an organization, the answer is almost always an organization trail configured in the management account that delivers to a central S3 bucket.
3. Immutability = S3 Object Lock (Compliance Mode): If a question asks about preventing log tampering or ensuring logs cannot be deleted, the answer is S3 Object Lock in Compliance mode. Remember that even the root user cannot delete objects in Compliance mode before the retention period expires.
4. SCPs for enforcement: If a question asks how to prevent member accounts from disabling CloudTrail or deleting logs, the answer is Service Control Policies applied at the organizational level.
5. Know the difference between CloudWatch Logs Subscriptions and S3 delivery: CloudTrail delivers directly to S3. CloudWatch Logs can be exported to S3 (batch, with a delay) or streamed in near-real-time via subscription filters to Kinesis or Lambda. For real-time analysis, subscription filters are preferred.
6. Encryption questions: When asked about encrypting centralized logs, the answer is SSE-KMS with a customer-managed key. The KMS key policy must grant decrypt permissions to the accounts/services that need to read the logs and encrypt permissions to the services writing the logs.
7. Cross-account KMS key access: Remember that for cross-account log delivery with KMS encryption, the KMS key policy in the Log Archive account must allow the source accounts or services to use the key for encryption. CloudTrail requires the key policy to allow cloudtrail.amazonaws.com to call kms:GenerateDataKey.
8. Athena vs. OpenSearch: If the question mentions ad-hoc querying or cost-effective analysis, choose Athena. If the question mentions real-time dashboards, near-real-time alerting, or SIEM-like capabilities, choose OpenSearch Service.
9. Look for keywords: "tamper-proof" → S3 Object Lock. "real-time" → Kinesis + OpenSearch or CloudWatch Alarms. "all accounts" → Organization Trail or Security Hub delegated administrator. "compliance audit" → CloudTrail + S3 with retention policies. "investigate security incident" → Amazon Detective or Athena queries on CloudTrail logs.
10. Security Hub delegated administrator: Know that Security Hub supports a delegated administrator account that automatically receives findings from all member accounts. This is different from the management account and is the recommended pattern.
11. Do not confuse CloudTrail with Config: CloudTrail records API calls (who did what, when). Config records resource configuration state (what changed and whether it is compliant). Both are important for centralized logging, but they serve different purposes.
12. VPC Flow Logs destination: VPC Flow Logs can be sent directly to S3 (including cross-account S3 buckets) or to CloudWatch Logs. For centralized logging, sending directly to a central S3 bucket is simpler and more cost-effective. If real-time analysis is needed, send to CloudWatch Logs and use subscription filters.
13. Lifecycle and retention: Be prepared for questions about long-term log retention. The pattern is: S3 Standard for recent logs → S3 Standard-IA or S3 Intelligent-Tiering after 30-90 days → S3 Glacier after 1 year → S3 Glacier Deep Archive for long-term compliance retention (7-10 years).
14. CloudWatch cross-account observability: AWS now supports CloudWatch cross-account observability, allowing a monitoring account to view metrics, logs, and traces from source accounts. Know this exists for scenarios asking about centralized monitoring without moving logs.
15. Eliminate distractors: If an answer option suggests using AWS Trusted Advisor or AWS Personal Health Dashboard for centralized logging, it is likely a distractor. Stick with CloudTrail, CloudWatch Logs, S3, Kinesis, Athena, OpenSearch, and Security Hub as the core centralized logging services.
