VPC Flow Logs

VPC Flow Logs: Complete Guide for AWS Security Specialty

VPC Flow Logs are one of the most critical logging mechanisms in AWS and a heavily tested topic on the AWS Security Specialty exam. Understanding how they work, what they capture, and their limitations is essential for both real-world security operations and exam success.

Why Are VPC Flow Logs Important?

VPC Flow Logs provide visibility into the network traffic flowing through your Virtual Private Cloud. Without them, you are essentially blind to what is happening at the network layer. They are important because:

- They enable detection of anomalous traffic patterns, such as unusual port scanning, data exfiltration attempts, or unauthorized access attempts.
- They support compliance requirements by providing an audit trail of network communications.
- They help with troubleshooting connectivity issues, such as identifying overly restrictive security group or NACL rules.
- They are a foundational component of a defense-in-depth security strategy on AWS.
- They integrate with other AWS services like Amazon CloudWatch, Amazon S3, and Amazon Athena for analysis and alerting.

What Are VPC Flow Logs?

VPC Flow Logs capture information about the IP traffic going to and from network interfaces in your VPC. They record metadata about each network flow — they do not capture the actual packet payload or content.

A flow log record includes fields such as:
- version — The VPC Flow Logs version
- account-id — The AWS account ID of the owner
- interface-id — The ID of the network interface (ENI)
- srcaddr — The source IP address
- dstaddr — The destination IP address
- srcport — The source port
- dstport — The destination port
- protocol — The IANA protocol number (e.g., 6 = TCP, 17 = UDP)
- packets — The number of packets transferred during the capture window
- bytes — The number of bytes transferred during the capture window
- start — The start time of the capture window (Unix epoch)
- end — The end time of the capture window (Unix epoch)
- action — ACCEPT or REJECT
- log-status — OK, NODATA, or SKIPDATA

With version 3 and later, you can include additional fields such as vpc-id, subnet-id, az-id, pkt-srcaddr, pkt-dstaddr, region, flow-direction, and traffic-path.

How Do VPC Flow Logs Work?

VPC Flow Logs can be created at three levels:

1. VPC level — Captures all traffic for all network interfaces in the entire VPC.
2. Subnet level — Captures all traffic for all network interfaces in a specific subnet.
3. Network Interface (ENI) level — Captures traffic for a specific network interface only.

Key architectural details:

- Flow logs can be published to Amazon CloudWatch Logs, Amazon S3, or Amazon Kinesis Data Firehose.
- When publishing to CloudWatch Logs, a log group is used, and each ENI gets its own log stream.
- When publishing to S3, logs are stored in a structured folder hierarchy and can be queried using Amazon Athena for cost-effective analysis.
- Flow logs have an aggregation interval — either 10 minutes (default) or 1 minute. The 1-minute interval provides more granular data but generates more log volume.
- Flow logs are not real-time. There is a delay of several minutes before log data appears.
- Once a flow log is created, its configuration cannot be modified. You must delete it and create a new one.
- VPC Flow Logs require an IAM role with appropriate permissions when publishing to CloudWatch Logs. For S3, a bucket policy is used instead.
- Flow logs do not impact network throughput or latency — they are captured outside the data path.

What VPC Flow Logs Do NOT Capture

This is a critical exam topic. VPC Flow Logs do not log the following types of traffic:

- Traffic to and from 169.254.169.254 (instance metadata service)
- Traffic to and from 169.254.169.123 (Amazon Time Sync Service)
- DHCP traffic within the VPC
- Traffic to the reserved IP address for the default VPC router
- Traffic between an endpoint network interface and a Network Load Balancer network interface
- DNS traffic to the Amazon-provided DNS server (note: traffic to a custom DNS server IS logged)
- Windows license activation traffic to/from the Amazon Windows license server
- Traffic generated by instances when contacting the Amazon Windows KMS server
- Mirrored traffic (traffic already captured by Traffic Mirroring)

Using VPC Flow Logs for Security Analysis

Common security use cases include:

- Identifying rejected traffic: Filter for action = REJECT to find traffic blocked by security groups or NACLs. This helps identify potential attack attempts or misconfigurations.
- Detecting port scanning: Look for a single source IP sending small packets to many different destination ports.
- Data exfiltration detection: Monitor for unusually large outbound data transfers (high byte counts) to unfamiliar external IP addresses.
- Troubleshooting connectivity: If traffic shows ACCEPT at the security group level but connectivity still fails, the issue may be at the NACL level (since NACLs are stateless and both inbound and outbound rules must allow traffic).
- Integration with GuardDuty: Amazon GuardDuty uses VPC Flow Logs as one of its data sources (note: GuardDuty accesses this data independently — you do not need to enable VPC Flow Logs separately for GuardDuty to analyze them).

Querying VPC Flow Logs with Amazon Athena

When flow logs are stored in S3, you can use Amazon Athena to run SQL queries against them. This is a common pattern tested on the exam. For example:

- Query for all rejected SSH traffic (dstport = 22, action = REJECT)
- Query for top talkers by source IP address
- Query for traffic patterns over specific time ranges

Using Parquet format for flow logs stored in S3 reduces query costs and improves Athena performance compared to plain text.

VPC Flow Logs vs. Other Logging Mechanisms

- VPC Flow Logs: Network-level metadata (IP addresses, ports, accept/reject). No payload inspection.
- AWS CloudTrail: API-level activity logging. Captures who did what via AWS API calls.
- Traffic Mirroring: Full packet capture (including payload) for deep packet inspection. More expensive and resource-intensive.
- ELB Access Logs: HTTP/HTTPS request-level logs from load balancers.
- GuardDuty: Threat detection service that analyzes VPC Flow Logs, DNS logs, and CloudTrail events using machine learning and threat intelligence.

Custom Log Format

With VPC Flow Logs, you can specify a custom format to select only the fields you need. This is useful for reducing storage costs and focusing on relevant data. If you use the default format, only version 2 fields are included. To get version 3+ fields like flow-direction, traffic-path, and pkt-srcaddr/pkt-dstaddr, you must use a custom format.

The pkt-srcaddr and pkt-dstaddr fields are especially important when traffic traverses a NAT gateway or other intermediate device, because srcaddr and dstaddr reflect the translated addresses while pkt-srcaddr and pkt-dstaddr show the original addresses.

Transit Gateway Flow Logs

You can also enable flow logs at the Transit Gateway level to capture traffic flowing through transit gateway attachments. This is useful for monitoring traffic between VPCs, on-premises networks, and other connected networks.

---

Exam Tips: Answering Questions on VPC Flow Logs

1. Remember the three levels: VPC Flow Logs can be enabled at the VPC, subnet, or ENI level. If a question asks about capturing traffic for the entire VPC, the answer is VPC-level flow logs.

2. Know what is NOT logged: This is one of the most commonly tested areas. Memorize the exclusions — especially instance metadata traffic (169.254.169.254), DHCP traffic, and Amazon DNS traffic. If an exam question asks why certain traffic is not appearing in flow logs, these exclusions are likely the answer.

3. Flow logs are not real-time: If a question requires real-time packet inspection, VPC Flow Logs are NOT the answer. Consider Traffic Mirroring or a third-party IDS/IPS solution instead.

4. Flow logs capture metadata, not content: If a question asks about inspecting packet payloads, HTTP headers, or application-layer data, VPC Flow Logs cannot do this. Use Traffic Mirroring for payload inspection.

5. Immutability: You cannot change a flow log after creation. If you need to modify the configuration, you must delete and recreate it.

6. Security Group vs. NACL troubleshooting: If flow logs show traffic as ACCEPT but the connection still fails, think about NACL outbound rules (since NACLs are stateless). If flow logs show REJECT, it could be either a security group or NACL blocking the traffic.

7. GuardDuty and Flow Logs: GuardDuty uses VPC Flow Log data as a source, but it obtains this data independently. You do NOT need to have VPC Flow Logs enabled for GuardDuty to function. However, enabling your own VPC Flow Logs is still recommended for custom analysis.

8. Cost optimization: When a question involves cost-effective analysis of flow logs, the answer typically involves publishing to S3 and querying with Athena (not CloudWatch Logs, which is more expensive for large volumes).

9. pkt-srcaddr vs. srcaddr: If a question involves identifying the original source IP behind a NAT gateway, the answer is to use the pkt-srcaddr field in a custom log format.

10. Cross-account delivery: VPC Flow Logs can be delivered to an S3 bucket in a different AWS account (central logging account) by configuring the appropriate bucket policy. This is a common pattern in multi-account architectures.

11. Encryption: Flow logs published to S3 can be encrypted with SSE-S3 or SSE-KMS. If a question mentions encryption requirements for flow log data at rest, these are the relevant options.

12. Aggregation interval: If a question requires more granular network visibility, choose the 1-minute aggregation interval over the default 10-minute interval.

13. CloudWatch Logs integration: When flow logs are published to CloudWatch Logs, you can create metric filters and CloudWatch Alarms to alert on specific traffic patterns (e.g., alert when rejected traffic exceeds a threshold). This is a common exam scenario for automated security monitoring.