Compromised EC2 Instances: A Comprehensive Guide for AWS Security Specialty
Why Compromised EC2 Instances Matter
Compromised EC2 instances represent one of the most critical security threats in AWS environments. When an EC2 instance is compromised, an attacker can potentially exfiltrate sensitive data, launch further attacks against other resources in your VPC, use the instance for cryptocurrency mining, participate in botnets, or pivot laterally across your infrastructure. Understanding how to detect, respond to, and remediate compromised EC2 instances is a core competency tested on the AWS Security Specialty exam and is essential for any cloud security professional.
The AWS Shared Responsibility Model places the security in the cloud squarely on the customer. This means that while AWS secures the underlying infrastructure, you are responsible for securing your EC2 instances, including their operating systems, applications, and data. A compromised instance can lead to regulatory violations, data breaches, financial losses, and reputational damage.
What Is a Compromised EC2 Instance?
A compromised EC2 instance is an instance that has been accessed, modified, or controlled by an unauthorized entity. This can happen through various attack vectors, including:
- Stolen or leaked credentials (SSH keys, IAM role credentials)
- Unpatched vulnerabilities in the operating system or applications
- Misconfigured security groups allowing unrestricted inbound access
- Malware infection through phishing, supply chain attacks, or drive-by downloads
- Exploited application-level vulnerabilities (SQL injection, remote code execution)
- Compromised IAM role credentials extracted from the instance metadata service (IMDS)
Signs of compromise may include unusual network traffic patterns, unexpected CPU utilization spikes, unauthorized outbound connections, new or modified user accounts, changed system files, or GuardDuty findings.
How Detection Works
AWS provides several services that work together to detect compromised EC2 instances:
1. Amazon GuardDuty
GuardDuty is the primary detection service for identifying compromised EC2 instances. It analyzes VPC Flow Logs, DNS logs, and CloudTrail events using machine learning, anomaly detection, and integrated threat intelligence. Key finding types related to compromised EC2 instances include:
- Backdoor:EC2/DenialOfService.Tcp – Instance is participating in a DDoS attack
- CryptoCurrency:EC2/BitcoinTool.B!DNS – Instance is querying cryptocurrency mining pool domains
- Trojan:EC2/BlackholeTraffic – Instance is communicating with a known black hole IP
- UnauthorizedAccess:EC2/TorClient – Instance is making connections to the Tor network
- UnauthorizedAccess:EC2/MaliciousIPCaller.Custom – Instance is communicating with IPs on a custom threat list
- Recon:EC2/PortProbeUnprotectedPort – An unprotected port is being probed
- Behavior:EC2/TrafficVolumeUnusual – Instance is generating unusually large traffic volume
2. AWS CloudTrail
CloudTrail logs API calls made by or on behalf of the EC2 instance (via its IAM role). Unusual API activity, such as attempts to access resources the instance does not normally use, can indicate compromise.
3. VPC Flow Logs
VPC Flow Logs capture information about the IP traffic going to and from network interfaces. They can reveal unusual communication patterns, connections to known malicious IPs, or data exfiltration attempts.
4. Amazon CloudWatch
CloudWatch can monitor metrics like CPU utilization, network traffic, and disk I/O. Alarms can be configured to trigger when abnormal patterns are detected. The CloudWatch agent can also collect system-level logs.
5. AWS Security Hub
Security Hub aggregates findings from GuardDuty, Inspector, Macie, and third-party tools, providing a centralized view of security alerts including those related to compromised instances.
6. Amazon Inspector
While primarily a vulnerability assessment tool, Inspector can identify vulnerabilities that may have been exploited to compromise an instance.
Incident Response Procedure for Compromised EC2 Instances
The AWS-recommended incident response procedure for a compromised EC2 instance follows a specific order of operations. This is heavily tested on the exam:
Step 1: Isolate the Instance
- Do NOT terminate the instance immediately – you need it for forensic investigation
- Change the instance's security group to a forensic isolation security group that allows no inbound or outbound traffic (or only very restricted access for forensic analysts)
- Remove the instance from any Auto Scaling groups (by detaching it, or suspending ASG processes) to prevent ASG from terminating it
- Deregister the instance from any Elastic Load Balancers
- Important: Do NOT simply stop the instance, as volatile memory data will be lost
Step 2: Capture Metadata and Evidence
- Tag the instance with incident-related metadata (incident ID, date, analyst name)
- Capture instance metadata (instance ID, IP addresses, security groups, IAM role, launch time)
- Enable termination protection to prevent accidental termination
- Take EBS snapshots of all attached volumes for forensic analysis
- If possible, capture a memory dump before any other actions
- Collect relevant logs (CloudTrail, VPC Flow Logs, system logs, application logs)
Step 3: Investigate and Contain
- Launch a new forensic EC2 instance in an isolated forensic VPC
- Attach the EBS snapshot volumes to the forensic instance as secondary volumes (do NOT boot from them)
- Perform forensic analysis: examine file system changes, review logs, check for malware, identify the attack vector
- Review CloudTrail logs to identify any API calls made using the instance's IAM role credentials
- Check for lateral movement to other instances or services
Step 4: Revoke and Rotate Credentials
- If the instance had an IAM role, invalidate any temporary credentials that may have been stolen by creating a deny all SCP or IAM policy with a condition on the token issue time, or by revoking the role's active sessions
- Rotate any other credentials that were accessible from the instance (database passwords, API keys stored on the instance, SSH keys)
- Revoke any key pairs associated with the instance
Step 5: Eradicate and Recover
- Do NOT try to clean the compromised instance – always replace it
- Launch a new, clean instance from a known-good AMI
- Apply all security patches and hardening measures
- Restore data from clean backups (verified to be pre-compromise)
- Update security groups, NACLs, and other security controls as needed
Step 6: Post-Incident Activities
- Document the incident, timeline, actions taken, and lessons learned
- Update detection rules and monitoring
- Review and improve security controls to prevent recurrence
- Consider implementing AWS Systems Manager Patch Manager for automated patching
- Review IMDSv2 enforcement to prevent SSRF-based credential theft
Key AWS Services for Automation
- AWS Lambda + GuardDuty + EventBridge: Automate the isolation response when GuardDuty detects a compromised instance. EventBridge rules can trigger Lambda functions that automatically change security groups, snapshot volumes, and notify the security team.
- AWS Step Functions: Orchestrate complex multi-step incident response workflows
- AWS Systems Manager (SSM): Run commands on instances for forensic data collection, and use Automation runbooks for standardized response procedures
- AWS SSM Automation Document (Runbook) AWS-IsolateEC2Instance: A pre-built automation for isolating instances
Protecting the Instance Metadata Service (IMDS)
A common attack vector involves exploiting Server-Side Request Forgery (SSRF) vulnerabilities to access the EC2 instance metadata service at 169.254.169.254 and steal IAM role credentials. Mitigations include:
- Enforce IMDSv2 (requires session tokens, mitigating SSRF attacks)
- Set HttpTokens to required when launching instances
- Use HttpPutResponseHopLimit of 1 to prevent container-based attacks
- Use AWS Config rules to enforce IMDSv2 across your environment
Exam Tips: Answering Questions on Compromised EC2 Instances
1. Never terminate a compromised instance immediately. The correct first step is always to isolate the instance by changing its security group to one with no inbound or outbound rules. Termination destroys forensic evidence.
2. Isolation comes before investigation. If a question asks about the order of operations, isolation (changing security groups) is always the first priority to prevent further damage and lateral movement.
3. Know the difference between security groups and NACLs for isolation. Security groups are stateful and can be swapped instantly. NACLs are stateless and apply at the subnet level (affecting other instances). For isolating a single instance, changing the security group is the preferred approach.
4. EBS snapshots are critical for forensics. Always snapshot volumes before any remediation. Mount snapshots on a separate forensic instance – never boot from the compromised volume.
5. Understand IAM role credential revocation. If an instance's IAM role credentials were potentially stolen, you need to revoke active sessions. This is done by adding a deny policy with an aws:TokenIssueTime condition, or by using the IAM console's Revoke Sessions feature. Simply removing the role does not invalidate already-issued temporary credentials (they remain valid until they expire).
6. GuardDuty is the primary detection service. If a question mentions detecting cryptocurrency mining, DDoS participation, communication with known malicious IPs, or unusual DNS queries, GuardDuty is almost always the answer.
7. Automation is preferred over manual response. If a question offers both manual and automated approaches, the automated approach (EventBridge + Lambda, Step Functions, SSM Automation) is typically the correct answer.
8. Replace, don't repair. AWS best practice is to terminate the compromised instance and launch a new one from a clean AMI. Never try to clean malware from a compromised instance.
9. Auto Scaling considerations. If the compromised instance is part of an Auto Scaling group, detach it from the group (or suspend processes) before isolation to prevent ASG from terminating it and launching a replacement that might also be compromised.
10. Memory capture before stopping. If a question involves volatile data or memory forensics, remember that stopping an instance destroys RAM contents. Memory dumps should be captured while the instance is running.
11. Look for the IMDS angle. Questions about SSRF attacks leading to credential theft from EC2 instances are asking about IMDSv2. The solution is to enforce IMDSv2 with HttpTokens set to required.
12. CloudTrail + Athena for investigation. When a question asks how to investigate what API calls a compromised instance made, the answer typically involves querying CloudTrail logs with Amazon Athena, filtering by the instance's IAM role session.
13. Know the complete response workflow: Isolate → Tag/Metadata → Snapshot → Forensic Analysis → Credential Revocation → Eradication → Recovery → Post-Incident Review
14. VPC Flow Logs for network forensics. Questions about understanding network communication patterns of a compromised instance point to VPC Flow Logs. For deeper packet inspection, you would need a third-party tool or traffic mirroring.
15. Security Hub for centralized visibility. If a question asks about aggregating and prioritizing findings across multiple accounts or services, Security Hub is the answer. It integrates with GuardDuty, Inspector, and other services to provide a unified dashboard.
By mastering the detection, isolation, investigation, and remediation workflow for compromised EC2 instances, and understanding which AWS services play which roles, you will be well-prepared to answer these questions confidently on the AWS Security Specialty exam.
