Secrets Management

Secrets Management – AWS Security Specialty Guide

Secrets Management is a foundational pillar of data protection in AWS and a critical topic for the AWS Security Specialty exam. This guide covers what it is, why it matters, how it works in AWS, and how to approach exam questions confidently.

Why Is Secrets Management Important?

Applications, services, and infrastructure components constantly need to authenticate to databases, APIs, third-party services, and other resources. This authentication typically requires credentials such as passwords, API keys, tokens, and certificates. If these secrets are hardcoded in source code, stored in plain text configuration files, or managed manually, they become highly vulnerable to exposure, theft, and misuse.

Poor secrets management can lead to:
- Data breaches from leaked credentials
- Compliance violations (PCI DSS, HIPAA, SOC 2 all require proper credential handling)
- Lateral movement by attackers who compromise one set of credentials
- Operational failures when credentials expire and are not rotated

Proper secrets management ensures that sensitive credentials are encrypted at rest, transmitted securely, automatically rotated, and accessed only by authorized entities with full audit trails.

What Is Secrets Management?

Secrets management is the practice of securely storing, distributing, accessing, and rotating sensitive information (secrets) used by applications and services. A secret is any piece of sensitive data that grants access to a resource—database passwords, SSH keys, OAuth tokens, TLS certificates, API keys, and encryption keys all qualify.

A robust secrets management solution provides:
- Centralized storage with encryption at rest
- Fine-grained access control via policies
- Automatic rotation of credentials on a defined schedule
- Audit logging for every access and modification
- Programmatic retrieval so applications never need hardcoded secrets

How It Works in AWS

AWS provides two primary services for secrets management:

1. AWS Secrets Manager

AWS Secrets Manager is the purpose-built service for managing secrets throughout their lifecycle.

Key Features:
- Encryption: All secrets are encrypted at rest using AWS KMS (Key Management Service). You can use the default AWS-managed key or a customer-managed CMK (Customer Master Key).
- Automatic Rotation: Secrets Manager can automatically rotate secrets for supported services like Amazon RDS (MySQL, PostgreSQL, Oracle, SQL Server, MariaDB), Amazon Redshift, Amazon DocumentDB, and custom services via Lambda functions.
- Rotation Lambda Functions: For supported databases, Secrets Manager provides pre-built Lambda rotation templates. For other services, you write custom Lambda functions that implement four steps: createSecret, setSecret, testSecret, and finishSecret.
- Resource-based Policies: You can attach resource-based policies to secrets to enable cross-account access.
- Versioning: Secrets Manager maintains version stages (AWSCURRENT, AWSPREVIOUS, AWSPENDING) to ensure seamless rotation without downtime.
- VPC Endpoint Support: You can create a VPC endpoint (AWS PrivateLink) so that traffic between your VPC and Secrets Manager never leaves the AWS network.
- Integration with CloudFormation: Secrets can be dynamically referenced in CloudFormation templates using dynamic references (e.g., {{resolve:secretsmanager:MySecret}}).
- CloudTrail Integration: Every API call to Secrets Manager is logged in AWS CloudTrail for auditing.
- Pricing: Secrets Manager charges per secret per month and per 10,000 API calls.

How Retrieval Works:
1. An application (e.g., on EC2 or Lambda) calls the GetSecretValue API.
2. IAM evaluates whether the caller has the required permissions (both IAM policy and any resource-based policy on the secret).
3. KMS decrypts the secret.
4. The plaintext secret is returned to the application over TLS.
5. The application uses the credential and does NOT store it persistently.

How Rotation Works:
1. Secrets Manager invokes the configured Lambda rotation function.
2. The Lambda function creates a new version of the secret (AWSPENDING stage).
3. The function sets the new credential in the target service (e.g., changes the database password).
4. The function tests the new credential to confirm it works.
5. The function marks the new version as AWSCURRENT and the old version as AWSPREVIOUS.

2. AWS Systems Manager Parameter Store

Parameter Store is part of AWS Systems Manager and can also store secrets, though with different capabilities:

Key Differences from Secrets Manager:
- SecureString Parameters: Parameter Store can encrypt values using KMS (SecureString type), but it does NOT natively support automatic rotation. You must implement rotation yourself (e.g., via Lambda + CloudWatch Events/EventBridge).
- Hierarchical Storage: Parameters are organized in a hierarchy (e.g., /prod/database/password) which integrates well with organizational structures.
- Cost: Standard parameters (up to 10,000) are free. Advanced parameters have a cost. This makes Parameter Store cheaper for simple use cases.
- Throughput: Standard tier has lower throughput (40 TPS default, up to 1,000 with higher throughput). Advanced tier and Secrets Manager support higher throughput.
- Size Limits: Standard parameters hold up to 4 KB; Advanced parameters hold up to 8 KB. Secrets Manager supports up to 64 KB.
- No Cross-Account Sharing via Resource Policy: Unlike Secrets Manager, Parameter Store does not support resource-based policies for cross-account access (you must use IAM roles and assume-role patterns).

When to Use Which:
- Use Secrets Manager when you need automatic rotation, cross-account secret sharing via resource policies, or managing database credentials.
- Use Parameter Store (SecureString) for simpler configuration values or when cost is a primary concern and rotation is not needed or can be custom-built.

3. Other Related AWS Services and Concepts

- AWS KMS: The backbone of encryption for both Secrets Manager and Parameter Store. Understanding CMK policies, key grants, and envelope encryption is essential.
- IAM Policies: Control who can call GetSecretValue, PutSecretValue, DescribeSecret, and other APIs. The principle of least privilege is critical.
- AWS CloudTrail: Logs all Secrets Manager and Parameter Store API calls for audit and compliance.
- Amazon EventBridge (CloudWatch Events): Can trigger on Secrets Manager events such as rotation success/failure, secret creation, and deletion.
- AWS Config: Can be used to check compliance rules, such as ensuring secrets are rotated within a specified period.
- VPC Endpoints (PrivateLink): Ensure that secrets retrieval traffic stays within the AWS network and never traverses the public internet.
- AWS Lambda: Used for secret rotation functions in Secrets Manager. The Lambda function must have network access to both Secrets Manager and the target resource (e.g., RDS in a VPC).
- EC2 Instance Metadata / IMDS: While not a secrets management tool, it is important to understand that temporary credentials from IAM roles attached to EC2 instances are obtained via the instance metadata service. IMDSv2 (requiring session tokens) should be enforced to prevent SSRF attacks from stealing these temporary credentials.

Best Practices for Secrets Management in AWS

1. Never hardcode secrets in application code, environment variables, or configuration files checked into source control.
2. Use IAM roles instead of long-term access keys wherever possible.
3. Enable automatic rotation and set the shortest practical rotation interval.
4. Use VPC endpoints to access Secrets Manager and Parameter Store without traversing the internet.
5. Apply least privilege IAM policies—only grant access to specific secrets, not all secrets.
6. Use customer-managed CMKs when you need fine-grained control over who can decrypt secrets (via KMS key policy).
7. Monitor with CloudTrail and EventBridge—set up alarms for unusual secret access patterns or rotation failures.
8. Use resource-based policies on secrets for cross-account scenarios rather than sharing credentials.
9. Tag secrets and use tag-based access control (ABAC) for scalable permission management.
10. Implement multi-user rotation strategies (alternating users) for databases to avoid downtime during rotation.

Secrets Manager Rotation Strategies

- Single-user rotation: The same database user's password is changed. There is a brief period during rotation where the old password may not work. Suitable for non-critical workloads.
- Alternating-user (multi-user) rotation: Two database users are used alternately. While one is being rotated, the other continues to serve requests. This ensures zero downtime and is the recommended strategy for production workloads.

Common Exam Scenarios

1. An application on EC2 needs database credentials that rotate every 30 days. → Use AWS Secrets Manager with automatic rotation configured with a 30-day interval. The EC2 instance role should have permission to call GetSecretValue, and the rotation Lambda needs VPC access to the database.

2. A developer committed API keys to a Git repository. → Immediately rotate the compromised key, remove it from the repository history, store the new key in Secrets Manager, and update the application to retrieve it programmatically.

3. Cross-account access to a secret is needed. → Use a resource-based policy on the Secrets Manager secret to grant access to the other account, along with a KMS key policy that allows the other account to decrypt.

4. An application in a private subnet needs to access Secrets Manager without internet access. → Create a VPC endpoint (interface endpoint via PrivateLink) for Secrets Manager in the VPC. Ensure the security group on the endpoint allows inbound HTTPS (port 443) from the application's security group.

5. Choosing between Secrets Manager and Parameter Store. → If the question mentions automatic rotation, cross-account sharing, or database credentials, the answer is almost always Secrets Manager. If the question emphasizes cost savings, simple configuration storage, or hierarchical parameters, Parameter Store may be appropriate.

---

Exam Tips: Answering Questions on Secrets Management

1. Secrets Manager vs. Parameter Store: When a question asks about automatic rotation, the answer is AWS Secrets Manager. Parameter Store does NOT have built-in automatic rotation. This is one of the most commonly tested distinctions.

2. Know the rotation architecture: Understand that rotation uses a Lambda function, and that Lambda function needs network connectivity to both Secrets Manager (via VPC endpoint or NAT Gateway) AND the target service (e.g., RDS). If either connectivity path is broken, rotation fails.

3. Cross-account access requires TWO policies: A resource-based policy on the secret AND a KMS key policy allowing the external account to use the encryption key. If either is missing, access is denied. This is a frequent exam trap.

4. KMS integration is always relevant: If a question mentions encryption of secrets, remember that Secrets Manager always encrypts secrets with KMS. If the question asks about controlling decryption access, the answer involves KMS key policies and grants.

5. Look for the words "automatic" and "rotate": These are strong signals pointing to Secrets Manager. If the question says "with minimal operational overhead," it reinforces choosing the managed rotation capability.

6. VPC Endpoints for security: Any question about keeping traffic private or avoiding internet exposure when accessing secrets should point you toward VPC endpoints (AWS PrivateLink).

7. Eliminate hardcoding answers: If any answer choice involves storing credentials in environment variables, EC2 user data, code, or AMIs, it is almost certainly wrong. The correct answer will involve a managed secrets service.

8. CloudTrail for auditing: If the question asks how to audit who accessed a secret, the answer is CloudTrail. If it asks about monitoring rotation failures, the answer involves CloudTrail plus EventBridge (CloudWatch Events) with SNS notifications.

9. Alternating-user strategy for zero downtime: If a question mentions rotation without application downtime for database credentials, the multi-user / alternating-user rotation strategy is the answer.

10. Understand version stages: AWSCURRENT is the active version, AWSPREVIOUS is the last rotated version, and AWSPENDING is the version being created during rotation. If an application is getting old credentials after rotation, check that it is retrieving AWSCURRENT and not caching the secret.

11. Deletion and recovery: Secrets Manager enforces a minimum 7-day recovery window (up to 30 days) before a secret is permanently deleted. You cannot delete a secret immediately unless you use ForceDeleteWithoutRecovery. Exam questions may test this recovery period.

12. Replica secrets: Secrets Manager supports replicating secrets to multiple AWS regions for disaster recovery. If a question involves multi-region architectures and secrets availability, this feature is relevant.

By mastering these concepts and recognizing the key patterns in exam questions, you will be well-prepared to handle any Secrets Management question on the AWS Security Specialty exam.