Infrastructure as Code with CloudFormation, CDK, and SAM

Infrastructure as Code: CloudFormation, CDK & SAM – Complete Guide for AWS Data Engineer Associate

Infrastructure as Code (IaC) with CloudFormation, CDK, and SAM

Why Is Infrastructure as Code Important?

Infrastructure as Code (IaC) is a foundational practice in modern cloud engineering, and it is especially critical for data engineers. Here's why:

• Reproducibility: Data pipelines, analytics environments, and ETL workflows often involve dozens of interconnected AWS services (S3 buckets, Glue jobs, Kinesis streams, Redshift clusters, IAM roles, etc.). IaC ensures you can recreate these environments identically every time.
• Version Control: By defining infrastructure in code, you can track changes over time using Git, enabling rollbacks and auditability—essential for compliance in data-heavy organizations.
• Consistency Across Environments: IaC eliminates configuration drift between dev, staging, and production environments, reducing bugs caused by environmental differences in data pipelines.
• Automation & Speed: Manual provisioning is error-prone and slow. IaC enables CI/CD for infrastructure, allowing rapid deployment of data engineering resources.
• Cost Management: You can tear down and recreate entire environments on demand, avoiding unnecessary costs from idle resources.

---

What Is AWS CloudFormation?

AWS CloudFormation is the native IaC service from AWS. It allows you to define your cloud infrastructure using declarative templates written in JSON or YAML.

Key Concepts:
• Template: A JSON or YAML file that describes the desired state of your AWS resources. It contains sections like Parameters, Mappings, Resources (required), Outputs, Conditions, and Transform.
• Stack: A collection of AWS resources created and managed as a single unit from a CloudFormation template. When you create a stack, CloudFormation provisions all the resources defined in the template.
• StackSet: Enables you to deploy stacks across multiple AWS accounts and regions from a single template—useful for enterprise data platforms.
• Change Set: A preview of changes that will be applied to a stack before executing the update, helping you understand the impact of modifications.
• Drift Detection: CloudFormation can detect when the actual resource configuration differs from the template definition.
• Nested Stacks: Stacks that are created as part of other stacks, enabling modular and reusable template design.
• Intrinsic Functions: Built-in functions like !Ref, !Sub, !GetAtt, !Join, !Select, Fn::ImportValue that allow dynamic references within templates.
• Cross-Stack References: Using Outputs with Export and Fn::ImportValue to share values between stacks.

How CloudFormation Works:
1. You author a template (YAML/JSON) defining resources and their configurations.
2. You upload the template to CloudFormation (via Console, CLI, or API).
3. CloudFormation creates a stack and provisions resources in the correct order based on dependency analysis.
4. If creation fails, CloudFormation automatically rolls back all changes (by default).
5. To update, you submit a modified template or create a change set for review.
6. To delete, you delete the stack and CloudFormation removes all associated resources (unless deletion protection or retain policies are set).

Data Engineering Relevance:
CloudFormation templates commonly define AWS Glue databases, crawlers, and jobs; S3 buckets with lifecycle policies; Kinesis streams; Redshift clusters; Step Functions state machines; Lambda functions; IAM roles and policies; and Lake Formation permissions.

---

What Is the AWS Cloud Development Kit (CDK)?

The AWS CDK is an open-source framework that lets you define cloud infrastructure using familiar programming languages such as TypeScript, Python, Java, C#, and Go. Under the hood, CDK synthesizes your code into CloudFormation templates.

Key Concepts:
• App: The root of your CDK project; it contains one or more stacks.
• Stack: Equivalent to a CloudFormation stack—a deployable unit of resources.
• Construct: The basic building block of CDK. Constructs represent cloud resources and come in three levels:
  - L1 (CFN Resources): Direct CloudFormation mappings (e.g., CfnBucket). Low-level, 1:1 with CloudFormation resources.
  - L2 (Curated Constructs): Higher-level abstractions with sensible defaults and convenience methods (e.g., s3.Bucket). These are the most commonly used.
  - L3 (Patterns): Opinionated, multi-resource patterns that solve common use cases (e.g., aws-ecs-patterns.ApplicationLoadBalancedFargateService).
• cdk synth: Generates the CloudFormation template from your CDK code.
• cdk deploy: Deploys the synthesized template to AWS.
• cdk diff: Shows the differences between the deployed stack and the current code.
• cdk bootstrap: Sets up the environment (S3 bucket, IAM roles) needed for CDK deployments in a given account/region.

How CDK Works:
1. You write infrastructure code using constructs in your chosen language.
2. Run cdk synth to generate a CloudFormation template.
3. Run cdk deploy to deploy the template via CloudFormation.
4. CDK manages the CloudFormation stack lifecycle on your behalf.

Why CDK for Data Engineers?
• You can use loops, conditionals, and object-oriented programming to manage complex data pipelines.
• Libraries and reusable constructs make it easy to standardize data platform components.
• Type safety (in TypeScript/Java/C#) helps catch configuration errors at compile time rather than deployment time.
• CDK integrates well with CI/CD pipelines for automated data infrastructure deployment.

---

What Is the AWS Serverless Application Model (SAM)?

AWS SAM is an open-source framework specifically designed for building and deploying serverless applications. It extends CloudFormation with a simplified syntax for serverless resources.

Key Concepts:
• SAM Template: A YAML template that uses the Transform: AWS::Serverless-2016-10-31 header. It introduces simplified resource types like AWS::Serverless::Function, AWS::Serverless::Api, AWS::Serverless::SimpleTable, AWS::Serverless::LayerVersion, and AWS::Serverless::StateMachine.
• SAM CLI: A command-line tool that provides local testing, building, packaging, and deployment capabilities.
  - sam init: Initialize a new SAM project.
  - sam build: Build your serverless application.
  - sam local invoke: Locally invoke a Lambda function.
  - sam local start-api: Start a local API Gateway endpoint.
  - sam deploy: Deploy your application to AWS.
  - sam validate: Validate a SAM template.
• Under the Hood: SAM templates are transformed into full CloudFormation templates during deployment. The Transform macro expands SAM shorthand into standard CloudFormation resources.
• SAM Policy Templates: Pre-built IAM policy templates (e.g., S3ReadPolicy, DynamoDBCrudPolicy, KinesisStreamReadPolicy) that simplify permission management for serverless functions.

How SAM Works:
1. Define your serverless resources in a SAM template (YAML).
2. Run sam build to prepare your application code and dependencies.
3. Optionally test locally with sam local invoke or sam local start-api.
4. Run sam deploy --guided to package and deploy via CloudFormation.
5. SAM transforms the simplified template into a full CloudFormation template and creates/updates a stack.

Data Engineering Relevance:
SAM is particularly useful when your data ingestion or transformation pipelines rely on Lambda functions triggered by S3 events, Kinesis streams, DynamoDB streams, SQS queues, or scheduled EventBridge rules. It simplifies the definition of these event-driven, serverless data workflows.

---

Comparing CloudFormation, CDK, and SAM

CloudFormation:
- Language: JSON/YAML (declarative)
- Scope: All AWS resources
- Abstraction Level: Low-level
- Best For: Full control, broad resource coverage, enterprise governance

CDK:
- Language: TypeScript, Python, Java, C#, Go (imperative)
- Scope: All AWS resources (via CloudFormation)
- Abstraction Level: High-level with constructs
- Best For: Complex infrastructure, reusable patterns, developer-friendly workflows

SAM:
- Language: YAML (declarative, extended CloudFormation)
- Scope: Primarily serverless resources (with ability to include standard CloudFormation)
- Abstraction Level: Medium (simplified serverless syntax)
- Best For: Serverless applications, Lambda-based data processing, local testing

Key Relationship: Both CDK and SAM ultimately produce CloudFormation templates. CDK synthesizes code into CloudFormation. SAM templates are transformed into CloudFormation at deployment. You can even use CDK to generate SAM-compatible outputs.

---

Exam Tips: Answering Questions on Infrastructure as Code with CloudFormation, CDK, and SAM

1. Know What Runs Under the Hood: Both CDK and SAM use CloudFormation as their deployment engine. If a question asks about the underlying mechanism for provisioning resources in CDK or SAM, the answer is CloudFormation.

2. Template Sections Matter: Remember that the only required section in a CloudFormation template is Resources. Know the purpose of Parameters (input values), Mappings (static key-value lookups), Conditions (conditional resource creation), Outputs (return values), and Transform (for macros like SAM).

3. CDK Construct Levels: If a question mentions L1, L2, or L3 constructs, remember: L1 = raw CloudFormation (CfnXxx), L2 = curated with defaults, L3 = multi-resource patterns. Most developers use L2 constructs.

4. SAM = Serverless Focus: If the question is about deploying Lambda functions, API Gateway, DynamoDB tables, or Step Functions in a simplified way, SAM is likely the answer. Look for the Transform: AWS::Serverless-2016-10-31 indicator.

5. Local Testing = SAM CLI: If a question asks about locally testing Lambda functions or API Gateway endpoints before deployment, the answer is sam local invoke or sam local start-api. CDK does not have built-in local testing for Lambda.

6. Cross-Stack References: When a question involves sharing outputs between stacks (e.g., an S3 bucket ARN from one stack used in another), think Outputs with Export and Fn::ImportValue.

7. Change Sets for Safety: If the question is about previewing changes before updating a stack, the answer is Change Sets. For CDK, the equivalent command is cdk diff.

8. Rollback Behavior: By default, CloudFormation rolls back all changes if stack creation fails. Know that you can disable rollback for debugging purposes. For updates, failed changes roll back to the previous known good state.

9. DeletionPolicy and RetainPolicy: Questions about retaining resources (like S3 buckets or RDS databases) after stack deletion involve the DeletionPolicy attribute. Options include Delete, Retain, and Snapshot (for supported resources like RDS, EBS, Redshift).

10. Drift Detection: If asked how to check whether deployed resources match the template definition, the answer is CloudFormation Drift Detection.

11. Imperative vs. Declarative: CDK is imperative (you write procedural code). CloudFormation and SAM are declarative (you describe the desired end state). If a question emphasizes using programming constructs like loops and conditionals for infrastructure, CDK is the answer.

12. cdk bootstrap: Remember that before deploying CDK in a new account/region, you must run cdk bootstrap. This is a common exam detail.

13. Parameters vs. Mappings: Parameters allow user input at deployment time. Mappings are hardcoded lookup tables (e.g., mapping region to AMI ID). If the question asks about runtime input, choose Parameters. If it's about static lookups, choose Mappings.

14. Nested Stacks vs. Cross-Stack References: Nested stacks are for modular template design (parent creates children). Cross-stack references are for loosely coupled stacks sharing values. If stacks have different lifecycles, prefer cross-stack references.

15. Data Engineering Scenarios: When exam questions describe deploying Glue jobs, Kinesis streams, S3 event notifications triggering Lambda, or Redshift clusters as part of a repeatable pipeline, the correct approach is IaC. Choose CloudFormation for broad resource definitions, CDK if the scenario emphasizes programming or complex logic, and SAM if the scenario focuses on serverless data processing with Lambda.

16. Watch for Distractors: Terraform, Ansible, and other third-party tools are not AWS-native IaC services and are unlikely to be correct answers on the AWS Data Engineer Associate exam. Stick with CloudFormation, CDK, and SAM unless specifically noted otherwise.

17. SAM and CloudFormation Compatibility: SAM templates can include any standard CloudFormation resource alongside serverless resources. This means SAM is not limited to only serverless resources—it's an extension, not a replacement.

18. Packaging Artifacts: For Lambda deployment packages and nested templates, CloudFormation uses the aws cloudformation package command (or sam package) to upload artifacts to S3 and rewrite template references. This is a common operational detail tested in exams.

Summary: Master the relationships between CloudFormation, CDK, and SAM. Understand that CloudFormation is the foundation, CDK adds programming power, and SAM simplifies serverless. For the exam, focus on when to use each tool, their key commands, and how they support repeatable, automated deployment of data engineering resources on AWS.