Amazon DynamoDB for NoSQL Data Storage

Amazon DynamoDB for NoSQL Data Storage – Complete Guide for AWS Data Engineer Associate

Why Is Amazon DynamoDB Important?

Amazon DynamoDB is one of the most critical services tested on the AWS Data Engineer Associate exam. As a fully managed NoSQL database service, DynamoDB is at the heart of many modern data architectures on AWS. It provides single-digit millisecond performance at any scale, making it essential for real-time data pipelines, IoT ingestion, session management, and event-driven architectures. Understanding DynamoDB is important because:

• It is a foundational building block for serverless and event-driven data engineering solutions.
• It integrates seamlessly with other AWS services like Lambda, Kinesis, S3, Glue, and Athena.
• Many exam scenarios revolve around choosing the right data store, and DynamoDB is frequently the correct answer for low-latency, high-throughput NoSQL workloads.
• Data engineers must understand how to design, optimize, and manage DynamoDB tables for cost-efficiency and performance.

What Is Amazon DynamoDB?

Amazon DynamoDB is a fully managed, serverless, key-value and document NoSQL database service provided by AWS. It is designed for applications that need consistent, single-digit millisecond latency at any scale. Key characteristics include:

• Fully Managed: AWS handles provisioning, patching, scaling, replication, and backups.
• Serverless: No servers to manage. You interact with tables, items, and attributes.
• Flexible Data Model: Supports key-value and document data models. Each item (row) can have a different set of attributes (columns), except for the primary key.
• Highly Available: Data is automatically replicated across three Availability Zones (AZs) within a region.
• Scalable: Can handle more than 10 trillion requests per day and support peaks of more than 20 million requests per second.

Core Concepts:

• Table: A collection of items (similar to a table in a relational database).
• Item: A single data record in a table (similar to a row). Maximum item size is 400 KB.
• Attribute: A fundamental data element (similar to a column). Attributes can be scalar, nested, or multi-valued (sets).
• Primary Key: Uniquely identifies each item. Two types:
  - Partition Key (Simple Primary Key): A single attribute used as the hash key. DynamoDB uses an internal hash function to determine the partition where data is stored.
  - Composite Primary Key (Partition Key + Sort Key): Two attributes together uniquely identify an item. The partition key determines the partition, and the sort key determines the order within that partition.

How DynamoDB Works

1. Data Distribution and Partitioning

DynamoDB stores data across multiple partitions. Each partition is an allocation of storage backed by SSDs. The partition key value is hashed to determine which partition stores the item. A well-designed partition key distributes data evenly across partitions to avoid hot partitions (partitions that receive disproportionate traffic).

2. Capacity Modes

DynamoDB offers two capacity modes for read/write throughput:

• Provisioned Capacity Mode: You specify the number of reads and writes per second (RCUs and WCUs). Suitable for predictable workloads. You can use Auto Scaling to dynamically adjust capacity. Reserved Capacity is available for cost savings on long-term commitments.
  - 1 RCU = 1 strongly consistent read per second (up to 4 KB) or 2 eventually consistent reads per second (up to 4 KB).
  - 1 WCU = 1 write per second (up to 1 KB).

• On-Demand Capacity Mode: DynamoDB automatically scales to accommodate workloads. You pay per request. Ideal for unpredictable or spiky workloads. No capacity planning required.

3. Secondary Indexes

Indexes allow you to query data using alternative keys beyond the primary key:

• Global Secondary Index (GSI): An index with a partition key and optional sort key that can be different from the base table's primary key. GSIs have their own provisioned throughput (separate RCUs/WCUs). You can create up to 20 GSIs per table. GSIs support eventual consistency only.

• Local Secondary Index (LSI): An index that has the same partition key as the base table but a different sort key. Must be created at table creation time. You can create up to 5 LSIs per table. LSIs support both strongly consistent and eventually consistent reads. LSIs share the base table's throughput capacity.

4. Consistency Models

• Eventually Consistent Reads (Default): The response might not reflect the results of a recently completed write. Consistency is usually reached within one second.
• Strongly Consistent Reads: Returns the most up-to-date data. Consumes twice the RCUs of eventually consistent reads.

5. DynamoDB Streams

DynamoDB Streams captures a time-ordered sequence of item-level modifications in a table and stores this information in a log for up to 24 hours. Stream records can include:
• KEYS_ONLY – Only the key attributes of the modified item.
• NEW_IMAGE – The entire item as it appears after modification.
• OLD_IMAGE – The entire item as it appeared before modification.
• NEW_AND_OLD_IMAGES – Both the new and old images of the item.

DynamoDB Streams integrates with AWS Lambda to create event-driven architectures (triggers). This is commonly used for replication, analytics, and real-time processing.

6. DynamoDB Accelerator (DAX)

DAX is a fully managed, in-memory cache for DynamoDB. It delivers up to a 10x performance improvement (microsecond response times). DAX is ideal for read-heavy and latency-sensitive workloads. DAX is a write-through cache: writes go through DAX to DynamoDB. DAX is compatible with existing DynamoDB API calls with minimal code changes.

7. Global Tables

DynamoDB Global Tables provide a fully managed, multi-region, multi-active replication solution. Data is automatically replicated across selected AWS Regions. This enables low-latency access for globally distributed applications and provides disaster recovery capabilities. Global Tables require DynamoDB Streams to be enabled.

8. Time to Live (TTL)

TTL allows you to define a per-item expiration timestamp. DynamoDB automatically deletes expired items within 48 hours at no additional cost. This is useful for session data, temporary logs, and cache entries. TTL deletions are captured by DynamoDB Streams.

9. Backup and Restore

• On-Demand Backup: Create full backups of tables at any time with no impact on performance.
• Point-in-Time Recovery (PITR): Continuous backups that allow you to restore a table to any point in time within the last 35 days (with per-second granularity).

10. Integration with AWS Data Services

• AWS Glue: Can crawl DynamoDB tables to create a schema in the Glue Data Catalog, enabling queries via Athena or ETL jobs.
• Amazon Kinesis Data Streams: DynamoDB can stream change data to Kinesis Data Streams for advanced analytics and real-time processing.
• Amazon S3: You can export DynamoDB table data to S3 (using Export to S3 feature) or import data from S3 (using Import from S3). Exports do not consume RCUs and require PITR to be enabled.
• Amazon Athena: Query DynamoDB data exported to S3 or via federated queries using the Athena DynamoDB connector.
• AWS Lambda: Commonly triggered by DynamoDB Streams for event-driven processing.
• Amazon EMR: Can read from and write to DynamoDB using Apache Hive.

11. Security

• Encryption at rest is enabled by default using AWS-owned keys, with options for AWS managed keys (aws/dynamodb) or customer managed KMS keys.
• Encryption in transit via TLS/HTTPS.
• Fine-grained access control using IAM policies and conditions.
• VPC Endpoints (Gateway type) for private access without traversing the internet.

12. PartiQL

DynamoDB supports PartiQL, a SQL-compatible query language. This allows you to use familiar SQL-like syntax (SELECT, INSERT, UPDATE, DELETE) to interact with DynamoDB, making it easier for teams familiar with SQL.

13. Transactions

DynamoDB supports ACID transactions across multiple items and tables. TransactWriteItems and TransactGetItems API calls allow you to group multiple actions into a single, all-or-nothing operation. Transactions consume twice the WCUs/RCUs compared to standard operations.


How to Answer Exam Questions on Amazon DynamoDB for NoSQL Data Storage

When you encounter DynamoDB questions on the AWS Data Engineer Associate exam, follow this structured approach:

Step 1: Identify the Workload Characteristics
• Is the workload requiring low-latency, high-throughput access? → DynamoDB is likely the answer.
• Is the data model flexible or schema-less? → DynamoDB supports flexible schemas.
• Does the question mention key-value or document storage? → DynamoDB.
• Is the workload transactional with complex joins? → Consider RDS or Aurora instead.

Step 2: Determine the Access Patterns
• If the question asks about querying by non-primary-key attributes → Think GSI or LSI.
• If the question mentions querying data across multiple regions with low latency → Think Global Tables.
• If the question describes read-heavy workloads needing microsecond latency → Think DAX.

Step 3: Evaluate Cost and Capacity
• Predictable, steady workloads → Provisioned capacity with Auto Scaling.
• Unpredictable or spiky workloads → On-Demand capacity mode.
• Need to reduce costs on known steady-state usage → Reserved Capacity.

Step 4: Consider Data Lifecycle and Integration
• Automatic deletion of expired data → TTL.
• Real-time event processing from table changes → DynamoDB Streams + Lambda.
• Analytics on DynamoDB data → Export to S3 + Athena/Glue, or Kinesis Data Streams integration.
• Change Data Capture (CDC) → DynamoDB Streams or Kinesis Data Streams for DynamoDB.


Exam Tips: Answering Questions on Amazon DynamoDB for NoSQL Data Storage

• Tip 1 – Partition Key Design: Always look for answers that promote even data distribution. A high-cardinality partition key prevents hot partitions and throttling. If the question describes uneven access patterns, the answer likely involves redesigning the partition key or using write sharding.

• Tip 2 – GSI vs. LSI: Remember that LSIs must be created at table creation time and share the table's throughput. GSIs can be created at any time and have their own throughput. If the question mentions adding a new access pattern to an existing table, GSI is the answer. If strong consistency on secondary queries is required, LSI is needed.

• Tip 3 – DAX vs. ElastiCache: DAX is purpose-built for DynamoDB and requires minimal code changes. ElastiCache is more general-purpose. If the scenario specifically involves caching DynamoDB reads, DAX is the preferred choice.

• Tip 4 – Item Size Limit: DynamoDB items are limited to 400 KB. If the question involves storing large objects (images, videos, large documents), the best practice is to store the object in S3 and keep a reference (S3 URL) in DynamoDB.

• Tip 5 – DynamoDB Streams vs. Kinesis Data Streams: DynamoDB Streams has a 24-hour retention and is simpler for Lambda triggers. Kinesis Data Streams for DynamoDB offers longer retention (up to 365 days), more consumers, and integration with Kinesis services. If the scenario requires longer retention or multiple consumers, Kinesis Data Streams is preferred.

• Tip 6 – Export to S3: The DynamoDB Export to S3 feature does NOT consume read capacity from the table. It requires Point-in-Time Recovery (PITR) to be enabled. This is ideal for analytics workloads without impacting production performance.

• Tip 7 – Global Tables: If the question mentions multi-region, active-active replication, disaster recovery across regions, or low-latency global access, Global Tables is the answer. Remember: Global Tables require DynamoDB Streams.

• Tip 8 – Transactions: DynamoDB transactions consume 2x the normal WCUs and RCUs. If the question involves atomicity across multiple items, TransactWriteItems/TransactGetItems is the answer.

• Tip 9 – TTL for Cost Optimization: TTL deletes expired items at no extra cost and does not consume WCUs. This is a common answer for questions about managing data lifecycle and reducing storage costs automatically.

• Tip 10 – Consistency: Default reads are eventually consistent. If the question requires the latest data, strongly consistent reads are needed (but they cost 2x RCUs). GSIs only support eventually consistent reads.

• Tip 11 – Backup Strategy: PITR provides continuous backup with 35-day retention and per-second granularity restore. On-demand backups are for long-term archival. Both have zero performance impact on the table.

• Tip 12 – WCU/RCU Calculations: Be prepared to calculate throughput. For example: Writing 10 items per second, each 2.5 KB → rounds up to 3 KB per item → 3 WCUs per item × 10 = 30 WCUs. Reading 20 items per second, each 6 KB, strongly consistent → rounds up to 8 KB → 2 RCUs per item × 20 = 40 RCUs. For eventually consistent reads, divide by 2 → 20 RCUs.

• Tip 13 – Conditional Writes and Optimistic Locking: DynamoDB supports conditional writes to ensure data integrity. If a question asks about preventing overwrites or implementing optimistic concurrency control, conditional expressions are the answer.

• Tip 14 – Scan vs. Query: Query is efficient and targets specific partition key values. Scan reads every item in the table and is expensive. If a question mentions performance optimization, look for answers that replace Scans with Queries using appropriate indexes.

• Tip 15 – When NOT to Choose DynamoDB: If the scenario requires complex SQL joins, relational data modeling, or OLAP-style analytics directly on the database, DynamoDB is NOT the right choice. Consider Amazon RDS, Aurora, or Redshift instead.

By mastering these concepts and tips, you will be well-equipped to handle any DynamoDB-related question on the AWS Data Engineer Associate exam with confidence.