Amazon EMR Architecture

Correct Answer: AWS Glue Data Catalog

AWS Glue Data Catalog provides a centralized, serverless, and highly available metadata repository that is Hive metastore-compatible and can be shared across multiple EMR clusters. By configuring EMR to use the Glue Data Catalog as the Hive metastore, you avoid recreating schemas and table definitions on each cluster, reducing operational overhead and costs. Amazon S3 stores data and files but does not provide a Hive metastore or schema registry. Amazon Kinesis Data Streams is for real-time data streaming, not metadata management. AWS Elastic Beanstalk is an application deployment service and is unrelated to EMR metadata management.

Correct Answer: Enable autoscaling on the EMR cluster.

Enabling EMR autoscaling (EMR Managed Scaling or instance-group auto scaling) lets the cluster automatically add capacity during peaks and scale down during idle periods based on workload and resource utilization. This helps ensure sufficient compute is available when needed so jobs complete within their SLAs, while minimizing cost and manual intervention. Using instance fleets with weighted capacities improves capacity flexibility and cost efficiency but does not by itself scale the cluster in response to demand. Creating separate clusters violates the single-cluster constraint and increases operational overhead. Simply choosing larger core node types does not adapt to workload variability and can increase fixed costs without ensuring timely completion under peak load.

Correct Answer: Use the EMR S3DistCp utility to parallelize transfers and consolidate many small files into larger objects, reducing S3 request overhead and improving both read and write throughput.

S3 latency often rises during peak load when jobs read and write large numbers of small objects, driving up the per-object request rate. EMR’s S3DistCp parallelizes S3 transfers, uses multipart operations, and can group many small files into fewer, larger objects. This reduces the number of GET/PUT requests, improves throughput per object, and lowers overall read/write latency without changing the architecture. Adjusting reducer counts primarily affects compute/shuffle stages, not S3 I/O latency. S3 Select can cut read bytes for supported formats but does not address write latency and is not a general fix for pipeline I/O. Kinesis Data Streams is a different streaming ingestion service and does not solve S3 read/write latency for an EMR batch pipeline.