Maintaining and Automating Data Workloads

Optimizing resources, automating workflows, monitoring processes, and ensuring fault tolerance for production data workloads on Google Cloud.

5 minutes 5 Questions

Maintaining and Automating Data Workloads is a critical domain for Google Cloud Professional Data Engineers, focusing on ensuring data pipelines run reliably, efficiently, and with minimal manual intervention. **Data Pipeline Maintenance** involves monitoring pipeline health using tools like Cloud…

Concepts covered
Interactive vs Batch Query JobsCost Optimization for Data WorkloadsResource Provisioning for Business-Critical ProcessesPersistent vs Job-Based Data ClustersJob Scheduling and Repeatable OrchestrationBigQuery Editions and Capacity ReservationsWorkload Management for Jobs and Compute CapacityFault Tolerance and Restart ManagementMulti-Region and Multi-Zone Data JobsData Replication and Failover StrategiesCloud Monitoring and Logging for Data ProcessesDAG Creation for Cloud ComposerTroubleshooting Errors, Billing Issues, and QuotasData Corruption and Missing Data Recovery
Test mode:
Start practice test

GCP Data Engineer - Maintaining and Automating Data Workloads Example Questions

Test your knowledge of Maintaining and Automating Data Workloads

Question 1

A team stores event data in BigQuery and runs ad-hoc queries using on-demand pricing. They notice that most queries use SELECT * even though dashboards only need 5 out of 40 columns. A colleague suggests switching to flat-rate pricing to control costs. What is the most cost-effective first step to reduce query expenses?

Correct Answer: Update queries to select only the 5 required columns instead of using SELECT *

Selecting only the 5 required columns instead of SELECT * is the most cost-effective first step because BigQuery on-demand pricing charges based on the amount of data scanned, and reducing from 40 columns to 5 dramatically cuts the bytes processed per query with zero infrastructure changes.

Switching to flat-rate pricing does not reduce costs proportionally and requires a committed reservation, which may not be justified before optimizing queries. Materialized views add complexity and are a secondary optimization. Partitioning and clustering help filter rows but do not address the core problem of scanning unnecessary columns.

Key concept tested: BigQuery columnar storage and on-demand pricing based on bytes scanned.

Question 2

What is the primary reason job-based (ephemeral) Dataproc clusters are deleted after a job completes?

Correct Answer: To eliminate costs for idle compute resources between job runs

Ephemeral Dataproc clusters are created on demand, run a specific job, and are immediately deleted afterward primarily to avoid paying for idle compute resources that sit unused between job runs, since Dataproc billing is per-second for the cluster VMs.

While fresh cluster state is a secondary benefit, it is not the primary driver behind the ephemeral pattern. Reducing data corruption risk is not a significant concern since output data is typically stored in Cloud Storage, not on cluster disks. Autoscaling policies operate on running clusters and have no relationship to cluster deletion and recreation.

Key concept tested: cost optimization through ephemeral cluster lifecycle management in Dataproc.

Question 3

A team wants to know when their Cloud Dataflow pipeline stops processing data. They need a simple way to receive email notifications if the pipeline goes down. Which Google Cloud service should they use to set up alerts based on pipeline metrics?

Correct Answer: Cloud Monitoring with alerting policies configured for Dataflow metrics

Cloud Monitoring with alerting policies is the correct choice because it natively integrates with Dataflow metrics and supports configuring notification channels such as email, allowing teams to receive alerts when pipeline metrics indicate the pipeline has stopped processing data.

Cloud Logging with log-based alerts monitors log entries rather than pipeline metrics directly, making it less suitable for detecting when a pipeline stops processing. Cloud Scheduler is a cron job service and does not monitor or alert on pipeline health. Cloud Monitoring with dashboards and logs-based metrics provides visualization but dashboards alone do not send email notifications without alerting policies.

Key concept tested: using Cloud Monitoring alerting policies for proactive pipeline health notifications.

More Maintaining and Automating Data Workloads questions
585 questions (total)
Practice 100 questions
One session at a time, always new questions