Ingesting and Processing Data

Planning, building, and deploying data pipelines for both batch and streaming workloads using Google Cloud services like Dataflow, Dataproc, and Pub/Sub.

5 minutes 5 Questions

Ingesting and processing data in Google Cloud involves collecting raw data from various sources and transforming it into meaningful, usable formats for analytics and decision-making. **Data Ingestion** refers to bringing data into Google Cloud from diverse sources such as on-premises databases, st…

Concepts covered
Data Transformation and Orchestration LogicData Encryption in PipelinesDataflow and Apache Beam for Data ProcessingCloud Data Fusion for Data IntegrationStreaming Processing and Windowing StrategiesAI-Based Data EnrichmentPub/Sub for Messaging and Event StreamingCI/CD for Data PipelinesDefining Data Sources and SinksNetworking Fundamentals for Data PipelinesData Cleansing TechniquesDataproc and Apache Spark for Data ProcessingBatch Processing TransformationsLate-Arriving Data HandlingData Acquisition and Import StrategiesJob Automation with Cloud Composer and Workflows
Test mode:
Start practice test

GCP Data Engineer - Ingesting and Processing Data Example Questions

Test your knowledge of Ingesting and Processing Data

Question 1

In Apache Airflow, what does the term 'task dependency' define?

Correct Answer: The execution order between tasks, ensuring one task completes before a downstream task begins

In Apache Airflow, task dependency defines the execution order between tasks in a DAG (Directed Acyclic Graph), ensuring that upstream tasks complete successfully before their downstream tasks begin execution, using operators like >> or set_downstream().

Data format requirements are not what dependencies define; they relate to data validation logic within individual tasks. Resource allocation strategies concern the executor and worker configuration, not task relationships. Scheduling intervals are properties of the DAG itself (via schedule_interval), not individual task dependencies.

The key concept tested is understanding Airflow's DAG-based task orchestration model, where dependencies establish directed edges between tasks to control execution flow.

Question 2

What does Customer-Managed Encryption Keys (CMEK) allow users to control when encrypting data in Google Cloud pipelines?

Correct Answer: The lifecycle and access policies of the encryption keys used to protect data

Customer-Managed Encryption Keys (CMEK) allow users to create, manage, and control the lifecycle and access policies of their own encryption keys through Cloud KMS, giving them authority over who can use the keys, when keys are rotated, and when keys are disabled or destroyed.

CMEK does not let users choose the encryption algorithm or cipher suite, as Google Cloud services use standard algorithms like AES-256. Users do not control the physical location of hardware security modules for Google-managed keys. CMEK replaces Google-managed default keys with customer-controlled keys, so managing Google-managed key rotation schedules is not applicable.

The key concept tested is understanding the scope of control CMEK provides over encryption key management in Google Cloud.

Question 3

A data engineering team is building an Apache Beam streaming pipeline that aggregates sales data. They need to write strict unit test cases to ensure the application correctly manages data elements arriving after the allowed lateness window expires. Which approach allows the engineers to explicitly simulate this specific event insertion and deterministic time progression logic within their local test environment?

Correct Answer: Construct a TestStream object to programmatically advance the system watermark past the allowed lateness timeframe while sending delayed mock events.

The correct approach uses the TestStream class because it is explicitly designed for testing Apache Beam streaming pipelines by allowing developers to precisely manipulate event timestamps, processing time, and watermarks to deterministically simulate late data.
Evaluating processing time against mock data via PAssert only verifies output rather than controlling the inputs and watermarks. Establishing a MockClock is not the standard Beam API for manipulating streaming boundaries. Implementing a DoFnTester isolates a single transformation rather than testing pipeline level windowing or allowed lateness triggers.
The key concept tested is deterministic unit testing of Apache Beam streaming pipelines.

More Ingesting and Processing Data questions
669 questions (total)
Practice 100 questions
One session at a time, always new questions