Data Validation Techniques for GCP Professional Data Engineer
Data Validation Techniques
Why Data Validation Is Important
Data validation is a critical component of any data processing system. Without proper validation, organizations risk making decisions based on incorrect, incomplete, or corrupted data. In Google Cloud Platform (GCP) environments, data flows through complex pipelines involving ingestion, transformation, storage, and analysis. At each stage, data can be introduced with errors, schema mismatches, missing values, duplicates, or format inconsistencies. Data validation ensures:
- Data Quality: Guarantees that downstream consumers (analytics, ML models, dashboards) receive accurate and reliable data.
- Pipeline Reliability: Prevents pipeline failures caused by unexpected data formats or values.
- Regulatory Compliance: Ensures data meets standards required by regulations such as GDPR, HIPAA, or PCI-DSS.
- Cost Efficiency: Catching errors early prevents costly reprocessing and debugging later in the pipeline.
- Trust: Stakeholders and data consumers can trust the data they use for decision-making.
What Is Data Validation?
Data validation is the process of checking and verifying that data meets predefined rules, constraints, formats, and quality standards before it is processed, stored, or consumed. It encompasses several categories:
1. Schema Validation: Ensuring data conforms to an expected schema (column names, data types, required fields). For example, verifying that a record in BigQuery has the correct number of columns with the expected types (STRING, INT64, TIMESTAMP, etc.).
2. Format Validation: Checking that values adhere to expected formats, such as email addresses matching a regex pattern, dates in ISO 8601 format, or phone numbers in E.164 format.
3. Range and Constraint Validation: Verifying that numeric values fall within acceptable ranges (e.g., age between 0 and 150), string lengths are within limits, or values belong to an allowed set of enumerations.
4. Completeness Validation: Ensuring that required fields are not null or empty and that all expected records are present (e.g., row count checks).
5. Uniqueness Validation: Checking for duplicate records or ensuring primary key uniqueness.
6. Referential Integrity Validation: Verifying that foreign key relationships hold between datasets (e.g., every order references a valid customer ID).
7. Cross-Dataset Validation: Comparing data across source and destination to ensure consistency, such as row counts, aggregate sums, or hash comparisons.
8. Statistical Validation: Using statistical measures (mean, standard deviation, distribution checks) to detect anomalies or data drift.
9. Business Rule Validation: Applying domain-specific logic, such as ensuring an order total equals the sum of line items, or that a transaction date is not in the future.
How Data Validation Works on GCP
GCP provides several tools and services that support data validation at different stages of a data pipeline:
1. Cloud Dataflow (Apache Beam)
Apache Beam pipelines running on Dataflow can implement inline validation. You can use ParDo transforms to validate each record, routing valid records to the main output and invalid records to a dead-letter queue (side output). This pattern allows pipelines to continue processing valid data while capturing problematic records for later review.
Example pattern:
- Read data from Pub/Sub or Cloud Storage
- Apply a ParDo that checks schema, format, and business rules
- Valid records proceed to BigQuery or another sink
- Invalid records are written to a separate BigQuery table or Cloud Storage bucket for analysis
2. Cloud Data Fusion
Data Fusion provides a visual interface with built-in validation plugins. You can add Wrangler steps to validate, clean, and transform data. Validation directives can check for null values, data types, patterns, and ranges. Invalid records can be routed to error datasets.
3. BigQuery
BigQuery supports validation through:
- SQL assertions: Writing SQL queries that check data quality conditions (e.g., SELECT COUNT(*) FROM table WHERE column IS NULL).
- Scheduled queries: Automating validation checks on a regular basis.
- BigQuery Data Quality (Dataplex): Google's Dataplex service includes data quality rules that can be applied to BigQuery tables. You define expectations (rules) and Dataplex automatically scans data and reports on quality scores.
- Column-level constraints: BigQuery supports NOT NULL constraints and can enforce primary key and foreign key constraints (as of recent updates, for validation/optimization purposes).
4. Dataplex Data Quality
Dataplex provides an integrated data quality framework:
- Define data quality rules using YAML or through the console
- Rules can include null checks, range checks, regex matching, set membership, uniqueness, row-level SQL expressions, and aggregate SQL expressions
- Dataplex runs automated scans and produces quality scores and reports
- Integration with Cloud Logging and Cloud Monitoring for alerting on quality issues
5. Dataform
Dataform (integrated into BigQuery) allows you to define assertions on your SQL-based transformations. Assertions are SQL queries that should return zero rows if data is valid. For example, asserting that no rows have a negative revenue value. If assertions fail, the pipeline can be halted or alerts triggered.
6. Cloud Pub/Sub with Schema Registry
Pub/Sub supports schema validation at the topic level. You can associate an Avro or Protocol Buffer schema with a topic, and Pub/Sub will reject messages that do not conform to the schema. This provides early-stage validation at the ingestion layer.
7. Cloud Composer (Apache Airflow)
Cloud Composer orchestrates pipelines and can include validation tasks as DAG nodes. Common patterns include:
- Running SQL-based validation queries after a load step
- Using sensors to check for data completeness before proceeding
- Branching logic to handle validation failures (retry, alert, or route to error handling)
8. Great Expectations and dbt Tests
Open-source tools like Great Expectations can run on GCP infrastructure (Dataproc, Compute Engine, or Cloud Functions) to perform comprehensive data validation. dbt (data build tool) also supports tests that validate data models in BigQuery.
9. Dataproc (Apache Spark)
When using Spark on Dataproc, you can implement validation using DataFrame operations, such as checking for nulls, applying schema enforcement with StructType, filtering invalid records, and using libraries like Deequ (an open-source data quality library for Spark).
Common Data Validation Patterns
- Dead-Letter Queue Pattern: Invalid records are routed to a separate queue or storage for later investigation, while valid records continue processing. This is extremely common in streaming pipelines with Dataflow and Pub/Sub.
- Pre-Load Validation: Data is validated before being loaded into the target system. For example, checking file format and schema before loading into BigQuery.
- Post-Load Validation: After data is loaded, validation queries run to check row counts, null percentages, duplicate counts, and aggregate consistency.
- Reconciliation: Comparing source and destination data to ensure completeness and accuracy. This might involve comparing row counts, checksums, or aggregate values between the source system and BigQuery.
- Canary Checks: Running lightweight checks on a sample of data before processing the full dataset.
- Schema Evolution Handling: Using tools that support schema evolution (like Avro with Pub/Sub schema registry or BigQuery schema auto-detection) while still validating that changes are expected.
Handling Validation Failures
A well-designed system must define what happens when validation fails:
- Reject and Alert: Reject invalid data and send alerts via Cloud Monitoring or Cloud Alerting.
- Quarantine: Move invalid data to a quarantine zone (dead-letter table or bucket) for manual review.
- Retry: In cases of transient issues, retry the validation after a delay.
- Default/Impute: Replace invalid values with defaults or imputed values (use with caution).
- Halt Pipeline: Stop the pipeline entirely if critical validation fails, preventing bad data from propagating.
Exam Tips: Answering Questions on Data Validation Techniques
1. Understand the Dead-Letter Queue Pattern: This is one of the most frequently tested concepts. Know that in Dataflow, you use side outputs to route invalid records, and in Pub/Sub, you can configure dead-letter topics for undeliverable messages. If a question mentions handling invalid or malformed records in a streaming pipeline, dead-letter queues are almost always the answer.
2. Know When to Use Schema Validation at Ingestion: If a question describes data arriving from external sources with unpredictable schemas, look for answers involving Pub/Sub schema registry, Dataflow schema validation, or BigQuery schema enforcement. Pub/Sub schema validation rejects non-conforming messages at the topic level — this is the earliest possible validation point.
3. Dataplex for Data Quality at Scale: Questions about organization-wide data quality, automated scanning, or governance-driven validation typically point to Dataplex data quality features. Dataplex is the answer when the scenario involves managing quality across multiple datasets or lakes.
4. Dataform Assertions for SQL-Based Pipelines: When the question describes ELT pipelines in BigQuery with transformation logic in SQL, Dataform assertions are the appropriate validation mechanism.
5. Distinguish Between Pre-Load and Post-Load Validation: Pre-load validation catches issues before data enters the target system (saving storage and compute costs). Post-load validation provides a safety net after ingestion. The exam may present scenarios where one is more appropriate than the other. Pre-load is better for catching schema issues; post-load is better for aggregate consistency checks.
6. Reconciliation Is Key for Migration and ETL: For questions about data migration or ETL correctness, look for answers involving row count comparison, checksum validation, or aggregate value reconciliation between source and destination.
7. Read the Scenario Carefully: The exam often describes a specific problem (e.g., null values causing ML model failures, duplicate records in reports, schema changes breaking pipelines). Match the problem to the appropriate validation type:
- Null values → completeness validation (NOT NULL checks)
- Duplicates → uniqueness validation (deduplication)
- Schema changes → schema validation (schema registry, schema enforcement)
- Out-of-range values → range/constraint validation
- Format issues → format validation (regex, type casting)
8. Consider Cost and Performance: The exam may ask you to choose between validation approaches. Inline validation in Dataflow adds processing overhead but catches errors in real-time. Batch validation after loading is less resource-intensive but introduces a delay. Choose based on the requirements described in the question (real-time vs. batch, cost sensitivity, latency requirements).
9. Cloud Composer for Orchestrated Validation: When a question describes multi-step pipelines where validation must occur between steps (e.g., validate after extraction, before transformation), Cloud Composer is typically the orchestration tool that manages this workflow.
10. Think About Error Handling Holistically: The best answers usually include not just detection but also handling. If an answer choice includes both validation AND a clear error handling strategy (dead-letter queue, alerting, quarantine), it is likely the most complete and correct answer.
11. BigQuery Constraints: Know that BigQuery supports NOT NULL constraints and primary/foreign key constraints. However, BigQuery primary and foreign key constraints are not enforced on data ingestion by default — they are used for query optimization. If the exam asks about enforced uniqueness, you may need to implement it through validation logic rather than relying solely on BigQuery constraints.
12. Eliminate Distractors: Some answer choices may mention validation techniques that are valid in general but not appropriate for GCP or the specific scenario. For example, using a relational database trigger for validation when the scenario is about BigQuery would be incorrect. Always choose GCP-native solutions when available.
