Data Encryption and Key Management

Data Encryption & Key Management – GCP Professional Data Engineer Guide

Why Data Encryption & Key Management Matters

Data encryption and key management sit at the heart of every secure data processing system. For the GCP Professional Data Engineer exam, this topic is critical because Google Cloud Platform offers multiple layers of encryption and key management options, and the exam frequently tests your ability to choose the right approach for a given scenario. In real-world architectures, a failure in encryption strategy or poor key management can lead to data breaches, regulatory non-compliance, and loss of customer trust. Understanding how GCP handles encryption by default—and how you can extend or customize it—is essential for both the exam and professional practice.

What Is Data Encryption?

Data encryption is the process of converting plaintext data into an unreadable format (ciphertext) using a cryptographic algorithm and a key. Only entities with access to the correct decryption key can revert the data to its original form. Encryption protects data in three states:

• Data at rest – Data stored on disk, in databases, or in object storage (e.g., Cloud Storage, BigQuery, Cloud SQL).
• Data in transit – Data moving between services, between a client and a server, or across networks.
• Data in use – Data actively being processed in memory (addressed by Confidential Computing).

Google Cloud encrypts all data at rest by default using AES-256 (or AES-128) without any action required from the user. Data in transit within Google's network is also encrypted by default using protocols like TLS and ALTS (Application Layer Transport Security).

What Is Key Management?

Key management refers to the creation, storage, rotation, and destruction of cryptographic keys used to encrypt and decrypt data. Proper key management ensures that encryption remains effective over time and that keys are not exposed or misused.

GCP Encryption & Key Management Options

Google Cloud provides a layered key management hierarchy. Understanding these layers is vital:

1. Google Default Encryption (Google-managed encryption keys – GMEK)
• Google manages every aspect of key generation, storage, rotation, and destruction.
• Data at rest is encrypted with AES-256 by default.
• The encryption key hierarchy uses a Data Encryption Key (DEK) to encrypt data and a Key Encryption Key (KEK) to encrypt the DEK. The KEK is managed by Google's internal Key Management Service (KMS).
• No configuration is required from the user.
• Suitable when you have no regulatory requirement to manage your own keys.

2. Customer-Managed Encryption Keys (CMEK)
• You create and manage keys in Cloud KMS (Key Management Service).
• You control the key lifecycle: creation, rotation schedule, enabling/disabling, and destruction.
• GCP services (BigQuery, Cloud Storage, Dataflow, Pub/Sub, Compute Engine, etc.) integrate with Cloud KMS so that your CMEK is used instead of the Google-managed key.
• You can set IAM policies on keys to control who can use, manage, or administer them.
• Keys can be symmetric (AES-256) or asymmetric (RSA, Elliptic Curve) depending on the use case.
• Cloud KMS keys reside in Google-managed HSMs (for Cloud HSM key protection level) or in software.
• Key Rings organize keys; they are region-specific and cannot be deleted once created.
• CMEK is the recommended choice when compliance mandates require you to control key rotation and access but you still trust Google infrastructure to store the key material.

3. Customer-Supplied Encryption Keys (CSEK)
• You generate and supply your own encryption key with each API request.
• Google uses your key to encrypt/decrypt but does not store your key permanently (it is held in memory only during the operation).
• Supported primarily for Cloud Storage and Compute Engine persistent disks.
• If you lose the key, the data is irrecoverable.
• Provides maximum control but also maximum responsibility.
• Not supported by many managed services (e.g., BigQuery, Dataflow).

4. Cloud External Key Manager (Cloud EKM)
• Keys are stored and managed in a third-party, external key management system (e.g., Thales, Fortanix, Equinix).
• Cloud KMS references the external key via a URI.
• Data never leaves Google Cloud unencrypted, but the key material never enters Google Cloud.
• Provides a key access justification feature so you can see exactly why Google is requesting access to the key.
• Ideal for organizations with strict sovereignty or regulatory requirements that prohibit key material from being stored in any cloud provider.

5. Cloud HSM
• A Cloud KMS protection level that stores keys in FIPS 140-2 Level 3 validated Hardware Security Modules.
• Keys never leave the HSM boundary in plaintext.
• Used when compliance requires hardware-backed key protection (e.g., PCI-DSS, HIPAA, FedRAMP).

How the Encryption Key Hierarchy Works

Google's envelope encryption model works as follows:

1. A unique Data Encryption Key (DEK) is generated for each data chunk.
2. The DEK encrypts the data (using AES-256).
3. The DEK itself is then encrypted (wrapped) by a Key Encryption Key (KEK).
4. The wrapped DEK is stored alongside the encrypted data.
5. The KEK is stored in Cloud KMS (or an external system if EKM is used).
6. To decrypt, the KEK first unwraps the DEK, then the DEK decrypts the data.

This approach ensures that even if encrypted data is accessed, the DEK cannot be used without the KEK, and the KEK is protected by the key management system's access controls.

Key Rotation

• Automatic rotation: Cloud KMS supports configurable automatic key rotation (e.g., every 90 days). When a key is rotated, a new key version is created and becomes the primary version for new encryption. Old versions remain available for decrypting data encrypted with them.
• Manual rotation: You can manually rotate keys at any time.
• Re-encryption: After rotation, previously encrypted data still uses the old key version. For maximum security, you should re-encrypt data with the new key version, though this is not automatic in most services.
• Google-managed keys are rotated automatically and transparently.

IAM and Key Access Control

Cloud KMS integrates with Cloud IAM for fine-grained access control:

• roles/cloudkms.admin – Manage key rings and keys but cannot encrypt/decrypt.
• roles/cloudkms.cryptoKeyEncrypterDecrypter – Can encrypt and decrypt data using keys.
• roles/cloudkms.cryptoKeyEncrypter – Can only encrypt.
• roles/cloudkms.cryptoKeyDecrypter – Can only decrypt.
• Separation of duties: The person who manages keys should not be the same person who encrypts/decrypts data. This is a common exam scenario.

Encryption Across Key GCP Data Services

• BigQuery: Supports GMEK and CMEK. CSEK is not supported. You can set a default CMEK key at the dataset level. Temporary tables and query results can also be encrypted with CMEK.
• Cloud Storage: Supports GMEK, CMEK, and CSEK. You can set a default encryption key at the bucket level.
• Cloud SQL: Supports GMEK and CMEK. Backups are encrypted with the same key as the instance.
• Dataflow: Supports CMEK for pipeline state and temporary data. Specify the CMEK key when launching the pipeline.
• Pub/Sub: Supports CMEK for messages at rest.
• Dataproc: Supports CMEK for cluster disks and data.
• Cloud Spanner: Supports GMEK and CMEK.

Data in Transit Encryption

• Google encrypts all data in transit between its data centers using ALTS or TLS.
• Client-to-Google traffic uses TLS (HTTPS).
• You can enforce TLS minimum versions on load balancers and App Engine.
• For VPN connections, IPsec encryption is used (Cloud VPN).
• Dedicated Interconnect traffic is not encrypted by default at the link layer; you may need to add application-level encryption or use MACsec (available with some partners) or a VPN overlay.

Confidential Computing (Data in Use)

• Confidential VMs use AMD SEV (Secure Encrypted Virtualization) to encrypt data in memory.
• Confidential GKE Nodes extend this to Kubernetes workloads.
• Protects data while it is being processed, addressing the "data in use" encryption gap.

DLP and Tokenization

While not strictly key management, the Cloud Data Loss Prevention (DLP) API is often tested alongside encryption topics. It can:
• Identify and classify sensitive data (PII, PHI, PCI).
• De-identify data using tokenization, masking, pseudonymization, or format-preserving encryption.
• Integrate with Cloud KMS for wrapping/unwrapping tokenization keys.

Common Exam Scenarios

1. "A company needs to control its own encryption keys but does not want to manage key material outside Google Cloud." → CMEK with Cloud KMS.

2. "A company's security policy requires that encryption keys never reside in any cloud provider." → Cloud EKM.

3. "A company wants hardware-backed key storage for compliance." → Cloud HSM.

4. "A company supplies its own key with every API call and wants Google to not store the key." → CSEK.

5. "Who should have the ability to manage keys versus use keys?" → Separation of duties: Admin role manages keys, EncrypterDecrypter role uses them.

6. "Data needs to remain encrypted even during processing." → Confidential Computing.

7. "How to encrypt BigQuery data with custom keys?" → CMEK at the dataset or table level (CSEK not supported for BigQuery).

8. "Interconnect traffic must be encrypted." → Add VPN overlay on top of Dedicated Interconnect, or use application-level TLS.

---

Exam Tips: Answering Questions on Data Encryption and Key Management

1. Know the four key management options cold: GMEK, CMEK, CSEK, and EKM. Understand when each is appropriate. The exam loves scenario-based questions asking you to pick the right one based on control requirements and compliance needs.

2. Understand envelope encryption: Be ready to explain or recognize the DEK/KEK hierarchy. Questions may describe the mechanism and ask you to identify it.

3. Separation of duties is a favorite topic: If a question mentions that one team manages keys and another team uses them, the answer involves assigning different IAM roles (admin vs. encrypter/decrypter).

4. Remember which services support which key types: BigQuery does NOT support CSEK. Cloud Storage supports all three (GMEK, CMEK, CSEK). This is a common trap in exam questions.

5. Key rotation does not re-encrypt old data: New data uses the new key version; old data still uses the old version unless you explicitly re-encrypt. The exam may test whether you understand this nuance.

6. CSEK means total responsibility: If you lose a CSEK key, data is gone forever. The exam may present a scenario where data recovery is needed after losing a CSEK key—the answer is that recovery is impossible.

7. Default encryption is always on: If a question implies that data at rest in GCP is unencrypted by default, that is incorrect. All data at rest is encrypted by default with Google-managed keys.

8. Cloud EKM for sovereignty: When the question emphasizes that key material must not reside in any cloud, Cloud EKM is the answer, not CSEK (CSEK keys are transmitted to Google temporarily).

9. Cloud HSM for hardware compliance: Whenever FIPS 140-2 Level 3 or hardware-based key protection is mentioned, choose Cloud HSM as the protection level in Cloud KMS.

10. DLP + KMS integration: If the question involves de-identifying sensitive data with the ability to re-identify later, the answer typically involves Cloud DLP with a wrapped key stored in Cloud KMS (CryptoKey with DLP tokenization).

11. Read carefully for "least privilege" and "minimal operational overhead": If the scenario says the company wants minimal management, lean toward GMEK or CMEK (not CSEK). If they want maximum control with minimal cloud trust, lean toward EKM.

12. Interconnect encryption: Remember that Dedicated Interconnect is NOT encrypted at the link layer by default. If encryption in transit over Interconnect is required, the answer is to use a VPN tunnel over the Interconnect or application-layer TLS.

13. Eliminate obviously wrong answers first: In multi-choice questions, eliminate options that mention unsupported features (e.g., CSEK for BigQuery) or options that grant overly broad permissions.

14. Think in layers: The exam may combine encryption with other security measures (VPC Service Controls, IAM, audit logging). A complete answer often involves encryption plus access controls plus monitoring. Choose the answer that addresses the encryption requirement most directly while respecting the principle of least privilege.

By mastering these concepts and practicing scenario-based reasoning, you will be well-prepared to handle any data encryption and key management question on the GCP Professional Data Engineer exam.