Pub/Sub for Messaging and Event Streaming

Pub/Sub for Messaging and Event Streaming – GCP Professional Data Engineer Guide

Why Pub/Sub for Messaging and Event Streaming Is Important

In modern data architectures, systems need to communicate asynchronously, handle massive volumes of real-time events, and decouple producers from consumers. Google Cloud Pub/Sub is a foundational service that enables all of this. For the GCP Professional Data Engineer exam, Pub/Sub is one of the most heavily tested topics because it sits at the heart of nearly every streaming and event-driven architecture on Google Cloud. Understanding how Pub/Sub works, when to use it, and how it integrates with other GCP services is essential for both the exam and real-world data engineering.

What Is Google Cloud Pub/Sub?

Google Cloud Pub/Sub is a fully managed, serverless, real-time messaging and event ingestion service. It follows the publish-subscribe messaging pattern, where:

• Publishers send messages to a topic.
• Subscribers receive messages from a subscription attached to that topic.

Key characteristics include:

• Global availability: Pub/Sub is a global service; topics and subscriptions are not bound to a single region (though you can use message storage policies to restrict data locality).
• At-least-once delivery: Pub/Sub guarantees that every message is delivered at least once to each subscription.
• Serverless and auto-scaling: There is no infrastructure to manage. Pub/Sub scales automatically from zero to millions of messages per second.
• Durability: Messages are persisted and replicated across multiple zones for reliability.
• Retention: Messages are retained for up to 7 days (configurable, with a default of 7 days). Acknowledged messages can also be retained if configured.

Core Concepts

1. Topics
A topic is a named resource to which publishers send messages. Think of it as a channel or a category for messages. A topic can have zero or more subscriptions.

2. Subscriptions
A subscription is a named resource representing the stream of messages from a single topic to be delivered to a subscriber. Each subscription receives a copy of every message published to its topic. There are several delivery types:

• Pull Subscription: The subscriber explicitly calls the Pub/Sub API to retrieve messages and then acknowledges them. This is the most common pattern and is used with Dataflow, custom applications, and batch processing.
• Push Subscription: Pub/Sub sends messages to a configured HTTP(S) endpoint (e.g., a Cloud Run service, App Engine, or Cloud Function). The endpoint must return a success status code to acknowledge the message.
• BigQuery Subscription: Messages are written directly to a BigQuery table without requiring any subscriber code. This is ideal for analytics pipelines.
• Cloud Storage Subscription: Messages are written directly to Cloud Storage as files (Avro, JSON, or text). This is useful for archiving or batch downstream processing.

3. Messages
A message consists of:
• Data: The payload (up to 10 MB per message).
• Attributes: Key-value pairs of metadata (optional, up to 256 attributes).
• Message ID: A unique identifier assigned by Pub/Sub.
• Publish time: The timestamp when the message was published.

4. Acknowledgment (Ack)
When a subscriber processes a message, it sends an acknowledgment back to Pub/Sub. If the message is not acknowledged within the acknowledgment deadline (configurable, default 10 seconds, max 600 seconds), Pub/Sub redelivers it. This ensures at-least-once delivery.

5. Dead-Letter Topics (DLT)
If a message fails to be processed after a configurable number of delivery attempts, it can be forwarded to a dead-letter topic for later inspection, debugging, or reprocessing.

How Pub/Sub Works – The Flow

1. A publisher application publishes a message to a topic.
2. Pub/Sub durably stores the message and replicates it.
3. Each subscription on that topic receives a copy of the message.
4. Depending on the subscription type (pull, push, BigQuery, or Cloud Storage), the message is delivered to the subscriber.
5. The subscriber processes the message and sends an acknowledgment.
6. Pub/Sub removes the acknowledged message from the subscription's backlog.

Ordering and Exactly-Once Delivery

• Message Ordering: By default, Pub/Sub does not guarantee ordering. However, you can enable message ordering by assigning an ordering key to messages. Messages with the same ordering key are delivered in the order they were published. This is critical for scenarios like change data capture (CDC) or event sourcing.
• Exactly-Once Delivery: Pub/Sub supports exactly-once delivery at the subscription level (for pull subscriptions). When enabled, Pub/Sub ensures that redelivered messages are deduplicated on the server side. Note: This is a subscription-level setting and requires the subscriber to use the exactly-once delivery feature.

Pub/Sub vs. Pub/Sub Lite

• Pub/Sub (Standard): Global, fully managed, auto-scaling, higher cost, no capacity planning needed. Best for most use cases.
• Pub/Sub Lite: Zonal or regional, lower cost, requires capacity planning (you must provision throughput and storage). Best for high-volume, cost-sensitive workloads where you can manage capacity. Note: Pub/Sub Lite is being deprecated in favor of standard Pub/Sub with reservation-based pricing, but it may still appear on the exam.

Key Integrations

• Dataflow: Pub/Sub is the primary source for streaming pipelines in Apache Beam/Dataflow. Dataflow reads from Pub/Sub subscriptions and processes data in real time.
• BigQuery: Use BigQuery subscriptions to stream data directly into BigQuery without intermediate processing.
• Cloud Functions / Cloud Run: Use push subscriptions to trigger serverless event-driven processing.
• Cloud Storage: Use Cloud Storage subscriptions to archive or batch messages.
• Dataproc / Spark Streaming: Can consume from Pub/Sub using connectors.
• Apache Kafka: The Pub/Sub Kafka connector allows migration from or integration with Kafka-based systems.

Schema Management

Pub/Sub supports schemas (Avro or Protocol Buffers) that can be associated with topics. When a schema is attached, Pub/Sub validates messages on publish, rejecting any that do not conform. This enforces data quality at the ingestion layer.

Security and Access Control

• IAM roles: Control who can publish (roles/pubsub.publisher), subscribe (roles/pubsub.subscriber), or administer (roles/pubsub.admin) topics and subscriptions.
• Encryption: Messages are encrypted at rest and in transit by default. You can also use CMEK (Customer-Managed Encryption Keys) for additional control.
• VPC Service Controls: Restrict Pub/Sub access to resources within a VPC perimeter.
• Message storage policy: Restrict where messages are stored to specific regions for data residency compliance.

Monitoring and Operations

• Cloud Monitoring metrics: Key metrics include subscription/num_undelivered_messages (backlog size), subscription/oldest_unacked_message_age, and topic/send_request_count.
• Alerting: Set alerts on backlog growth or oldest unacknowledged message age to detect processing bottlenecks.
• Seek and Replay: You can seek a subscription to a specific timestamp or a snapshot to replay messages. This is useful for reprocessing or recovering from failures.
• Snapshots: Capture the state of a subscription's message backlog at a point in time for later replay.

Common Use Cases

• Real-time event ingestion from IoT devices, mobile apps, or web applications
• Decoupling microservices in event-driven architectures
• Streaming data into Dataflow for real-time ETL/ELT
• Log and telemetry aggregation
• Change data capture (CDC) event distribution
• Fan-out: One topic with multiple subscriptions for different downstream consumers
• Buffering between data producers and consumers operating at different speeds

How to Answer Exam Questions on Pub/Sub

The exam frequently presents scenario-based questions where you must choose the right architecture or configuration. Here is a systematic approach:

1. Identify the messaging pattern: Is it one-to-one, one-to-many (fan-out), or many-to-one? Multiple subscriptions on a single topic enable fan-out.
2. Determine ordering requirements: If order matters, look for ordering keys. If the question mentions CDC or sequential event processing, ordering is likely required.
3. Assess delivery guarantees: Pub/Sub provides at-least-once by default. If the question requires exactly-once processing, consider exactly-once delivery or idempotent processing in downstream systems like Dataflow.
4. Evaluate the subscriber type: If the scenario involves serverless event handling (Cloud Functions, Cloud Run), push subscriptions are appropriate. For Dataflow or custom applications, pull subscriptions are standard. For direct analytics, BigQuery subscriptions are ideal.
5. Consider cost and throughput: If the question emphasizes cost optimization for very high throughput, Pub/Sub Lite (or reservation-based pricing) may be relevant.
6. Think about data quality: If the question mentions schema enforcement or data validation at ingestion, schemas on topics are the answer.

---

Exam Tips: Answering Questions on Pub/Sub for Messaging and Event Streaming

• Tip 1: When a question asks about decoupling producers and consumers, or buffering between systems of different speeds, Pub/Sub is almost always the correct answer. It is the default asynchronous messaging service on GCP.

• Tip 2: Remember that each subscription gets its own copy of every message. If a question describes multiple systems that each need to process the same data independently (fan-out), the answer is one topic with multiple subscriptions, not multiple topics.

• Tip 3: If a question mentions writing streaming data directly to BigQuery with minimal code or infrastructure, consider the BigQuery subscription. This eliminates the need for Dataflow in simple ingestion scenarios.

• Tip 4: For questions about message ordering, the key concept is ordering keys. Messages with the same ordering key are delivered in order. Without ordering keys, Pub/Sub does not guarantee order. If a question asks how to ensure ordered processing of events for a specific entity (e.g., per-user or per-device), use ordering keys.

• Tip 5: If a question describes a scenario where messages are being processed multiple times (duplicates), understand that Pub/Sub's default is at-least-once delivery. Solutions include enabling exactly-once delivery on the subscription or building idempotent processing logic in the consumer (e.g., using Dataflow's built-in deduplication).

• Tip 6: Dead-letter topics are the answer when a question asks how to handle messages that repeatedly fail processing. Configure a max delivery attempts count, and unprocessable messages are routed to the DLT.

• Tip 7: If a question asks about replaying or reprocessing historical messages, the answer involves Seek (to a timestamp or snapshot). Remember that Pub/Sub retains messages for up to 7 days, and you can also retain acknowledged messages if configured.

• Tip 8: For data residency or compliance questions, remember that Pub/Sub allows you to set a message storage policy to restrict message storage to specific regions.

• Tip 9: When the question involves monitoring a streaming pipeline, the two most important Pub/Sub metrics are oldest_unacked_message_age (indicates processing lag) and num_undelivered_messages (indicates backlog size). If these are growing, the subscriber is not keeping up.

• Tip 10: Push vs. Pull distinction is critical. Push is for serverless endpoints (Cloud Functions, Cloud Run, App Engine) and when you want Pub/Sub to drive the delivery. Pull is for when the subscriber controls the rate of consumption (Dataflow, custom workers). If a question mentions an HTTP endpoint, the answer is push. If it mentions Dataflow, the answer is pull.

• Tip 11: Be wary of answer choices that suggest using Kafka on GCE/GKE when Pub/Sub can accomplish the same goal with less operational overhead. The exam favors managed services. Choose Pub/Sub unless there is a specific requirement for Kafka compatibility or features not available in Pub/Sub.

• Tip 12: Remember the 10 MB message size limit. If a question involves sending very large payloads, the recommended pattern is to store the payload in Cloud Storage and send a reference (URI) in the Pub/Sub message.

• Tip 13: For questions about schema evolution or data contracts between producers and consumers, Pub/Sub schemas (Avro/Protocol Buffers) with revision management are the answer.

• Tip 14: Understand the difference between topic-level and subscription-level configurations. Schemas and message storage policies are set on topics. Acknowledgment deadlines, dead-letter policies, retry policies, message retention, and exactly-once delivery are set on subscriptions.

By mastering these concepts and tips, you will be well-prepared to handle any Pub/Sub question on the GCP Professional Data Engineer exam.