Stream Processing with Event Hubs and Structured Streaming
Why is Stream Processing with Event Hubs and Structured Streaming Important?
In modern data engineering, organizations need to process and analyze data in real time to make timely business decisions. Azure Event Hubs combined with Apache Spark Structured Streaming (available in Azure Databricks and Azure Synapse Analytics) provides a powerful, scalable, and fault-tolerant solution for ingesting and processing millions of events per second. For the DP-203 exam, understanding this integration is critical because it represents a core pattern for building real-time data pipelines on Azure.
What is Stream Processing with Event Hubs and Structured Streaming?
Stream processing is the practice of continuously ingesting, processing, and outputting data as it arrives, rather than waiting for batch intervals. The two key components in this pattern are:
Azure Event Hubs: A fully managed, real-time data ingestion service capable of receiving and processing millions of events per second. It acts as a distributed streaming platform similar to Apache Kafka. Event Hubs uses a partitioned consumer model, where data is organized into partitions to enable parallel processing. It supports features such as Event Hubs Capture (for automatic archiving to Azure Blob Storage or Azure Data Lake Storage), consumer groups, and a Kafka-compatible endpoint.
Apache Spark Structured Streaming: A scalable, fault-tolerant stream processing engine built on the Spark SQL engine. It treats a live data stream as an unbounded table that is continuously appended. You write streaming queries using the same DataFrame and Dataset APIs used for batch processing, making it intuitive and consistent. Structured Streaming provides exactly-once processing semantics and supports event-time processing with watermarking.
How Does It Work?
The architecture follows these steps:
1. Data Ingestion into Event Hubs:
Producers (IoT devices, applications, services) send events to an Event Hub namespace. Events are distributed across partitions based on a partition key or round-robin. Each event contains a body, properties, and system properties such as offset, sequence number, and enqueued time.
2. Connecting Structured Streaming to Event Hubs:
In Azure Databricks or Synapse Spark, you use the azure-eventhubs-spark connector library. You configure a connection string that includes the Event Hub namespace, name, shared access key, and optionally a consumer group. The read stream is initialized like this:
df = spark.readStream.format("eventhubs").options(**ehConf).load()
The ehConf dictionary includes the connection string (encrypted using EventHubsUtils) and optional settings like maxEventsPerTrigger, startingPosition, and consumerGroup.
3. Processing the Stream:
The incoming DataFrame contains columns such as body (the event payload as binary), partition, offset, sequenceNumber, enqueuedTime, and properties. You typically cast the body to string and parse it (e.g., using from_json) to extract structured fields. You can then apply transformations such as filtering, aggregation, joins, and windowed operations.
4. Watermarking for Late Data:
When performing time-based aggregations, you use watermarking to define how late data can arrive before being dropped. For example:
df.withWatermark("enqueuedTime", "10 minutes").groupBy(window("enqueuedTime", "5 minutes")).count()
This tells Spark to wait up to 10 minutes for late-arriving data relative to the event time.
5. Output Sinks:
Processed data is written to an output sink using writeStream. Common sinks include:
- Delta Lake (most common for DP-203 scenarios)
- Azure Blob Storage or ADLS Gen2 (Parquet, JSON, CSV)
- Console (for debugging)
- Memory (for debugging)
- Another Event Hub (for chaining pipelines)
- Kafka
Example: df.writeStream.format("delta").option("checkpointLocation", "/checkpoint/path").start("/output/path")
6. Checkpointing:
Structured Streaming uses checkpointing to track the progress of a streaming query. The checkpoint location stores offsets, state data, and metadata. This enables exactly-once processing and allows a query to resume from where it left off after a failure. Checkpointing is mandatory for production workloads.
7. Trigger Modes:
- Default (micro-batch): Processes data as soon as the previous batch completes
- Fixed interval: Processes at a specified interval (e.g., every 30 seconds)
- Once: Processes all available data in a single batch and stops (useful for cost-effective incremental processing)
- Available Now: Similar to once but processes in multiple micro-batches and then stops
- Continuous (experimental): Low-latency processing with at-least-once guarantees
8. Output Modes:
- Append: Only new rows are written to the sink (default, used when there are no aggregations or after watermarking)
- Complete: The entire result table is written to the sink after every trigger (used with aggregations)
- Update: Only rows that were updated since the last trigger are written
Key Concepts for the DP-203 Exam:
Event Hubs Partitions: More partitions enable higher throughput and parallelism. The number of partitions is set at creation and cannot be changed (except in premium/dedicated tiers). Structured Streaming creates one task per partition for parallel reading.
Consumer Groups: Each consumer group provides an independent view of the event stream. Different applications reading from the same Event Hub should use separate consumer groups. The default consumer group is $Default.
Event Hubs Capture: Automatically archives streaming data to Azure Blob Storage or ADLS Gen2 in Avro format. This is useful for creating a cold-path or batch-processing layer alongside the hot-path stream processing.
Event Hubs vs. Kafka Endpoint: Event Hubs provides a Kafka-compatible endpoint, allowing Kafka clients and Structured Streaming's Kafka connector to connect to Event Hubs without code changes (just configuration changes). This is relevant when migrating existing Kafka-based pipelines to Azure.
Throughput Units (Standard) and Processing Units (Premium): These control the capacity of Event Hubs. One throughput unit allows 1 MB/s ingress and 2 MB/s egress. Auto-inflate can automatically scale throughput units.
Exactly-Once vs. At-Least-Once: Structured Streaming with checkpointing and idempotent sinks (like Delta Lake) provides exactly-once end-to-end guarantees. The connector tracks offsets in the checkpoint, not in Event Hubs, so it manages its own progress tracking.
Integration with Delta Lake: Writing streaming data to Delta Lake enables ACID transactions, schema enforcement, time travel, and the ability to run batch and streaming queries on the same table simultaneously (the Lakehouse pattern).
Schema Evolution: When processing streaming JSON or Avro data, schema changes must be handled carefully. Delta Lake supports schema evolution with the mergeSchema option.
Exam Tips: Answering Questions on Stream Processing with Event Hubs and Structured Streaming1. Know When to Choose Event Hubs: If a question involves high-throughput event ingestion from IoT devices, application telemetry, or clickstreams, Event Hubs is typically the correct choice. If the scenario mentions Kafka compatibility, remember that Event Hubs can serve as a Kafka endpoint.
2. Understand Checkpointing: Many exam questions test whether you know that a checkpoint location is required for fault tolerance and exactly-once processing. Always look for answers that include a checkpoint location configuration.
3. Differentiate Output Modes: The exam may present scenarios where you need to choose between append, complete, and update modes. Remember: append mode works only when Spark can determine rows will not change (no aggregations, or aggregations with watermarking). Complete mode is for aggregations where the entire result must be rewritten. Update mode outputs only changed rows.
4. Watermarking Questions: If a question involves late-arriving data or time-windowed aggregations, look for answers that include
withWatermark(). Without watermarking, Spark must maintain all state indefinitely, which is not practical.
5. Recognize the Connector Configuration Pattern: Know that the Event Hubs connector uses
format("eventhubs") and requires an encrypted connection string. If a question shows Kafka format (
format("kafka")) with Event Hubs, it implies using the Kafka-compatible endpoint.
6. Partitions and Parallelism: If a question asks how to increase throughput, increasing the number of Event Hub partitions and Spark executors is the correct approach. Each partition is read by one Spark task.
7. Event Hubs Capture vs. Structured Streaming: Capture is for automatic archival (cold path) without custom code. Structured Streaming is for real-time processing with transformations (hot path). The exam may test whether you can distinguish between these two approaches.
8. Trigger Once / Available Now: If a scenario describes cost-sensitive workloads that need incremental processing without a continuously running cluster, trigger once or trigger available now is the correct pattern. This combines the benefits of streaming (offset tracking, checkpointing) with the cost model of batch.
9. Consumer Groups: If two different applications need to read from the same Event Hub independently, the answer involves creating separate consumer groups, not separate Event Hubs.
10. End-to-End Architecture: The exam often presents architecture-level questions. A common correct pattern is:
Producers → Event Hubs → Structured Streaming (Databricks/Synapse) → Delta Lake → Downstream Analytics (Power BI, Synapse SQL). Recognize this pattern and the role each component plays.
11. Read the Question Carefully for Latency Requirements: If a question specifies sub-second latency, continuous processing mode may be relevant. For most scenarios, micro-batch processing (default) with intervals of seconds is sufficient and provides exactly-once guarantees.
12. Remember Key Limits: Event Hubs Standard supports up to 32 partitions per Event Hub and 20 consumer groups. Premium and Dedicated tiers offer higher limits. Maximum event size is 1 MB (Standard) or 1 MB (Premium, configurable up to 1 MB by default but batching allows larger logical messages).
