Flink Event-Time Watermarks

A tweet sent at 10:00:14 can reach the system at 10:00:16 because of mobile signal, retries, buffering, or Kafka batching. If windows use the processing machine's clock, that tweet lands in the wrong 15-second window and the counts change on every run. The job needs correct, repeatable counts without waiting forever for stragglers.

Screenshots from the running Docker stack, Kafka UI, and the Flink dashboard. Tap any figure to zoom.

1. Tap to zoom

   Fig. 01 · Step 1 · Docker

   The stack is up

   Kafka, ZooKeeper, JobManager, TaskManager, and Kafka UI all running. Before any job, docker compose ps, the service logs, the Flink UI on port 8081, and a Kafka produce/consume smoke test confirm the infrastructure is healthy.

2. Tap to zoom

   Fig. 02 · Step 2 · Ingestion

   Datasets in Kafka

   Each CSV line became one message: 1,545 in social_facebook (709 KB) and 3,107 in social_twitter (986 KB), one partition each for a single-broker setup.

3. Tap to zoom

   Fig. 03 · Step 3a · Verification

   Flink reads both topics

   Two verifier jobs consume each topic and print the lines to the TaskManager logs, proving the Kafka-to-Flink connection before adding windowing logic.

4. Tap to zoom

   Fig. 04 · Step 3b · Event-time windows

   Hashtag counters running

   TwitterHashtagCounter and FacebookHashtagCounter running with event-time windows and watermarks. The completed list keeps the earlier verifier and test runs.

5. Tap to zoom

   Fig. 05 · Step 3c · Output

   Results back in Kafka

   The window results land in twitter-hashtag-counts and facebook-hashtag-counts, ready for any downstream consumer.

6. Tap to zoom

   Fig. 06 · Step 4 · Optimisation

   Scaling out to two partitions

   Both input topics altered to 2 partitions with kafka-topics --alter, so two Flink subtasks can each read one partition.

7. Tap to zoom

   Fig. 07 · Step 4 · Job graph

   Partition-aware job graph

   Source → filter → map runs at parallelism 2, one subtask per partition, each tracking its own watermark. The global windowAll operator runs at parallelism 1 and receives the minimum watermark across partitions.

• Periodic watermarks

  steady streams

  Emitted on a timer (e.g. every 200 ms) as the latest timestamp minus an allowed lateness. Easy to configure and predictable, but needs tuning and isn't flexible during traffic spikes.

• Punctuated watermarks

  log files, batches

  Emitted only when a special marker event arrives (e.g. end_of_batch). Strict and accurate for structured logs, but social media has no such boundary events.

• Event-time watermarks

  social media, IoT

  Advance with the timestamps actually observed. They adapt to network delays, bursts, uneven arrival, and slow partitions, so fewer events are wrongly marked late.

• Accuracy: processing-time windows use the Flink machine's clock, so a delayed tweet falls into the next window and counts change between runs. Event time with watermarks always puts it in the window it was written in, so results are correct and reproducible.
• Latency: processing-time windows close the moment the clock passes the boundary. Event-time windows deliberately wait for the watermark, adding up to the 5-second lateness bound.
• Throughput: processing time is simpler and handles slightly more events per second. Event time must read each timestamp, compare it with the watermark, and buffer out-of-order events.
• CPU and memory: event time holds late events and timers in state until each window is complete, so it uses more resources.
• Verdict: when accuracy matters (hashtag trends, billing, monitoring, anomaly detection) event time with watermarks is the industry standard and worth the extra cost.

• Before: one global watermark meant a slow or idle partition held back event time for the whole job, so data from faster partitions waited.
• After: the WatermarkStrategy is applied directly on the KafkaSource, which tracks a watermark per partition and forwards the minimum to downstream operators. withIdleness(10 s) stops an idle partition from blocking progress.
• Accuracy was unchanged between parallelism 1 and 2, because timestamps, not arrival order or task assignment, decide which window an event belongs to.
• Throughput rose and latency improved under uneven traffic: ingestion, timestamp extraction, filtering, and mapping now run in parallel.
• Limitation: windowAll is a single, non-parallel operator, so it becomes the bottleneck. Full scaling needs keyed windows, keyBy(hashtag), so each hashtag aggregates in parallel.

• Switch from windowAll to keyBy(hashtag) windows to remove the single-operator bottleneck.
• Enable checkpointing with transactional Kafka sinks for exactly-once results across restarts.
• Send events later than the watermark to a side output so they're counted separately instead of dropped.