ML & Data Science

Streaming data

Real-Time Traffic Analytics

Real traffic detector data streamed, processed, stored, and charted live, plus a hands-on study of event time in Flink.

Role
Data Engineer
Year
2025
Focus
Streaming data
Stack
5 technologies
  • Kafka
  • Flink
  • PostgreSQL
  • Grafana
  • Docker
Kafka pipelineFlink watermarksCamera analytics

  • Live

    Grafana dashboards

  • 15s

    Flink tumbling windows

  • 1 cmd

    Dockerised stack start-up

Overview

A producer publishes traffic detector readings to a raw Kafka topic. A processor enriches each record with an operational flag, and a sink writes the processed events to PostgreSQL. Grafana then shows a live time series, daily peaks and averages, and a sensor health table.

A companion project streams social-media datasets through Kafka into Apache Flink. It compares event-time windows with watermarks against processing-time windows, with hashtag counts every 15 seconds.

Architecture

Streaming pipeline

  1. 01

    Sensors

    Traffic detector readings, replayed in real time.

  2. 02

    Kafka · raw

    Producer publishes events to the raw topic.

  3. 03

    Processor

    Raw to processed events, adding an is_operational flag.

  4. 04

    PostgreSQL

    Sink stores events plus hourly averages, daily peaks, and availability.

  5. 05

    Grafana

    Live dashboards that refresh every few seconds.