Traffic Stream Controller

Inspiration

During the Google Cloud AI Partner Catalyst: Accelerate Innovation hackathon, I wanted to move beyond batch processing and REST-based systems and truly understand real-time data streaming. I was curious about how modern systems handle continuous flows of events—like traffic data—at scale. This curiosity led me to explore Apache Kafka and build a hands-on project around producer–consumer streaming patterns.

What it does

Traffic Stream Controller is a real-time traffic event monitoring system. It simulates vehicle traffic events, streams them through Kafka, processes them in real time, classifies traffic conditions, and displays everything on a live dashboard. The system shows how streaming architectures work end-to-end, from event generation to visualization and storage.

How we built it

Kafka (local via Docker) as the event backbone

Kafka Producer to generate simulated traffic events (speed, location, timestamp)

Kafka Consumer to process and classify events in real time

AI Layer using mock logic with optional Gemini API integration

Streamlit for a live, interactive dashboard

CSV + SQLite for local persistence and long-term analysis Everything runs locally, without managed cloud services, to deeply understand the internals.

Challenges we ran into

Understanding Kafka’s role beyond being a “middleman”

Managing Kafka locally without a managed Confluent cluster

Handling container restarts and broker availability

Designing a clean producer–consumer flow without tight coupling

Explaining real-time streaming concepts in a simple, visual way

Accomplishments that we're proud of

Built a complete real-time streaming system from scratch

Successfully ran Kafka locally using Docker

Designed a decoupled, scalable producer–consumer architecture

Created a live dashboard for real-time observability

Stored streaming data reliably for later analysis

What we learned

Why Kafka is essential for real-time systems over threads or direct calls

How decoupling producers and consumers enables scalability

The difference between event-driven systems and request-based systems

Practical Kafka concepts: topics, brokers, consumers, offsets

How streaming architectures are used in real-world systems like traffic management

What's next for Traffic Stream Controller

Add multiple Kafka consumers (analytics, alerts, predictions)

Use real traffic or IoT sensor data

Integrate advanced AI models for predictive traffic insights

Deploy on managed Kafka for large-scale testing

Add alerting and anomaly detection in real time

If you want, I can also write:

✅ Project name

✅ One-line elevator pitch

✅ Technical architecture diagram description

✅ Judge-friendly “Why this matters” summary

Built With

• kafka
• python