CoSense - Warehouse Safety with Streaming ML

Real-time warehouse safety coordination using Confluent Cloud's Flink ML_PREDICT to chain AutoML classification with Gemini summarization for human-robot collision prevention.

Comment

Inspiration

Warehouse automation is accelerating—Amazon deploys 750,000+ robots alongside human workers. But current safety systems batch-process sensor data, creating dangerous blind spots. A forklift at 15 mph covers 22 feet per second; 2-second batch processing means 44 feet of unmonitored movement. What if safety decisions happened at the speed of the data itself?

What It Does

CoSense Cloud is real-time warehouse safety coordination with three processing speeds:

  • 50ms decisions — QuixStreams joins robot telemetry, human positions, and zone context to issue STOP/SLOW/CONTINUE commands before collisions occur
  • 10-second anomaly detection — Flink's ML_DETECT_ANOMALIES uses built-in ARIMA to flag statistical outliers without external API calls
  • 5-minute AI reports — Two chained ML_PREDICT models: AutoML classifies the window (EQUIPMENT/ENVIRONMENTAL/HUMAN_FACTOR/NORMAL), then Gemini 2.5 Flash writes a natural language summary

An interactive Gemini 2.0 Flash copilot lets operators ask questions grounded in live Kafka state through 11 function tools.

How We Built It

Confluent Cloud provides the streaming backbone: 7 Kafka topics, Managed Flink with ML_DETECT_ANOMALIES and ML_PREDICT chaining, QuixStreams for sub-100ms processing.

Google Cloud powers AI and hosting: Vertex AI (Gemini 2.0/2.5, AutoML), Cloud Run, Firebase Hosting.

The architecture separates concerns by latency: streaming ML for streaming problems, LLMs for human interaction.

Challenges We Ran Into

  • Latency budget — Gemini's 500-2000ms response times were unacceptable in the critical path. Solution: LLM for on-demand copilot, Flink's built-in ML for real-time detection.
  • ML_PREDICT batching — Per-alert Gemini calls would flood the API. Solution: 5-minute tumble windows.
  • Watermark alignment — Tuning Flink watermarks across streams to avoid late data drops.

Accomplishments We're Proud Of

  • Sub-100ms end-to-end latency for safety decisions
  • Zero external API calls in anomaly detection—ML_DETECT_ANOMALIES runs entirely within Flink
  • ML_PREDICT chaining: AutoML → Gemini in a single Flink SQL pipeline
  • 11 function tools connecting Gemini to live Kafka state
  • Demonstrating streaming ML and conversational AI as complementary approaches

What We Learned

Use streaming ML for streaming problems. Confluent's ML_DETECT_ANOMALIES detects anomalies the moment they occur—no external round-trip.

ML_PREDICT chaining opens sophisticated AI pipelines within the streaming layer: AutoML for structure, Gemini for narrative.

Batching is key: aggregate streaming data into windows, then call LLMs for periodic summarization—AI insights without sacrificing real-time performance.

What's Next

  • ML_DETECT_ANOMALIES_ROBUST for multivariate anomaly detection
  • ML_FORECAST for predictive maintenance
  • Cluster Linking for multi-warehouse federation
  • Edge deployment with QuixStreams on warehouse devices

Built With

Share this project:

Updates