Eye of Sauron

Red Suv Found Demo

Inspiration

One night, Abhiram and I watched bodycam footage where an officer was bleeding out after a traffic stop, trying to radio a vague suspect description: “red sedan heading southbound.” It felt broken. In 2026, we already have thousands of highway cameras, but most systems still treat them like passive video feeds. That was the spark for Eye of Sauron: not more cameras, but cameras that can actually understand what they’re seeing in real time.

What it does

Eye of Sauron is a real-time city intelligence interface for the Bay Area. You can ask questions by text or voice (wake phrase: “hey sauron”), and it:

Resolves the location/context
Pulls nearby live traffic camera frames
Runs multimodal vision analysis on those frames
Combines results with live incident feeds (511 dispatch/traffic + SF 911 CAD)
Shows a grounded summary with map context, camera hits, and uncertainty notes Instead of scanning dozens of feeds manually, operators get an answer-first, evidence-backed view of what’s happening now. ## How we built it
Frontend: React + TypeScript + Vite + Cesium for 3D geospatial visualization
Backend: Node + TypeScript + Hono API routes for chat, vision, cameras, incidents, and voice
Vision pipeline: camera search → frame extraction → multimodal inference → ranked observations
Model resilience: AMD Cloud primary with Groq and Gemini fallbacks
Voice: speech recognition for wake phrase + ElevenLabs TTS (with browser fallback)
Data sources: Caltrans camera feeds, 511 traffic/dispatch APIs, DataSF police/fire dispatch
Deploy: Vercel (frontend + backend), with env keys stored in platform secrets ## Challenges we ran into
Unstable camera feeds and intermittent 404 frame failures
Provider incompatibilities (auth, endpoint shape, model version mismatches)
Wake phrase reliability in noisy real-world speech
Deployment/routing issues causing NOT_FOUND responses
Keeping secrets safe while debugging fast across environments ## Accomplishments that we're proud of
End-to-end working system from voice query to map-grounded answer
Multi-provider AI fallback that keeps the product running during failures
Unified view of camera intelligence + incident data in one interface
Production deployment with no secret leakage in client code ## What we learned
Reliability matters more than single-model accuracy in real-time systems
Geospatial UX is critical for trust; users need to see where evidence comes from
Prompting is only part of the solution; robust tool orchestration is the hard part
Observability and explicit uncertainty are essential for operator-grade AI ## What's next for Eye of Sauron
Continuous multi-camera vehicle tracking and handoff
Time-travel/replay timeline for incident reconstruction
Proactive alerts (“watch for this vehicle in this corridor”)
Better confidence scoring and source attribution per claim
Operator workflows (case notes, sharing, audit trails, role-based access)