Drishtis

Inspiration

Road surface degradation, craters, and potholes cause billions of dollars in vehicular damage annually, create severe accident risks for two-wheelers and cyclists, and degrade municipal transportation infrastructure. Traditional road maintenance monitoring depends heavily on sporadic citizen complaints via phone hotlines or exorbitant, multi-million-dollar dedicated LiDAR survey trucks that cities cannot deploy at scale. We envisioned Drishti (meaning "Vision" or "Insight") to democratize and automate road infrastructure inspection. By turning any standard smartphone, dashcam, or municipal fleet camera into an autonomous edge safety inspector, Drishti brings intelligent, real-time spatial vision directly to city roads. What it does Drishti ingests real-time dashcam and road video feeds, coordinating a 4-Agent Collaborative AI Specialization Team to observe, verify, size, and officially report road hazards: Agent 1: Vision Stream Observer — Continuously monitors video frames, inspecting road surface texture (e.g., bituminous asphalt, concrete), moisture levels (dry, damp, wet), and ambient lighting conditions. Agent 2: Spatial Pothole Specialization Agent — Scrutinizes optical candidate regions to verify genuine asphalt cavities while rejecting false positives (e.g., shadows, leaves, oil slicks, manholes). It generates high-precision normalized bounding coordinates and identifies lane positioning (Left Wheel Path, Center Lane, Right Wheel Path). Agent 3: Parameter & Triage Agent — Calculates real-world physical metrics (width , length , depth ), computes estimated patch material volume in kilograms, and assigns a strict severity flag: BIG: Surface Area or Depth

Emergency 24h Dispatch MEDIUM: or

High Priority 72h Queue SMALL: and

Routine Maintenance Agent 4: Municipal Government Filing Agent — Packages official Department of Transportation 311 work orders with real-time vehicle GPS coordinates , generates a cryptographic SHA-256 verification hash, and submits the docket directly into the municipal government ledger. Mathematical Surface & Patch Material Modeling Agent 3 models the cavity surface area as an ellipse and cavity volume as an elliptical paraboloid: With compacted asphalt mix density and safety factor , the required patch mix is estimated as:

How we built it

Google Gemini SDK 2 (@google/genai): Integrated server-side with gemini-3.7-flash, utilizing structured responseSchema definitions (Type.OBJECT) and system instructions to orchestrate the 4 sub-agents in a single high-speed inference pipeline. Node.js & Express Architecture: Implemented secure server-side API proxy routes (/api/agents/orchestrate-pothole, /api/government/potholes) to safeguard API secrets while hosting an in-memory 311 municipal work order database with live CRUD endpoints. Real-Time Video & Canvas Reticle HUD: Built an HTML5 Video and synchronized 2D Canvas rendering layer that draws ADAS lane trajectory guides, active laser scan lines, and color-coded bounding brackets with live physical dimension callouts. Procedural Asphalt Driving Simulator: Designed a custom 3D perspective road driving simulator in Canvas that procedurally generates moving road markers and approaching potholes with realistic depth perspective. Geospatial GIS & Navigation Subsystem: Developed real-time vehicle GPS tracking, waypoint interpolation along urban transit corridors, and an interactive municipal GIS map visualizer. Frontend Stack: Built with React 18, TypeScript, Tailwind CSS, and Lucide vector icons.

Challenges we ran into

Multi-Agent Pipeline Latency in Video Processing: Coordinating four distinct operational steps without stalling the live video stream. We solved this by designing a unified structured schema with Gemini 3.7 Flash, allowing all 4 agent roles to execute collaboratively in a single low-latency round trip. Perspective Foreshortening on Road Surfaces: Road cameras view asphalt at acute angles, causing distant potholes to appear severely compressed. We incorporated perspective geometry scaling to calibrate dimensional calculations relative to the road plane. Preventing False Positives: Differentiating real structural road cavities from dark tar crack sealant, puddles, and tree shadows required precise prompt engineering and optical reasoning rules in Agent 2. Accomplishments that we're proud of Seamless Multi-Agent Orchestration: Successfully coordinating 4 distinct specialized AI agents from raw optical feed ingestion all the way to municipal ticket creation. Instantaneous 311 Dispatch Integration: Bridging the gap between edge AI vision and civic public works systems with automated geotagging, dimension sizing, and asphalt demand forecasting. Zero-Setup Simulation & Universal Compatibility: Empowering anyone to test the entire road surveillance and filing system immediately using either simulated driving mode, asset video feeds, or live device hardware cameras.

What we learned

Power of Gemini SDK 2 Structured Outputs: Enforcing strict JSON schemas eliminates output parsing errors and guarantees dependable execution for civic infrastructure automation. Agent Specialization Clarity: Breaking complex tasks into discrete agents (Vision Observer Spatial Detector Parameter Sizing Government Filing) drastically enhances transparency and explainability for civic authorities. What's next for Drishti V2X (Vehicle-to-Everything) Hazard Broadcasting: Transmitting instant alerts to approaching connected vehicles to prevent wheel blowouts and suspension damage in real time. Fleet-Wide Road Degradation Heatmaps: Aggregating observations across city transit buses and delivery fleets to build predictive road deterioration models. Automated Repair Crew Route Optimization: Clustering open 311 work orders to automatically plot the most efficient daily repair routes for municipal asphalt patcher trucks.