UrbanSense-AI

Autonomous agentic auditing of road safety, active mobility facilities, and environmental view factors from street imagery using OpenCV 5 & AWS Graviton COOL.

Comment

Inspiration

Every 24 seconds, someone loses their life in a road traffic crash globally—totaling over 1.19 million deaths annually (UN SDG 3.6). Over half of these casualties are vulnerable road users (pedestrians, cyclists, and motorcyclists).

While the International Road Assessment Programme (iRAP) provides a globally proven standard for road safety engineering, conventional road safety audits remain manual, slow, labor-intensive, and financially out of reach for thousands of municipal jurisdictions. We built UrbanSense-AI to democratize and automate road infrastructure safety audits using OpenCV 5, AWS Graviton (COOL), and Agentic Computer Vision (MCP).

What It Does

UrbanSense-AI ingests street-level and $360^\circ$ panoramic imagery to autonomously perform:

  1. Deterministic iRAP v3.10 Star Ratings (1–5 Stars): Calculates exact crash type scores for all 4 road user groups (Vehicle Occupants, Motorcyclists, Bicyclists, Pedestrians) using official iRAP risk equations and produces actionable remedial engineering countermeasures.
  2. Accelerated Pavement Defect Quantification: Identifies longitudinal, transverse, and alligator cracks, detects potholes, and computes a standardized Pavement Condition Index (PCI: 0–100).
  3. Active Mobility & Traffic Facility Auditing: Detects zebra crossings, tactile paving (ADA/accessibility), cycle lane demarcations, bollards, and roadside guardrails via OpenCV 5 DNN and ONNX.
  4. Urban Environmental View Factors: Computes Sky View Factor (SVF) via hemispherical solid-angle integration, Green View Index (GVI) via HSV/Lab Excess Green Index filtering, and urban canyon building enclosure ($H/W$).
  5. Closed-Loop Agentic Vision (MCP): Operates a dynamic perception-decision-action loop where visual anomalies (e.g. cracked asphalt co-located with a pedestrian zebra crossing) dynamically trigger high-resolution Bird's-Eye-View (BEV) homography unwarping, adjust safety ratings, and initiate Human-in-the-Loop (HITL) engineering escalation.

How We Built It

  • Vision Engine (OpenCV 5 & AWS COOL): Developed high-speed equirectangular-to-rectilinear pinhole projections (cv.remap), metric top-down road plane homography (cv.warpPerspective), Black-Hat morphological gradient crack segmentors, and solid-angle SVF integration.
  • AWS Graviton Arm64 Acceleration: Optimized core workloads on AWS Graviton3 (c7g.2xlarge) using the Cloud-Optimized OpenCV Library (COOL), achieving 1.81x throughput speedup and 42% cloud cost savings ($0.209 / 10k frames).
  • Agentic Vision & Model Context Protocol (MCP): Standardized OpenCV 5 vision primitives into JSON-RPC MCP tool endpoints (unwarp_panoramic_roi, unwarp_road_bev, analyze_pavement_cracks, compute_view_factors, audit_active_mobility, calculate_irap_star_rating). The orchestrator agent interprets perceptual cues and dynamically adjusts its inspection path.
  • Cloud Backend & Interactive Frontend: Built a high-performance FastAPI backend with CORS and static asset streaming, paired with a glassmorphic Streamlit dashboard featuring interactive $360^\circ$ FOV sliders, defect heatmaps, iRAP star scorecards, and live MCP execution trace graphs.

Challenges We Ran Into

  1. Equirectangular Distortion Removal: Planar objects near the nadir and poles of $360^\circ$ panoramas suffer severe spherical distortion. We derived vectorized spherical-to-cartesian coordinate transforms with rotation matrices ($R_y, R_p, R_r$) fed into cv.remap for artifact-free rectilinear views.
  2. Accurate Solid-Angle SVF Calculation: Standard flat pixel counting fails on fisheye lenses due to projection angle compression. We implemented the Johnson & Watson discrete ring-weighted solid-angle integration formula: $$\text{SVF} = \sum_{r=1}^R \text{SkyRatio}(r) \cdot \left[ \sin^2\left(\frac{\pi r}{2R}\right) - \sin^2\left(\frac{\pi(r-1)}{2R}\right) \right]$$
  3. Agentic Determinism in Safety-Critical Auditing: Pure LLMs often hallucinate road ratings. We decoupled semantic perception from deterministic mathematics: the agent uses vision tools to code structured road attributes, which are fed into a deterministic, mathematically rigorous iRAP v3.10 calculator.

Accomplishments That We're Proud Of

  • 100% Deterministic iRAP v3.10 Compliance: Full mathematical fidelity with official international road safety standards across all 4 road user groups.
  • Sub-35ms Latency on AWS Graviton COOL: Processing high-resolution $2048 \times 1024$ panoramic frames in real time.
  • Genuine Closed-Loop Agentic Behavior: Visual anomalies directly alter subsequent tool calls and generate transparent, auditable JSON execution traces.
  • Zero-Dependency Portability: Complete Dockerized solution running seamlessly on AWS ECS on Fargate, Graviton EC2, or local edge hardware.

What We Learned

  • How Arm64 NEON vectorization in OpenCV COOL delivers substantial speedups on non-linear coordinate remapping and morphological operations.
  • The power of Model Context Protocol (MCP) in transforming traditional computer vision pipelines into transparent, agentic decision systems.

What's Next for UrbanSense-AI

  • Real-Time Dashcam/Drone Stream Ingestion: Ingesting live RTSP video feeds from municipal utility fleets for continuous city-wide road network auditing.
  • Automated Digital Twin Generation: Exporting surveyed corridors into OpenDRIVE and GeoJSON formats for integration with GIS platforms and autonomous driving simulators.

Built With

Share this project:

Updates

Submission history