**TrafficAgent-AI: Autonomous Vision & Incident Commander

• 

  2

• 

  1

• 

  3

Inspiration

Urban intersections with fixed timers cannot adapt when unpredictable events occur. A single stalled vehicle or minor fender-bender inside a junction box creates a cascade effect: upstream drivers continue entering on green lights, gridlocking the entire intersection and stalling cross traffic. We built TrafficAgent-AI to transition traffic cameras from passive video recorders into active, autonomous agents capable of perceiving bottlenecks, altering physical signals, and notifying emergency responders in real time.

What it does

TrafficAgent-AI is an end-to-end edge-to-cloud autonomous traffic management system: Perception: Continuously segments vehicles using HSV color-space processing, tracks individual movement centroids, and counts approach queue lengths across North, South, East, and West approaches.Yellow-Box Anti-Gridlock Sentinel: Detects stationary vehicles inside the junction box within 20 frames (~0.6s) and holds oncoming vehicles behind white stop lines so multi-car pileups cannot occur.Closed-Loop Agentic Action: When an incident or major queue asymmetry is detected, the agent autonomously executes three tools: tool_digital_zoom_roi: Dynamically extracts and sharpens a 2.0x zoomed crop of the affected vehicle into an active Picture-in-Picture window. tool_adjust_signals: Overrides signal phase durations to hold conflicting lines on red and flush unblocked lanes. tool_dispatch_incident_alert: Formats a structured emergency payload and transmits it to Amazon SNS and S3 for immediate dispatch.

How we built it

Vision Pipeline: Built with OpenCV 5 using morphological filtering, Euclidean centroid tracking, and AABB collision projection raycasting.Agent Architecture: Implemented a strict Perception $\rightarrow$ Decision $\rightarrow$ Action loop with dedicated tools and telemetry arbitration. Cloud & Edge Acceleration: Integrated with AWS using boto3 for Amazon SNS incident broadcasting and Amazon S3 evidence logging. The benchmark and deployment suite is tailored for Arm64 AWS Graviton3/4 instances utilizing the Cloud-Optimized OpenCV Library (COOL).

Challenges we ran into

Geometric Blindness in Cross-Traffic: Early iterations had approaching cars colliding with stationary cross-traffic vehicles. We solved this by implementing forward-projected bounding-box collision detection that anticipates obstacles in both parallel and perpendicular trajectories.

Luminance Threshold Drops: Initial grayscale luminance thresholding erased pure red stalled vehicles (R=255 grayed out to 76). Transitioning to dual-band HSV color-space segmentation made vehicle extraction robust against asphalt backgrounds regardless of color or lighting.

Accomplishments that we're proud of

Achieving 8.96 ms average latency and over 111 FPS throughput on real-time vehicle tracking and agent arbitration. Successfully demonstrating a working closed-loop system where vision findings immediately alter the physical environment and trigger cloud interventions. Building a full end-to-end prototype with clean queue physics, live HUD, active digital inspection PIP, and zero external runtime crashes.

What we learned

How to coordinate an agent loop with low-level computer vision pipelines without dropping frames. The performance benefits of vectorization and Cloud-Optimized OpenCV (COOL) on modern Arm64/Graviton architectures.

What's next for **TrafficAgent-AI: Autonomous Vision & Incident Commander

Deploying multi-camera fusion across connected intersections to coordinate green-wave corridors along entire arterial roads.

Integrating Vision-Language Models (VLMs) directly into the PIP inspection window to automatically describe vehicle make, model, and visible hazard states for first responders.

Built With

• amazon-sns
• amazon-web-services
• aws-graviton
• boto3
• numpy
• opencv
• python