overview and upload
Inspection evidence
innovation and architecture tab
Human review and audit
Inspection history

RoadGuard — Agentic AI Road Damage Inspector

Inspiration

Road damage is a persistent infrastructure challenge. Cracks, potholes, and deteriorating road surfaces can affect road safety, increase vehicle maintenance costs, and become more expensive to repair when they go unnoticed.

Traditional road inspections often rely on manual surveys, which can be time-consuming, difficult to scale, and inconsistent across large road networks.

We asked ourselves:

What if a road inspection system could do more than detect damage? What if it could inspect additional evidence, explain its findings, and help human inspectors make informed decisions?

This inspired RoadGuard, an agentic AI-powered road-damage inspection platform built with OpenCV 5, YOLOv8, ONNX, and Amazon Web Services (AWS).

Our objective is not to replace human road inspectors. It is to give them a practical tool that transforms road videos into organized, explainable, and auditable inspection evidence.

What It Does

RoadGuard allows users to upload a road video through a web dashboard and receive AI-generated road-damage inspection results.

Rather than treating every model prediction as a confirmed defect, RoadGuard uses a multi-stage inspection workflow:

Video inspection: OpenCV 5 processes video frames and runs YOLOv8 ONNX inference to identify potential road damage.
Damage classification: The system identifies four categories: longitudinal cracks, transverse cracks, alligator cracks, and other road damage.
Agentic reinspection: A confidence-aware inspection agent groups related observations and examines additional frames to collect supporting evidence.
Motion analysis: Sparse optical flow provides heuristic information about possible camera-motion artifacts.
Explainable evidence: RoadGuard produces original and annotated frames, confidence scores, supporting observations, and a human-readable inspection report.
Human review: Inspectors can confirm, reject, or request further inspection of individual damage candidates.
Persistent history: Inspection results and review decisions are stored in AWS, allowing users to revisit previous inspections.

The entire inspection workflow is available through a publicly hosted HTTPS web application, protected by a demonstration access code.

How We Built It

1. Building a multicountry road-damage dataset

We started with the publicly available RDD2022 road-damage dataset.

To improve geographical diversity, we prepared data from seven country and collection groups, including India, Japan, the Czech Republic, Norway, the United States, and two China-based collections.

We standardized the source annotations into four road-damage classes and created reproducible training, validation, and test splits.

The resulting dataset contained approximately 25,619 images and 55,230 cleaned annotations.

2. Training and exporting the detection model

We trained a YOLOv8n object detection model using Google Colab with GPU acceleration.

The trained model was exported to ONNX, allowing inference through OpenCV's Deep Neural Network module without requiring the full PyTorch training environment on the production server.

OpenCV 5 handles image preprocessing, ONNX inference, bounding-box postprocessing, non-maximum suppression, and supporting vision operations.

3. Developing the agentic inspection workflow

A major innovation in RoadGuard is its multi-step inspection agent.

Instead of relying on a single frame, the agent evaluates candidate detections, groups observations, performs additional frame-based inference when appropriate, and gathers temporal evidence.

It also uses sparse optical flow to identify possible motion-related inconsistencies.

The agent produces a structured inspection trace containing the observations, supporting evidence, uncertainty indicators, and processing decisions.

This is a deterministic, tool-driven inspection workflow rather than an LLM making unsupported visual judgments.

4. Deploying actual AI inference on AWS

We deployed RoadGuard to Amazon EC2 in the Mumbai region (ap-south-1).

Our cloud architecture combines:

Amazon EC2: Hosts the FastAPI backend and executes OpenCV 5 + ONNX inference on CPU.
Amazon S3: Privately stores uploaded videos, inspection traces, annotated images, review summaries, and human-review decisions.
AWS Lambda: Processes newly uploaded inspection traces and generates structured review summaries.
FastAPI: Provides endpoints for video uploads, inspection monitoring, reports, and review actions.
Nginx and HTTPS: Provide secure browser access to the hosted application.
HTML, CSS, and JavaScript: Power the interactive inspector dashboard.

The backend uploads the supporting evidence before publishing the inspection trace to S3, ensuring that the Lambda reporting workflow can access the referenced evidence.

5. Building an explainable inspection dashboard

We designed the dashboard around the complete inspector workflow.

Users can monitor live inspection progress, examine annotated evidence, understand the agent's reasoning, and record human-review decisions.

We also implemented Inspection History, allowing earlier reports and saved review decisions to be reopened without repeating inference.

To make the system accessible to nontechnical users, we added a human-readable report instead of requiring inspectors to interpret raw JSON.

Challenges We Faced

Generalizing road-damage detection

Road surfaces vary significantly across countries, lighting conditions, camera perspectives, and environmental settings.

Training on geographically diverse data helped us investigate these challenges, but class imbalance, confusing surface patterns, and visually similar damage types remain important limitations.

Distinguishing real damage from visual artifacts

Shadows, lane markings, camera movement, and changing illumination can resemble cracks.

We introduced additional frame inspection and optical-flow heuristics to gather more context. However, these checks do not guarantee that a detection is physically correct.

Moving from a local prototype to actual AWS inference

We initially developed and tested the computer vision pipeline locally. Deploying a complete video-processing workflow on EC2 introduced additional challenges involving CPU inference performance, video upload limits, Linux dependencies, server permissions, and asynchronous processing.

We addressed these through ONNX-based inference, bounded video processing, FastAPI background execution, private S3 storage, and Lambda-based report generation.

Making AI predictions understandable

Raw model outputs and JSON traces are difficult for road inspectors to interpret.

We developed original-versus-annotated evidence views, plain-language inspection summaries, and a human-review workflow so that model predictions remain understandable and open to correction.

Accomplishments and Evaluation

We successfully built and deployed an integrated system that combines real AWS-hosted inference, agentic reinspection, evidence generation, asynchronous cloud reporting, and human review.

Our multicountry YOLOv8n model achieved the following results on 3,833 held-out test images:

Metric	Result
Precision	0.5857
Recall	0.4815
mAP@50	0.5092
mAP@50–95	0.2518

These measurements evaluate the object detection model on held-out images. They are not end-to-end video-inspection accuracy measurements.

We also verified the AWS reporting workflow using a completed inspection containing six road-damage candidate events, with original and annotated evidence accessible for further review.

The dashboard supports saved inspection history and human-review records, demonstrating how detection results can be incorporated into an auditable inspection process.

What We Learned

Building RoadGuard taught us that creating a useful computer vision application requires much more than training a detection model.

We learned how to:

Prepare and evaluate geographically diverse object detection datasets.
Export YOLO models to ONNX and run inference using OpenCV 5.
Build evidence-driven, multi-step computer vision workflows.
Integrate FastAPI, Amazon EC2, S3, and AWS Lambda.
Design asynchronous video-processing and reporting pipelines.
Convert raw AI outputs into understandable, human-reviewable reports.
Handle real-world deployment challenges while preserving reproducibility and security.

Most importantly, we learned that AI confidence is not the same as verified real-world damage. Inspection systems should communicate uncertainty and retain human oversight.

Limitations and Responsible Use

RoadGuard is a decision-support prototype, not a certified road-safety or structural assessment system.

It may miss damage, confuse visually similar defects, or produce false positives under shadows, poor lighting, or challenging camera conditions.

Its confidence scores should not be interpreted as calibrated probabilities of physical damage. Temporal grouping does not guarantee that observations correspond to the same real-world defect.

Uploaded videos and associated evidence are stored in a private S3 bucket, and the dashboard requires an access code for inspection operations. Users should avoid sharing identifiable people, vehicle plates, or sensitive location information without appropriate authorization.

Human verification remains essential before making road-maintenance or safety decisions.

What's Next for RoadGuard

We plan to extend RoadGuard with larger real-world video evaluations, improved cross-frame object tracking, stronger false-positive analysis, and more robust handling of difficult lighting conditions.

Future improvements could also include GPS-based inspection mapping, scalable background processing, richer historical comparisons, and better model calibration.

Our long-term vision is to make road-damage inspection more transparent, reproducible, and accessible to infrastructure maintenance teams.

Try RoadGuard

Live application: https://13-204-218-48.sslip.io/

GitHub repository: https://github.com/akbundi/OpenCV-AI-Competition-/tree/roadguard-competition-final

Sample road videos are included in the repository so judges can evaluate the inspection workflow. Access-code details are provided separately for authorized reviewers.

RoadGuard: From road videos to explainable evidence and human-verified inspection decisions.