MargAI

Turn dashcam footage into auditable road surveys and human-reviewed repair decisions.

Comment

Judge quick start, updated 26 October 2026

We explicitly opt into the Agentic Vision Award. The submission's Special Award Consideration field has been saved as Agentic Vision Award.

Start with recorded evidence at https://margai.13.204.212.172.sslip.io/?survey=pothole_kumasi . Choose Review activity in the sidebar, then Work orders. This path reads stored evidence, tool traces, policy checks and recorded decisions without starting model inference or using live-review credits. The recorded Kumasi audit currently shows eight tool calls and six of six policy checks passed. Export activity and Export report are available. Live review is optional.

Fresh measurement, 26 October: the public-repository OpenCV 5 detector processed 29 existing demo keyframes in three passes, 87 timed calls after three warm-ups. Median latency was 368.97 ms, p95 400.00 ms, and sequential throughput 2.67 images/s, on an AMD EPYC 9V74 x86_64 shared host with one OpenCV thread. These are detector-only CPU timings, excluding file reads, privacy redaction and agent/network work. They do not update September mAP, precision or recall and are not AWS/Modal timings or a matched-hardware comparison. Method, model/input hashes and every timing: https://github.com/himanshu748/marg-ai/blob/master/docs/FRESH_LATENCY.md .

If the hosted demo is unavailable, use the repository evidence and local read-only fallback: https://github.com/himanshu748/marg-ai/blob/master/docs/JUDGING_GUIDE.md . The public endpoint and recorded browser journey were checked on 26 October; this is a point-in-time check, not a guarantee of uptime throughout judging.

Inspiration

A road-damage photograph is not a repair decision. The same pothole can appear in dozens of video frames, uncertain detections can waste inspection time, and a confident-looking map can hide synthetic GPS. MargAI gives road-survey reviewers a path from dashcam footage to evidence they can inspect and a repair recommendation they can accept or reject.

What it does

MargAI detects longitudinal cracks, transverse cracks, alligator cracks, and potholes in dashcam video. It selects stable keyframes, tracks observations into distinct instances, preserves best-observation evidence, and estimates severity and road-plane area. Optional timestamped GPX provides location; surveys without it are explicitly labeled synthetic.

Its review agent re-inspects uncertain regions using computer vision, compares inconsistent observations, dismisses unconfirmed detections, requests resurveys, and drafts work orders under explicit eligibility rules. Those visual results change subsequent tool calls and repair recommendations. A completed review and passing policy audit are required before a human can approve or reject a draft. Approval records a decision; it does not dispatch a repair.

The workbench supports uploads, evidence and map inspection, observable jobs, review traces, and recorded decisions. Our latest local pass adds next-step guidance, resolved-versus-unresolved evidence labels, outcome filters, and self-contained printable reports containing public evidence, audit results, provenance, and decision notes.

How we built it

OpenCV 5 does substantive video and image analysis: VideoCapture, Laplacian frame-quality scoring, ORB features and RANSAC homography for keyframe selection, DNN inference with an RDD2022 YOLOv8 ONNX export, Farneback optical flow, perspective-based area estimation, and classical texture/color cues. Best-effort face and plate redaction uses OpenCV Zoo YuNet and tiled LPD-YuNet. Failed privacy models stop new processing; raw inspection imagery is not served through the public media API.

Python and Pydantic define the survey contract. FastAPI and Leaflet provide the workbench. Explicit agent tools, a JSONL trace, a bounded call budget, and a pure auditor make decisions inspectable. The hosted live review uses deepseek-ai/DeepSeek-V4.1-Flash through an existing Modal shared endpoint. The model chooses OpenCV tool calls from structured evidence; it does not receive raw frames directly. A deterministic MockLLM remains available for local reproduction. The updated judge video shows the live hosted DeepSeek agent. Bedrock remains disabled after quota failures.

The current source is deployed to an existing AWS Lightsail host in ap-south-1 as an isolated Docker container behind Caddy HTTPS. The public judge demo preserves immutable recorded evidence while allowing isolated live reviews and private human decisions on sample copies. Public uploads stay disabled; the survey contract is served from local disk. It creates no new paid AWS resource. On October 22, OpenCV 5 rechecked 45 demo images for privacy and processed the licensed three-second video fixture on AWS: 90 source frames, 30 sampled frames, 8 keyframes, one D20 frame detection, and zero retained instances. The pipeline plus deterministic agent took 71.283 seconds under a 0.5 CPU cap. This is a smoke test, not an accuracy or matched-hardware benchmark. The S3/SQS/DynamoDB/ECS/Fargate/ALB CloudFormation implementation remains an alternate deployment path; it is not claimed as the current host architecture.

Challenges we ran into

Temporal identity and evidence association matter as much as per-frame confidence. Tight crops lost road context during re-inspection, so the agent uses a context window and IoU-aware confirmation. We corrected escalation eligibility so one or two severity-3 potholes cannot trigger repeated failed drafting, while three eligible instances or severity 4–5 can qualify.

We also repaired cloud admission failures: queued work remains blocked until acceptance, failed sends produce consistent local/cloud failure state and original cleanup, and retry behavior is covered by mocked cloud tests. Measurements now use the actual best evidence frame dimensions rather than assuming a 16:9 resize.

Accomplishments

Historical September evaluation on 600 labeled RDD2022 India images recorded all-point [email protected] of 0.7390, precision of 0.8376 and recall of 0.6974 at confidence 0.25. These are preserved historical results, not a new evaluation of the October changes. The India slice provenance and licensing caveat are documented.

Fresh October 26 detector-only CPU timing: 29 existing demo keyframes over three passes (87 timed calls after three warm-ups), one OpenCV thread on an AMD EPYC 9V74 shared host; median 368.97 ms and p95 400.00 ms. This excludes file reads, privacy processing and agent/network work; it measures latency, not new detector accuracy or an AWS/Modal benchmark. Method, input/model hashes and timings: https://github.com/himanshu748/marg-ai/blob/master/docs/FRESH_LATENCY.md .

The current gated demo fixtures retain 3 instances from 21 frame detections in the cars clip, and 3 from 96 detections in the Kumasi clip. Compression is not proof of detection accuracy; our evaluation also documents false positives, misses, and eight annotated failure cases.

The reviewed experience workflow and Modal integration are published in source baseline a5c78edd6402c4aa2f3cb65ea5a87ba4164658f2, with no outstanding unpushed implementation patch. Current master is 378665d0ba65943eb9e01c05cb6c0d901bcd3235; that newer commit only clarifies the README's current Lightsail deployment versus the alternate historical Fargate template. The integrated source passed 160 Python tests, 14 JavaScript tests, Ruff and CloudFormation lint locally on October 26, and source baseline a5c78ed passed GitHub CI. A real hosted DeepSeek review completed six model requests and eight tool calls, passed all six policy checks, and saved a sample human approval. Reload persistence, session isolation and protected shared evidence were verified. The final hosted check also verified that a run-limit error preserves the completed review and saved approval; the footer identifies the recorded survey provider separately. The browser journey covered a new upload, a checked review, saved human approval, and an HTML report with three embedded evidence images. Mobile layout and keyboard navigation were exercised. The full reviewed source is supplied in the judge archive with pinned verified dependencies and a hash-checked downloader for the three public models. The published source baseline preserves the separate October 26 latency-evidence work. This integrated source freshly passed 160 Python tests, 14 JavaScript tests, Ruff, JavaScript syntax checks and CloudFormation lint on the existing Mac environment. No new paid inference was run. Current source verification: https://github.com/himanshu748/marg-ai/blob/master/docs/CURRENT_VERIFICATION.md .

What we learned

Evidence must influence the workflow, not just decorate a model answer. A dismissed observation should remain auditable without continuing to look unresolved. Policy rules shared across planning, tools, and auditing prevent contradictory decisions. Honest provenance is also part of usability: mock review, synthetic GPS, physical measurement estimates, and historical deployment evidence must be visible.

What's next

Improve privacy recall for small plates and non-frontal faces, calibrate severity and area against field measurements, validate real GPS/map matching, clarify dataset rights, and broaden hard-negative evaluation.

We explicitly opt into the Agentic Vision Award, with perception-to-action traces and human control. COOL was not run, and we make no COOL award or matched Graviton cost/performance claim.

Evidence and reproducibility

Source: https://github.com/himanshu748/marg-ai (published source baseline a5c78ed; README-only clarification in master 378665d) Current deployment: AWS Lightsail with optional Modal inference; instructions: https://github.com/himanshu748/marg-ai/blob/master/docs/JUDGE_DEPLOYMENT.md Current source verification: https://github.com/himanshu748/marg-ai/blob/master/docs/CURRENT_VERIFICATION.md ; historical mechanisms and measurements: docs/TECHNICAL_REPORT.md Current deployment diagram: https://github.com/himanshu748/marg-ai/blob/master/docs/current-architecture.png ; alternate Fargate architecture: docs/ARCHITECTURE.md Evaluation: eval/results/ and docs/failures/ Build and tests: README.md and LOCAL_OPERATIONS.md

Live judge demo: https://margai.13.204.212.172.sslip.io/?survey=pothole_kumasi

First use Review activity and Work orders to inspect recorded evidence without inference credits. Optionally choose Open live review, then Run live review to exercise the live agent on a private sample copy. Inspect the tool trace and audit, then approve or reject a recommendation. Decisions do not dispatch repairs. The application enforces a $0.50 lifetime credit reservation, 100 model requests, two review attempts per browser, one concurrent review, and November 11 UTC expiry; exhausted or failed live inference is explicit. Existing Modal additional grants fund inference with no purchase or top-up. The broader upload workflow is reproducible locally from the source archive. The 3:15 replacement video is an edited walkthrough of real hosted browser captures of a fresh DeepSeek review, with model waiting shortened and Deepgram synthetic narration. AWS deployment and processing receipts are included in the judge archive. Bedrock remains disabled and COOL was not run.

Judge video (3:15, unlisted): https://youtu.be/yJ_DOt6JkOA

Narration: Deepgram Aura-2 Thalia synthetic voice, generated from 2,895 authored characters using user-confirmed existing credits with no purchase or top-up. Codex assisted with verification, packaging, deployment and video preparation. The replacement video shows a fresh live DeepSeek run with six model requests, eight tool calls, six passing policy checks and a saved sample human approval. The exact review JSON and provider receipts are included in the updated archive.

Video verification, 26 October: the saved recording is 194.601 seconds. The opening names sole team member Himanshu Kumar; it uses synthetic narration and does not show him on camera. Recorded results appear at 52 seconds (scene starts at 47.28 seconds). Actual frames and the limitation around the rules' team wording: https://github.com/himanshu748/marg-ai/blob/master/docs/VIDEO_VERIFICATION.md .

Built With

Share this project:

Updates

Submission history