Local AI Dashcam Incident Explainer

Offline AI that detects dashcam collisions with YOLO, narrates scenes via local VLMs, blurs faces/plates for privacy, and exports insurance-ready PDF reports — no cloud needed.

Comment

Devpost Submission — Local AI Dashcam Incident Explainer

📝 About the Project (copy the markdown below)

Inspiration

Road accidents kill over 1.35 million people annually worldwide, yet most dashcam footage sits unanalyzed on SD cards — requiring hours of manual review to identify what happened, who was at fault, and what to do next. Existing solutions rely on expensive cloud APIs, raising serious privacy concerns when sensitive footage (faces, licence plates, locations) is uploaded to third-party servers.

We asked: What if your dashcam could explain an accident to you — instantly, intelligently, and without your video ever leaving your machine?

That question became Local AI Dashcam Incident Explainer — a fully offline, privacy-preserving AI pipeline that transforms raw dashcam video into structured, insurance-ready incident reports using on-device computer vision and language models.

What It Does

Upload a dashcam clip, and the system autonomously:

  1. Detects incidents — YOLO11 object detection + SORT tracking + Farneback optical flow identify the exact collision moment (±5 seconds) from hours of footage
  2. Narrates the scene — MiniCPM-V (8B vision-language model via Ollama) analyzes keyframes and produces structured JSON: vehicles involved, fault analysis, severity, road conditions, and a chronological timeline
  3. Generates prose reports — Mistral 7B synthesizes the structured data into professional, human-readable insurance narratives
  4. Enforces privacy — Faces (via SSD ResNet10) and licence plates (via YOLO heuristic ROI) are automatically blurred before export
  5. Grounds with real-world data — SerpApi integration fetches relevant traffic statutes, nearby emergency/towing services, and OEM parts pricing via Google Search, Maps, and Shopping APIs
  6. Exports insurance-ready PDFs — One-click download of a polished PDF with severity badges, blurred keyframes, prose report, and structured metadata

All of this runs 100% locally on a MacBook or Linux PC — no cloud, no internet, no data leakage.

How We Built It

The system is architected as a 5-phase pipeline, each phase feeding the next:

Phase 1 — Keyframe Extraction

  • FFmpeg extracts frames at configurable FPS
  • A smart mode uses frame-differencing motion scores to skip static scenes (red lights, parking), dramatically reducing compute

Phase 2 — Incident Detection

  • YOLO11 detects vehicles, pedestrians, motorcycles, and bicycles
  • SORT tracker (Kalman filter + Hungarian algorithm) assigns persistent IDs across frames
  • Farneback optical flow computes dense per-pixel velocity vectors with rolling-window spike detection
  • Fusion trigger: If two tracked objects converge (IoU > 0.15) AND an optical flow spike occurs simultaneously → incident confirmed → FFmpeg slices a ±5s clip

Phase 3 — VLM Narration

  • MiniCPM-V 2.6 (via Ollama) receives base64-encoded pre-impact, impact, and post-impact keyframes
  • A structured prompt forces valid JSON output: vehicles, fault_analysis, severity, conditions, timeline
  • Confidence flagging detects hedging language ("might", "possibly") and warns the user
  • Mistral 7B converts the JSON into a polished prose narrative

Phase 4 — Privacy & Export

  • Face blurring: OpenCV DNN (SSD ResNet10) with Haar cascade fallback
  • Licence plate blurring: YOLO vehicle crops → heuristic ROI mapping → Gaussian blur (no separate ALPR model needed)
  • ReportLab generates professional PDFs with auto-coloured severity badges and embedded keyframes
  • Gradio powers a drag-and-drop web UI at localhost:7860 with real-time processing logs

Phase 4b — SerpApi Web Intelligence

  • Google Search API → Traffic law statutes and court precedents matching the detected fault
  • Google Maps/Local API → Nearest 24/7 towing services and trauma centers
  • Google Shopping API → OEM replacement part pricing for damaged vehicles

Phase 5 — Evaluation

  • BLEU-4 and ROUGE-L scoring against human-annotated references from CCD, DoTA, and DADA-2000 datasets
  • Ablation studies across 1/3/5 keyframe configurations

Challenges We Faced

  • Running VLMs locally: Getting MiniCPM-V to produce consistently valid JSON from visual input required extensive prompt engineering. The model would frequently hallucinate extra fields or break JSON syntax — we solved this with strict output format enforcement and post-processing validation
  • Incident detection precision: Naive collision detection (just IoU overlap) produced too many false positives. Fusing it with optical flow velocity spikes dramatically improved precision without sacrificing recall
  • Privacy without ALPR: Dedicated licence plate recognition models are heavy (~500MB+). We engineered a lightweight heuristic that uses YOLO's existing vehicle bounding boxes to estimate plate regions — achieving robust blurring at zero extra model cost
  • Real-time UI feedback: Gradio's generator-based yielding pattern was tricky to get right for streaming logs, gallery updates, and PDF downloads simultaneously across a multi-step pipeline
  • Balancing offline-first with web intelligence: The SerpApi grounding module needed graceful degradation — the entire system must work perfectly offline, with web intelligence as a pure enhancement layer

What We Learned

  • How to orchestrate multi-model AI pipelines (YOLO + SORT + optical flow + VLM + LLM) into a cohesive product
  • The importance of privacy-by-design — blurring faces and plates isn't an afterthought; it's architecturally baked in before any data is persisted or exported
  • That local AI is production-viable — MiniCPM-V and Mistral 7B via Ollama run comfortably on a MacBook with 8GB RAM
  • How to use SerpApi's multi-engine approach (Search + Maps + Shopping) to ground AI outputs in real-world data
  • Practical experience with NLP evaluation metrics (BLEU, ROUGE) for measuring AI-generated report quality against human baselines

What's Next

  • Real-time dashcam streaming — Process live video feeds instead of uploaded clips
  • Multi-camera fusion — Correlate incidents across front/rear/side dashcam angles
  • Fleet dashboard — Aggregate incident analytics across vehicle fleets
  • On-device mobile deployment — Port the pipeline to edge devices for in-car processing

🏷️ Built With (copy these as tags)

Python, YOLO11, Ollama, MiniCPM-V, Mistral 7B, OpenCV, FFmpeg, Gradio, SerpApi,
ReportLab, NumPy, SciPy, SORT, Kalman Filter, Optical Flow, Ultralytics,
NLTK, ROUGE, Docker, Pillow, Google Search API, Google Maps API, Google Shopping API

Count: 23 tags (within 25 limit)

Individual tags for easy copy-paste:

  1. Python
  2. YOLO11
  3. Ollama
  4. MiniCPM-V
  5. Mistral 7B
  6. OpenCV
  7. FFmpeg
  8. Gradio
  9. SerpApi
  10. ReportLab
  11. NumPy
  12. SciPy
  13. SORT
  14. Kalman Filter
  15. Optical Flow
  16. Ultralytics
  17. NLTK
  18. ROUGE
  19. Docker
  20. Pillow
  21. Google Search API
  22. Google Maps API
  23. Google Shopping API

Built With

Share this project:

Updates

Submission history