Local AI Dashcam Incident Explainer
Inspiration
Road accidents cause devastating human and financial consequences, yet dashcam footage often requires hours of manual review to reconstruct what happened. Reviewing collisions, identifying involved vehicles, and preparing incident documentation can be time-consuming. Meanwhile, uploading sensitive footage to cloud services introduces privacy concerns involving faces, licence plates, and location data.
We asked: What if your dashcam could help explain an incident without sending your video to the cloud?
That question inspired Local AI Dashcam Incident Explainer, a privacy-first application that combines computer vision and locally running language models to analyze dashcam footage and generate structured incident summaries and reports.
What It Does
Upload a dashcam video, and the application can:
- Detect potential incidents: Combine YOLO11 object detection, SORT tracking, and Farneback optical flow to identify candidate collision moments for review.
- Analyze the scene: Use MiniCPM-V through Ollama to interpret selected keyframes and generate structured information about vehicles, road conditions, severity, and the incident timeline.
- Generate readable reports: Use Mistral 7B to turn structured observations into a human-readable incident narrative.
- Protect privacy: Blur detected faces and estimated licence plate regions before exporting report imagery.
- Provide optional web intelligence: When internet access and API credentials are available, retrieve potentially relevant traffic regulations, nearby emergency or towing services, and replacement-part price references.
- Export reports: Generate PDF reports containing incident summaries, selected keyframes, severity indicators, and structured metadata.
Privacy-first by design: Video analysis and language-model inference are designed to run locally. Optional web lookups require internet access and transmit search queries to the relevant external service. The application should disclose this distinction clearly to users.
How We Built It
The application follows a modular, five-phase processing pipeline.
Phase 1 — Keyframe Extraction
- FFmpeg extracts frames at a configurable sampling rate.
- An optional motion-scoring mechanism reduces redundant processing by skipping relatively static portions of footage.
Phase 2 — Incident Detection
- YOLO11 detects supported road users, including vehicles and pedestrians.
- SORT uses Kalman filtering and assignment algorithms to maintain object tracks across frames.
- Farneback optical flow estimates motion between consecutive frames.
- A fusion mechanism combines object proximity, tracking information, and motion anomalies to identify potential incidents.
- FFmpeg extracts a configurable time window around candidate events for further analysis.
The system flags potential incidents for review; object overlap and motion anomalies alone do not establish that a collision occurred.
Phase 3 — Vision-Language Analysis
- MiniCPM-V, running through Ollama, analyzes selected pre-event, event, and post-event keyframes.
- Structured prompts request JSON containing vehicle descriptions, observed conditions, a chronological timeline, and a preliminary severity assessment.
- Output validation checks JSON structure and required fields.
- A confidence-warning mechanism flags uncertain or speculative descriptions.
- Mistral 7B converts the structured observations into a readable incident narrative.
Fault assessments are preliminary AI-generated interpretations, not definitive determinations of legal liability.
Phase 4 — Privacy, Web Intelligence, and Export
- OpenCV-based face detection identifies regions for blurring.
- A licence-plate detection or localization mechanism identifies candidate plate regions for privacy processing.
- ReportLab generates downloadable PDF reports with selected keyframes and structured findings.
- Gradio provides a local browser interface with video upload, processing logs, results, and report downloads.
- Optional SerpApi-powered lookups retrieve traffic-law references, nearby services, and indicative parts pricing when connectivity is available.
The core analysis pipeline is designed for offline operation. Web intelligence is an optional enhancement and is unavailable without connectivity.
Phase 5 — Evaluation
- Evaluate incident-detection precision and recall against appropriately annotated video data.
- Measure event-localization error relative to annotated incident timestamps.
- Evaluate generated narratives using suitable text-overlap metrics, such as ROUGE-L, alongside human assessment.
- Conduct ablation experiments comparing different keyframe sampling configurations.
- Measure processing time, memory consumption, and privacy-filter performance on supported hardware.
Dataset usage, licensing, annotation availability, and experimental results should be documented before making performance claims.
Challenges We Addressed
- Structured multimodal output: Vision-language models can return malformed JSON or introduce unsupported details. Schema validation and explicit uncertainty handling help improve reliability.
- False-positive incident detection: Object overlap alone is insufficient. Combining tracking and motion information can improve candidate-event detection, subject to empirical evaluation.
- Privacy-preserving image processing: Face and plate localization must work across changes in lighting, camera angle, resolution, and motion blur. Detection coverage needs testing before claiming complete anonymization.
- Interactive processing: A multi-stage pipeline requires clear progress reporting, error handling, and reliable delivery of generated reports.
- Offline-first architecture: Optional external data services must fail gracefully without interrupting local video analysis.
What We Learned
- How to combine object detection, multi-object tracking, optical flow, and local language models into one processing pipeline.
- Why privacy controls should be integrated into report generation rather than treated as an afterthought.
- How to design structured model outputs with validation and explicit uncertainty indicators.
- How to separate offline processing from optional network-dependent services.
- Why evaluating incident detection, narrative quality, and privacy filtering requires different metrics and test datasets.
What's Next
- Real-time processing of live dashcam feeds.
- Multi-camera incident reconstruction.
- Fleet-level incident analytics and reporting.
- Deployment on edge devices and compatible mobile hardware.
- Improved licence-plate localization and measurable privacy-filter coverage.
- Better incident localization, calibrated confidence estimates, and human-reviewed report validation.
Built With
Python, YOLO11, Ollama, MiniCPM-V, Mistral 7B, OpenCV, FFmpeg, Gradio, SerpApi, ReportLab, NumPy, SciPy, SORT, Kalman Filter, Optical Flow, Ultralytics, NLTK, ROUGE, Docker, Pillow, Google Search API, Google Maps API, Google Shopping API.
Log in or sign up for Devpost to join the conversation.