Inspiration
Traditional CCTV systems only record events but cannot understand what is happening in a scene. We wanted to build a system that not only detects people but also provides intelligent insights about crowd behavior, safety risks, and unusual activities in real time.
What it does
Multi Modal Video Auditor analyzes live or uploaded video streams using AI. It detects people with YOLOv8, monitors crowd density, identifies unusual crowd changes, and uses Google Gemini to generate contextual safety assessments and risk-level reports.
How we built it
We built the frontend using Streamlit for an interactive dashboard. YOLOv8 was used for real-time person detection and counting, while Google Gemini provided scene understanding and crowd behavior analysis. An anomaly detection engine was implemented to trigger AI analysis only when significant crowd changes occur, reducing unnecessary API usage.
Challenges we ran into
- Optimizing real-time video processing performance.
- Reducing Gemini API calls without losing important events.
- Designing reliable anomaly detection thresholds.
- Managing API quotas and handling rate limits.
- Integrating computer vision outputs with multimodal AI analysis.
Accomplishments that we're proud of
- Successfully combined Computer Vision and Generative AI in a single application.
- Built a real-time crowd monitoring dashboard.
- Implemented anomaly-triggered AI analysis to improve efficiency.
- Created actionable safety insights instead of simple object detection results.
- Developed a scalable architecture suitable for smart surveillance applications.
What we learned
- Practical integration of YOLOv8 with Streamlit applications.
- Using multimodal LLMs for contextual video understanding.
- Event-driven system design and API optimization.
- Real-time data visualization and analytics.
- Building end-to-end AI applications that solve real-world problems.
What's next for Multi Modal Video Auditor
- Multi-camera monitoring support.
- Crowd heatmap and movement visualization.
- Automated alerts via email, SMS, or webhooks.
- Historical analytics and trend reporting.
- Smart city and public event monitoring integrations.
- Edge deployment for faster and privacy-focused processing.
Log in or sign up for Devpost to join the conversation.