Inspiration
Security cameras record what happens, but someone still has to watch the footage and decide what matters. However, sometime mistakes happen and we miss certain events.
I built VIGIL to explore how AI could help minimize and prevent these anomalies. My goal was a system that considers events over time, explains its assessment, and speaks when an incident needs attention.
What it does
VIGIL analyzes live webcam feeds and uploaded videos through a two-stage pipeline. NVIDIA Cosmos Embed1 screens short video windows for unusual activity, then NVIDIA Nemotron 3 Nano Omni reviews any flagged windows to assess the situation, while minimizing compute.
The dashboard combines observations, risk scores, and timestamped evidence into an incident timeline. It draws person and vehicle bounding boxes, retains recent critical context, and uses ElevenLabs to deliver spoken dispatches without repeating an alert for every interval.
How I built it
I built the workspace with Next.js, React, and TypeScript, using server-side API routes to connect the model services.
NVIDIA Cosmos-Embed1 runs locally through a Python FastAPI service with PyTorch and CUDA. It screens overlapping sequences of frames and identifies candidates for further review.
NVIDIA Nemotron 3 Nano Omni performs the multimodal assessment. I integrated an OpenAI-compatible endpoint and also set up a self-hosted, quantized version through llama.cpp on Google Colab, connected to the application through an authenticated ngrok tunnel.
TensorFlow.js and COCO-SSD provide browser-based person and vehicle detection, while MediaPipe supplies separate posture cues. Finally, the ElevenLabs API synthesizes incident dispatches, with browser speech available as a fallback.
The application separates video playback from analysis, so reviewing footage remains responsive while model requests run.
Challenges I ran into
The biggest challenge I faced had to be getting accurate results, especially with my training dataset (Anomaly-Detection-Dataset-UCF) consisting of very low quality videos. Recognizing a confrontation does not necessarily mean the model can reliably identify a small object in someone’s hand. That pushed us to improve frame sampling and make uncertainty visible.
Another challenge was maintaining context. A model could identify a threat in one window and describe ordinary activity in the next simply because the weapon or confrontation was no longer visible. I added recent incident context to subsequent reviews and separated the current window’s observations from the overall clip assessment.
Rate limits and inference latency also affected coverage. I separated Cosmos screening from the slower Nemotron review queue, retained completed results, and added retry handling so a failed request did not erase progress or become a false “safe” verdict.
Accomplishments that I'm proud of
I connected local screening, multimodal review, visual evidence, and voice dispatch into one working application for both live and recorded footage.
I also got Cosmos running on a local GPU and connected a self-hosted Nemotron endpoint, giving us an alternative to shared API quotas.
I was also proud of the significant level of data I was able to keep local and private. With the rise of mass surveillance companies, people are becoming more and more aware of the importance of their data rights, being able to keep this data as local as possible minimizes the data of yours given away
Most importantly, I made the system’s decisions inspectable. Users can see when an event occurred, read the supporting observations, distinguish screened footage from reviewed footage, and compare interval assessments with the final clip review.
What I learned
A reliable video analysis system needs more than a capable model. Sampling, temporal context, request scheduling, and interface wording all affect how useful its output becomes.
I learned to separate severity from confidence: an event can have serious consequences even when the image is imperfect.
Finally, successful inference is not the same as validated accuracy. Real evaluation needs labeled incidents, ordinary lookalikes, and an honest record of missed detections and false alarms.
What's next for Vigil
My next priority is overall improving my quality of data. I want to train the model to recognize more objects, people, and situations, while also becoming more accurate. Part of that stems to improving off the original flagging model in cosmos, while the other obviously stems from obtaining more data.
I also plan to improve small-object recognition with targeted crops, evaluate dedicated weapon detectors, and strengthen incident tracking across longer recordings.
Lastly, I want this tool to also be used for more niche helpful cases. This would include things like searching for a missing car, kid, dog, etc. This system could be paired to AMBER alerts to help locate a child of matching descriptions.
Log in or sign up for Devpost to join the conversation.