Logo

Inspiration

Over the past decade, there has been a growing sense of mistrust between the general American public and local law enforcement. We created VigilAI to bolster trust between the general public and the American police through the means of data-driven machine learning to identify deviations between protocol and on-duty actions. We decided to create a neutral application to assess the constitutionality of actions taken by police officers to reinforce accountability and promote better training practices among officers.

What it does

VigilAI analyzes bodycam footage and provides the following insights:

Creates a personalized dashboard for officers and their respective station managers to view their incident reports generated from bodycam analysis
Identify the magnitude of difference between officers' actions while on duty and the rules mandated by protocol
Enables police station personnel to toggle instances in bodycam footage at which officers deviated from protocol with auto-generated captions and AI-generated incident summaries
Generates a personalized training quiz for officers based on their actions captured on bodycam footage

How we built it

VigilAI was created by combining multi-layered machine learning approaches to achieve a scalable and accurate model that maintains fairness and efficiency.
Speech Diarization and Similiarity AI: We spliced mp4 bodycam footage by finetuning Google Cloud's speech analysis model to separate the officer's speech from suspects and radio interactions for further detailed analysis. The separated audio was then analyzed by an optimized LLM to identify the suspects from the officer using a purely audio-based approach and output a detailed transcript.
Visual Analysis of Bodycam Footage: We analyzed the bodycam footage with Google Cloud's Video Intelligence model to provide annotations of visual stimuli. We then utilized a Sentiment Analysis model from Huggingface to categorize the autogenerated descriptions of the bodycam footage.
Multimodal LLM: We layered the data we collected from the two previous models to create a detailed AI-generated description of an officer's deviations from protocol with associated timestamps, justifications, and relevant laws.
Auto-Generated Training Quiz: We used a fine-tuned LLM to generate personalized training quizzes for officers based on their actions while on duty. Station managers can track the performance of officers on quizzes over time.

Challenges we ran into

Time complexity of speech analysis and multimodal algorithm for large data files
Audio compatibility with Google Cloud models
Hardware-specific module issues