Watchful.AI

Landing Page
Features
Design System
Architecture

📞 The problem at hand.

How might we leverage an alert agent to ensure 24/7 student safety from school shootings?

🗝️ Where did this stem from?

School shootings. There are 2 per week…and this number is NOT decreasing. Students are now worried about their safety but have no way to indicate potential danger. Just two days ago, a suspected shootout at UPenn left witnesses terrified, as suspects made sounds resembling gunshots while exiting. This ongoing sense of fear and uncertainty is unacceptable, especially when lives are at risk.

💻 What does Watchful.AI do?

Instant alert of weapons and suspicious activity, with the versatility and intelligence of GPT-4o
Search for and track suspicious activity through video semantic search
Monitor multiple video feeds and visualize position of threats in real time
Alert local security to resolve suspicious activity

How does it work?

GPT-4o is used in harmony with its distant cousin, CLIP, to progressively distill gigabytes of simultaneous video footage into key anomalies and threat events, which can be efficiently actioned on by a human in the loop. Utilizing CLIP also has the added benefit of enabling semantic search over CCTV footage, so officers can spend less time scrolling through reels of video footage.

🛠️ How we built it

UI/UX Design We used Figma and FigJam to lay out the user flow, using research from notable newspaper stations like The Daily Pennsylvanian and CNN to support our visual designs and design system with data.

Frontend Next.js, ShadCN, Tailwind CSS, MappedIn

Backend CLIP (PyTorch + Mac M2), GPT-4o, ChromaDB, FastAPI, OpenCV

🕵🏼 Challenges we ran into

A challenge we ran into was collecting data and mapping out the Penn Engineering Building manually. Many tools we wanted to use were unavailable, but we ended up contacting Mappedin and learning their software to create this crucial component of our product

Another significant challenge was utilizing multithreading to processes multiple video streams at a time on a single device. The backend has to ingest, embed, analyze, and serve huge amounts of data.

✔️ Accomplishments that we're proud of

We are proud to have built an impactful and well flushed out product in 36 hours! We were able to combine the knowledge of both our frontend team and backend team to work to our strengths and produce a fully functional tool that can be used to prevent an urgent and widespread issue – school shootings.

We were also proud to implement a new technique for cheap but accurate inference on large amounts of video data. Obviously calling GPT-4 for every frame of multiple video streams would be highly cumbersome, but by prescreening with embeddings (which run for free on local GPUs), then checking with GPT-4o, we get the accessibility of on-device AI with the power of huge hosted models.

☕️ What we learned

We learned that many students are anxious about school shootings in the US considering there’s an average of 2 shootings per week. Surprisingly, there aren’t any publicly available tools to alert citizens about potential shootings accurately, which is quite concerning. We also learned that we are able to create that tool we wished existed to ensure our safety. Learning how to use MappedIn was something new were discovered to do so.

⏭ What's next for Watchful.AI

We strive to implement Watchful.AI in every security system in the US to watch for potential shootouts and suspicious activity. We want to further develop the accuracy of our cameras and description search to ensure safety of not just students, but all citizens in the US.