Sentry

Sentry Landing Page
Live Drone View Dashboard
Drone View Processed with LLM Data

Inspiration

Our project was inspired by Andrey Liscovich from the Ukraine Defense Fund and the challenges faced by modern U.S. and Ukrainian military forces in anomaly and threat detection across wartime, maritime, and cross-border scenarios. In today's geopolitical landscape, border security is a top priority for many agencies—not only on land but also at sea.

What It Does

We developed a drone sentry system that integrates both camera and sonar inputs for above- and below-water applications. The camera system utilizes a fine-tuned YOLOv8 model for real-time object detection, while the sonar system employs a custom Support Vector Machine trained on underwater sonar representations of objects.

Both inputs feed into our agentic workflow, which leverages Google Gemini to build a chain of reasoning and action agents. This system generates response plans for the appropriate agencies and can automatically contact the relevant agency hotlines. Our function tools are designed for seamless integration into agency codebases.

Additionally, we built a real-time monitoring dashboard where control room personnel can view drone footage and prioritize threats based on severity levels. The system also counts and tracks people or objects involved in an activity, customizing action plans accordingly to ensure response teams are appropriately prepared.

How We Built It

For the frontend, we used React and Next.js to structure our framework, connecting it to a Python-based backend via Flask API servers. Authentication is handled through Auth0.

Our application integrates several technologies, including:

Google Gemini API and Modal for faster and more efficient code execution.
A fine-tuned YOLOv8 model trained with additional images of sea vessels and submarines, enabling it to detect unregistered ships and underwater mines.
Libraries like NumPy, Pandas, and Matplotlib to train a sonar data model using Kaggle datasets. This model processes sonar sensor data from submarines to differentiate between natural objects (e.g., rocks) and threats (e.g., mines), helping chart safer navigation routes.
A LLaMA 70B model, deployed via Modal, to generate more precise responses in emergency situations and determine which agencies to contact.
Cloudflare for local hosting on the tech domain.

Challenges We Faced

One of the biggest challenges was working with Modal—debugging and optimizing it to efficiently host LLaMA and YOLOv8 on high-performance GPUs. We also encountered difficulties in converting video codecs for frame-by-frame analysis with YOLOv8.

Another significant hurdle was using Cloudflare to locally host the website on macOS while ensuring proper domain visibility.

Accomplishments We're Proud Of

We successfully developed a multi-modal agent workflow and built custom AI-based frameworks that are scalable and adaptable. Our YOLOv8 dataset can be expanded to detect pirate activity, maritime threats, and illegal immigration attempts.

Additionally, we implemented an accurate agency-suggestion and reporting system, integrating Twilio for immediate agency contact.

What We Learned

This hackathon reinforced the importance of collaborative teamwork, especially when integrating multiple complex components. We also gained hands-on experience with new technologies, including Gemini API, YOLOv8, and custom model training. Achieving 91% accuracy with our Support Vector Machine model was a particularly rewarding milestone.

What's Next for Sentry

Our goal is to develop a multi-agency platform where security organizations can collaborate, centralize threat data, and coordinate responses. We aim to expand the use of drone data beyond aerial applications to include underwater surveillance, creating a unified intelligence hub for border protection.