Eagle AI

Inspiration

After using popular security systems like Ring, we noticed a large amount of false positive notifications. Eventually, many of us became desensitized to these notifications, making it hard to determine if the alert was for an actual emergency or for something as minor as a stray animal. Receiving notifications such as "There Is Motion Detected at Your Front Door" didn't tell us the severity of the notification. This is a huge problem, especially in home security, where every second counts.

What it does

Eagle AI (Enhanced Automated Guardian for Living Environments AI) uses Gemini to process the feed from a security camera to provide a summary of what it has detect, the suspiciousness of what it has seen, and potential steps based on that. That report is then sent as a text message to the user, who can decide on next steps instead of having to sift through the footage.

How we built it

Part 1: Google Nest API: Using the Google Device Access API, we are able to access the camera feed of a wired Nest camera. We then process the WebRTC stream frames through the YOLOv8n model and analyze whether or not there is a person being detected. If it detects a person, a video batch will be sent to Gemini.

Part 2: Gemini: Once the frames are sent to Gemini, the model will process the frames and also, with a given prompt that asks for a detailed report, it will generate a detailed report regarding the main events, the suspicion level, and recommended steps if needed.

Part 3: Text Message: Once the report is finalized, using Python, we will send a text message of the report to the individual user notifying them of the situation and what the recommended next steps are.

Challenges we ran into

Ensuring Gemini’s outputs were consistent, straightforward, and detailed for the user. Reliable prompt engineering.
Utilizing the WebRTC output from Nest camera in our Python backend. It was difficult to capture the frames from the video stream format.
Sending messages over SMTP servers

Accomplishments that we're proud of

Even though most online forums haven't achieved nest video streams, we developed a streamlined pipeline to analyze and notify the user of suspicious activities, utilizing a multitude of components and libraries.
Demonstration of a viable product that could change the industry

What we learned

Googles API tools and integration to the ecosystem, including Google’s Device Access API, video analysis with Gemini and LLM prompting
Utilization of the WebRTC protocol
MMS gateways and their functioning

What's next for Eagle AI

Addition of a chat feature to gain further insight to the event
A dashboard with events over the past few days
A better integration with the Google Home and Nest ecosystem
Authentication system with firebase
Additional integrations to different use cases including but non limited to: baby monitors, senior citizen safety cameras, garage safety cameras to warn of potential dangerous scenarios before they occur.