Inspiration
One of our team member's relatives suffer from Alzheimer’s disease. Seeing firsthand the memory issues that they face, we were inspired to create a project to help them keep track of what they had done throughout the day and who people they meet are.
What it does
We built an IOS app and a backend agentic AI service that constantly watches what an Alzheimer’s patient's actions and surroundings. It then records all important keypoints and records important faces that it seens. It then reminds the user if they miss something (ie: leaving the stove on, faucet on, forget to take pills) and acts as a recall for anytime the user may have forgotten what they did.
How we built it
User Facing: IOS app built in swift using the AVFoundation TTS and STT synthesizer. The IOS app records the camera and sends the frame to the cloud server. It also runs speech to text to listen for a keyword then it forwards queries to the backend to pull from the connected storage system Backend: We used the Gemma3:4b model to run inference on the initial frames received from the client. If an object we deem to be important is found, it is sent to a set of agentic models that create a on-demand detailed schema for the item based on what it deems is important (ie: color, size location, etc). It is then stored in a new hyperlinked-based vector database (https://arxiv.org/html/2502.12110v1) that can automatically create connections between different ideas.
Challenges we ran into
- No cloud providers have GPUs anymore, we ended up finding a peer-to-peer GPU sharing service finally to run on our model on
- Running lighter-weight models caused an issue with limited context size, so we sometimes used Chinese characters and embedded the prompt into the image
- Our phone camera picture size was over 25mb uncompressed (WOW) but any compression led to a decline in performance for facial recognition, so we has to selectively compress the file in certain situations
- Reading through and understanding a complex research paper on a novel agentic storage mechanism
Accomplishments that we're proud of
- It works?
- Running multiple AI agents that work in unison
What we learned
- Working with AI agents to split up complex tasks using limited hardware resources
- IOS app development in SWIFT
- Alternative storage options for LLMs
What's next for HawkEye
- team getting breakfast and sleep
- working on LIDAR integration so we have better context awareness
- more agentic integration for better context awareness
Log in or sign up for Devpost to join the conversation.