Start screen
Video being analyzed by Gemini
Search tab of the app
History tab of the app

Inspiration

We wanted to explore how computer vision can be used in joint with AI like Google Gemini. We also wanted our product to be something that could help people in more way than one which influenced out decision to include different disability aids and assists

What it does

Keeps track of items in a room and when prompted can tell the user where these items were last seen, describe their location to the user, and return clip of where this item was seen.

How we built it

We utilized Claude and Gemini to help develop our Python-based object finder, which uses the Gemini API to identify and track objects in a user's surroundings. We collaborated through a shared GitHub repository, allowing each team member to work on separate components of the project simultaneously. The backend focused on developing and refining prompts for Google Gemini by testing the program with videos of various objects, analyzing the results, and repeatedly improving object descriptions and identifications. while this was going on our don't end was focused on taking that data collected by the backend and displaying it in nice to look at and intuitive UI. We deployed the service using Render, connecting the backend to the web so users can submit videos, have them processed by Gemini, and receive the resulting object identification and tracking information.

Challenges we ran into

We faced a lot of challenges when trying to push and pull from our GitHub repository since this was many of our first times using something like this. It was also very challenging to get out test cases and continually running them many times and then going back and updating our references and instructions. It was a very long process and we wasted quite a bit of time since sometimes one member would be stuck waiting on something from another member and we certainly could've had the project planned out better, but we we're all very happy with how it turned out.

Accomplishments that we're proud of

Training the model to properly identify objects and out user interface.

What we learned

How to work as a group and how to utilize a GitHub repository to streamline the group coding process.

What's next for Lantern

Accounts so users can track their scans and improve the ability to recognize objects in dim lighting and other areas. Train the model based on user habits to better assist them.