Phone UI
Spot objects
Last Seen
Phone Camera

Inspiration

It's Halloween and Tony Stark is getting ready to go out as Iron Man, except he can't find his gauntlet. We figured if anyone should never lose anything, it's the guy with an AI running his house. Most of us lose our keys or AirPods at least once a week, so we wanted to build the J.A.R.V.I.S. feature we'd actually use.

What it does

JARVIS takes a photo of the room every few seconds. When you ask "Jarvis, where are my keys?", it looks back through those photos, finds the last one your keys showed up in, and tells you out loud where they were and how long ago. It also shows you the photo so you can check for yourself.

How we built it

A Python script uses OpenCV to grab a webcam frame every couple of seconds. Most frames are nearly identical, so we compare each one to the last saved frame and throw it away if nothing changed. That cut down how many images we had to deal with by a lot.

Each saved frame gets turned into a CLIP embedding. When you ask about an item, we embed the item name the same way and pull the closest matching frames. Out of those, we take the most recent one, since you care about where something is now, not where it was an hour ago. That frame goes to a vision language model, which describes where the item is compared to things around it.

The frontend is a simple web page. It uses the browser's speech recognition for the question and text to speech for Jarvis's answer.

Challenges we ran into

Our first idea was to send every photo to the vision model when someone asked a question. That was way too slow and would have burned through our API credits, so we switched to using CLIP to narrow things down first and only sending a few frames to the model.

We also had a bug where Jarvis would confidently point to where something used to be. Picking the closest match isn't the same as picking the latest sighting, so we had to change how we ranked results.

Small objects were harder than we expected. Common object detection models don't know what keys or earbuds are, which is part of why we went with CLIP instead.

Accomplishments that we're proud of

We got the whole thing working end to end in one afternoon, from the camera to the spoken answer. Watching it find something we'd moved a few minutes earlier was a good moment. We're also happy that the photos stay on the laptop and only a few get sent out when you actually ask a question.

What we learned

We learned how to use embeddings for image search and how to combine that with a vision model without making everything slow. We also learned that deciding what to skip mattered as much as the model we picked. And we got a lot better at cutting features when the clock was running down.

What's next for JARVIS where is my stuff?

We'd like to support more than one camera so it can keep track of things across rooms. Putting the camera on something wearable, like smart glasses, would mean it remembers wherever you go. We also want it to warn you if you're about to leave without your wallet, draw a box around the item in the photo, and eventually run fully on device so nothing leaves your computer.