Inspiration

For blind and low-vision people, small everyday things take a lot of effort: finding where you put your keys, getting across a room without walking into something, or spotting a friend in a crowd. Canes and guide apps help with some of it, but nothing we found handled all three in one hands-free tool you just talk to. We wanted something that feels like a friend walking beside you, describing the world out loud.

What it does

Nova is a voice assistant that sees through a phone worn on your chest. Remembers your things. It watches for your keys, wallet, phone, bottle and other everyday items, and remembers where it last saw them.

Ask "Where are my keys?" and it answers "On the wooden table, next to the laptop. Seen 3 minutes ago."

Never makes up a location. Every sighting gets double-checked. If the item isn't really in the photo, Nova says it isn't sure instead of guessing.

Warns you about obstacles, even walls. It names what's ahead ("Chair on your right, 1.7 metres") and tells you to stop when something is under a metre away.

Finds your friends. Say "Find my friend in the red hoodie", and Nova spots people, reads what colour they're wearing, and guides you to them by voice.

Describes the scene. "What's in front of me?" gets a short spoken description.

Companion app. A live dashboard for caregivers shows what Nova sees, where your belongings are, how clear the path is, and the conversation, all centered on Nova's animated orb.

How we built it

Vision: two YOLOv8 models run side by side, a stock one for everyday objects and people, and one we trained on keys, wallets and phones. We trained ours on Roboflow datasets in Google Colab, then fine-tuned it on our own photos taken from the chest camera. Depth: Depth Anything V2 (metric, indoor) estimates how far away everything in the walking path is. That's how Nova sees walls and doors, which an object detector can't. Voice: Whisper turns speech into text offline on the laptop, an offline question parser works out what you asked, and text-to-speech answers out loud. Gemini: describes the scene and double-checks saved sightings in the background, so answers come back instantly. OpenCV: colour analysis of shirts, direction and distance estimates, and a frame reader that always uses the newest frame so the view never lags. App: a Flask backend with a hand-built HTML/CSS/JS dashboard. The phone streams over wifi with IP Webcam.

Challenges we ran into

Recognising keys. Our first model, a "type any object" detector, called keys "glasses" and couldn't tell a black wallet from a phone. Training our own model on internet photos scored well on tests but still failed on our real objects. We only fixed it by adding our own photos from the chest camera. Hitting the API limit. Partway through, our Gemini key hit its daily limit, and every voice question failed. That pushed us to move speech recognition fully offline with Whisper, which also made it faster. Voice glitches. The mic was cutting off the first words of questions, so "my keys" came out as "Mikey's". The speech engine went silent after its first sentence. And Nova was hearing its own obstacle warnings while you asked something. We fixed each one: the mic now switches on before the beep, Whisper gets example questions to listen for, the speech engine restarts for every sentence, and warnings pause while you talk. False sightings. The detector sometimes "saw" a laptop that wasn't there. We added Gemini as a fact-checker, so Nova admits when it isn't sure. Hardware. Our ultrasonic sensor needed 5 V but our board runs at 3.3 V, and the wifi wristband's USB chip needed a driver we couldn't install in time. So we moved obstacle detection onto the camera with a depth model, which, unlike the ultrasonic sensor, can also say what is in the way.

Accomplishments that we're proud of

On our own test photos, fine-tuning took our detector from 0.00 to 0.99 accuracy (mAP50) on phones, 0.19 to 0.54 on wallets, and 0.00 to 0.43 on keys. Item finding, obstacle warnings, the friend finder and voice all run offline. Gemini only adds extra detail when there's internet. Honesty built in: Nova tells you when it isn't sure instead of inventing an answer, which matters when someone depends on it. Nova answers a voice question in about 2 seconds. What we learned A model that scores well on internet photos can still fail in the real world. A few dozen photos from your actual camera can matter more than hundreds of generic ones. Building for people who can't see the screen changes everything: timing, silence, and not talking over the user matter as much as accuracy. Plan for failures, like API limits, flaky wifi and hardware, and make sure something sensible happens when they occur.

What's next for Nova

A wristband with a button to talk and a vibration motor for quiet warnings. The ESP32 firmware is already written. Running on the phone alone, without a laptop. More items: training on medicine bottles, remotes and anything else people ask for. Personal recognition: finding a specific friend by face, with their permission, and not just by shirt colour. Testing with blind and low-vision users to tune what Nova says and when.