Nemo

Nemo view 1 (front, camera, eyes, side, button)
Nemo view 2 (back, clip)
Nemo view 3 (inside, board, camera, mic, speaker, screen)
Grandma is happy she doesn't lose her glasses anymore
Nemo in his home; seeing what you see

Inspiration

Most of us have watched a grandparent spend twenty minutes looking for their glasses, and it’s usually a family member who ends up searching with them. We looked at what already exists and nothing quite fit. Tracking tags need one on every item, and you can’t neatly tag glasses or a TV remote. Smart glasses with a memory feature only remember what you tell them to, which doesn’t help when the problem is forgetting.

We wanted something that notices where everyday things get put down without being asked, and answers out loud when you need it. We call it Nemo, after Mnemosyne, the Greek goddess of memory.

What it does

Nemo is a small camera you clip onto your shirt. As you move around your home, it notices where your everyday things get put down. When you can’t find something, you hold its button and ask out loud, and Nemo answers through its own speaker with where it last saw the item and how long ago.

The person wearing it never needs a screen or an app. Nemo’s display only shows a pair of eyes, open while it’s looking.

Answers come from an NVIDIA Nemotron agent that checks Nemo’s memory, which is kept in SpacetimeDB. If it hasn’t seen something or isn’t sure, it says so.

Nemo also keeps family in the loop. If the pill box hasn’t been used by its usual time, nothing has happened for hours, or the wearer seems to have left the house, a carer gets an iMessage through Photon. It’s a nudge to check in, not a diagnosis.

How we built it

The device. Nemo is a DFRobot ESP32-S3 AI Camera board, with the camera, microphone and speaker amp all on one board, plus a 0.96 inch OLED. It sits in a 3D-printed case and runs off a pocket power bank. The firmware is written in Arduino. It serves camera frames over WiFi and draws the eyes on the screen. Its BOOT button is push-to-talk: hold it to record, release, and the recording is sent to our backend, which sends back a spoken answer for the speaker to play.

Seeing. The backend is Python and FastAPI, running on a laptop on the same phone hotspot. Every second it pulls a burst of three frames from Nemo and keeps the sharpest, then drops anything blurry or nearly identical to the last frame. YOLOE finds our items and the furniture they’re on, using several text descriptions for each. DINOv2 compares the frame with reference frames of each area to work out where the wearer is, and compares each item with reference photos of our own things to tell look-alikes apart. Faces are blurred in the item thumbnails before they’re stored.

Remembering, with SpacetimeDB. Nemo’s memory lives in SpacetimeDB, and the decisions are made inside the database by reducers, not in Python:

Every sighting is recorded.
An item only moves in memory once it has been seen in its new spot twice in a row, which filters out stray detections.
The pill box counts as used when a clear frame shows its spot empty and it later reappears there. Uses are counted by hour, so Nemo learns when the pills are usually taken.
A scheduled reducer runs every minute and decides whether a carer alert is needed.

Answering, with NVIDIA Nemotron. Questions go to an agent built on Nemotron 3.5 Lightning. It has four tools:

find which item someone means, including nicknames like “specs” or “clicker”
look up where it was last seen
check its history
take a closer look at a snapshot

It plans its own steps and calls tools as it needs them. It answers only from what they return, and hedges anything it can’t be sure of. To keep answers quick, when a question names an item, that item is looked up before the first model call.

Nemotron 3.5 Super VL works in the background so it never slows an answer down. It:

describes the exact spot when something is put down
gives a second opinion when vision can’t tell two items apart
answers the agent’s snapshot questions
checks once a minute whether the wearer is indoors or outdoors

Speaking. faster-whisper turns the question into text on the laptop. ElevenLabs speaks the answer, with Piper as an offline fallback.

Texting carers, with Photon. A small Bun service uses Photon’s Spectrum SDK to send carer alerts as iMessages. When SpacetimeDB’s scheduled check raises an alert, the backend sends it and marks it as sent.

Seeing it work. We built a live page that shows what Nemo sees and remembers in real time: each frame with its detections, the area it thinks it’s in, its current memory of every item, and the last question with its answer. We also built a website for Nemo with a 3D model of the finished device.

Challenges we ran into

Our 3D-printed case wouldn’t print. The slicer kept rejecting the design for floating segments, however many supports we added. The people in the 3D printing room were seeing the same problem on other projects, and together we tracked it down to one setting in Bambu Studio. Turning on “support on build plate only” let the case print.

Blurry frames nearly broke vision. On our first walk around the hotel room, 278 of 279 frames were thrown out because they were taken mid-step. We switched to pulling three frames a second from Nemo and keeping only the sharpest.

Answers were too slow to feel natural. Early answers took around 5 seconds, and the first question of a session took up to 13. We warmed up the speech model at startup, cached item data, and looked up the item named in a question before the first model call. That brought the backend down to about 3.8 seconds.

The vision model couldn’t always tell our things apart, and Nemotron’s second opinion sometimes guessed wrong on small, blurry crops. We now only accept its pick when YOLOE or DINOv2 agrees, or when it gives the same answer on another frame.

The hardware had its own quirks. The board’s microphone records very quietly, so we boost the volume before transcribing. The board only works on 2.4 GHz WiFi and can’t join the venue network, so everything runs over a phone hotspot.

Accomplishments that we’re proud of

We turned a bare camera board (one none of us had ever worked with before), into a finished device you can wear, and the full loop works on it: hold the button, ask out loud, and hear Nemo answer from its own speaker in a few seconds.

We’re proud that Nemo is honest. It answers only from what it has actually seen, tells you when it hasn’t spotted something yet, and hedges anything it can’t be sure of instead of guessing. We’re also proud that the memory logic genuinely runs inside SpacetimeDB, not just in Python with the database for storage, including the scheduled check that decides when to text a carer. We’re proud of how we worked. Four of us built the device, vision, memory and agent, and website side by side, agreed on one shared document of interfaces between the parts, and merged everything into a single working system within 24 hours.

What we learned

Most of what went into Nemo was new to us this weekend: building firmware for an ESP32 camera board, open-vocabulary detection with YOLOE, matching images with DINOv2, building tool-using agents with Nemotron, writing logic as SpacetimeDB reducers, and sending iMessages through Photon. Learning them all at once taught us that a camera worn on your chest sees the world very differently from a phone held still, and that accuracy only counted once we tested on Nemo’s own footage in a real room. Careful rules about when to trust what the camera saw did more for us than bigger models. Waiting for two sightings in a row, keeping the sharpest of several frames, and only accepting a second opinion when another signal agrees removed most of the wrong answers. For something you talk to, speed and honesty matter more than detail. A short answer in a few seconds is worth more than a perfect one in ten, and “I haven’t seen it yet” is better than a confident guess. We also learned how to build fast as a team: agree early on how the parts connect, then let each person run with their own part.

What’s next for Nemo

The core of Nemo is built and running together on our hardware: seeing, remembering, answering out loud and texting carers. Looking further ahead, we’d like to:

build the battery into the device so it doesn’t need a power bank
let Nemo find things inside containers, like keys left in a coat pocket or a bag
test Nemo with older people and the families who look after them