posted an update —

How we built it

We used React and TypeScript for the interface, FastAPI for the backend, and SQLite to store the objects and their conversations.

Recognition happens in two stages. First, OpenCV with SIFT and ORB compares the webcam frame against our saved reference photos and narrows them down to a few likely candidates. This step is fast and good at matching specific printed objects, but it can be fooled by a matching background or a similar-looking poster.

Then Muse Spark makes the final call. We send it the live frame alongside the candidates’ reference photos and ask whether the camera is showing the same physical object as one of them, returning a structured answer with a confidence score. If Muse isn’t confident, or picks none of them, Spots says it doesn’t recognize the object instead of pulling up the wrong conversation. Once an object is confirmed, we use its saved ID to load the notes attached to it.

For voice, we record audio in the browser and use Whisper to turn it into text. A local Qwen model running through Ollama handles the conversation and figures out what someone is asking. The app handles the actual save or delete, then confirms once it’s done.