Spots

Inspiration

Instagram helps us keep up with people, but we felt like there was still a gap between seeing someone’s posts and actually connecting with them. We started wondering what it would look like to find shared interests while we’re out doing things, instead of only while scrolling.

Maybe you and someone else stop at the same concert poster or keep going back to the same café. There’s something you could connect over, but you might never know it. That’s where Spots came from: letting people leave a note in that moment so someone else can find it and respond.

What it does

Spots lets you leave notes on physical objects that other people can discover through their glasses.

Say you see a concert poster and leave “Anyone going Friday?” Someone else could look at that poster later, find your note, and reply by speaking through their glasses. You don’t have to be there at the same time for the conversation to start.

You can also ask questions and use voice commands to create, read, edit, or delete notes. For our prototype, we’re using camera-equipped glasses and a browser interface. The Instagram-style screen is just a mock for the demo. The experience we’re working toward would run through Meta Ray-Ban Display.

How we built it

We used React and TypeScript for the interface, FastAPI for the backend, and SQLite to store the objects and their conversations.

For recognition, we used OpenCV with SIFT and ORB to match what the camera sees against reference photos. Once we recognize an object, we use its saved ID to pull up the notes attached to it.

For voice, we record audio in the browser and use Whisper to turn it into text. A local Qwen model running through Ollama handles the conversation and helps figure out what someone is asking. The app handles the actual save or delete, then confirms once it’s done.

Challenges we ran into

One question we spent a lot of time on was whether we should recognize an object on its own or recognize it together with the background.

Using the background gives us more details to match, but that creates another problem: are we recognizing the poster, or just the wall behind it? Focusing only on the object makes more sense if it moves, but gives us less information to work with. We’re still working on getting that balance right.

Voice was another challenge. When people talk, they pause, change their minds, or say the same thing in completely different ways. We had to account for that and figure out when the app should ask a question instead of guessing.

Accomplishments that we're proud of

We’re proud that we got the different parts working together. Someone can speak a note, attach it to an object, and have another person find it later.

Getting beyond basic object labels was a big part of that. Knowing something is a poster doesn’t help much if we pull up the wrong poster’s conversation. We needed to connect what the camera sees to the right saved object.

We also ended up building something we’d want to use ourselves. We liked the idea of finding someone’s recommendation or invitation in real time, rather than hoping it shows up in a feed.

What we learned

We learned pretty quickly that detecting an object and knowing which object it is are two different problems. Getting the label right doesn’t mean you’ve found the right conversation.

We also realized how much the little things matter in a voice interaction. An answer can be correct but still feel awkward because of how it’s worded or how long it takes to respond.

A lot of our decisions came back to asking, “Would someone understand this if we weren’t standing next to them explaining it?” That helped us think more about how people would actually use Spots.

What's next for Spots

The first thing we want to do is bring Spots onto Meta Ray-Ban Display. Looking at something, finding a note, and replying through your glasses without taking out your phone is the experience we’ve wanted to build from the start.

After that, we want to test it in places like cafés and community spaces. We’ll watch how people use it, learn where they get stuck, and see whether the notes actually lead to conversations or plans.

We also want to explore integrations with apps like Beli, where people track and share restaurants with friends. Imagine looking at a restaurant’s menu through your glasses and seeing what your friends recommend, then leaving your own note after trying it. That’s the kind of integration we’d like to explore.

With permission, we could also help people connect over shared interests and places they enjoy. We’d start small and use what we learn from those early tests before rolling Spots out more widely.