Snapspot

App UI

Millions of people who are blind or have low vision navigate the world without a quick way to know what's around them in the moment. We wanted to build something that could turn a phone's camera into a second set of eyes and instantly describe objects and surroundings out loud, without needing to type a question or wait for a manual search.

What it does

Snapspot lets a user point their phone's camera at their surroundings and hear a spoken description of what's there in real time. The app continuously captures frames from the live camera feed, sends them to a backend object-detection service, and speaks back a natural-language description like "I see a dog and a person" using on-device text-to-speech.

How we built it

Frontend: Built with Expo and React Native, using expo-camera for the live camera feed and expo-speech for on-device voice output. Backend: A Python Flask server that receives image frames over HTTP, runs them through a YOLOv8 object detection model, and returns structured detection results as JSON. Connecting them: The app periodically captures a frame, sends it to the Flask server as a multipart form request, and the server responds with a plain-language description that's spoken aloud on the phone. Version control: The team collaborated using a shared GitLab repository, working across separate frontend and backend workstreams that were integrated together.

Challenges we ran into

Getting a phone (running the Expo Go app) to talk to a laptop server on the same network including CORS configuration, finding the right local IP address, and handling firewall/network quirks. Coordinating parallel development: the object-detection team and the app team built their pieces independently, which meant agreeing on a clear "contract" (a consistent JSON shape) so the two sides could be integrated smoothly once both were ready. Debugging cross-tool issues like merge conflicts, missing files between teammates' local setups, and dependency mismatches while working under time pressure. Balancing responsiveness against server load: sending every single camera frame would overwhelm the detection pipeline, so the app throttles how often it captures and sends frames. Accomplishments that we're proud of Successfully connecting a real-time mobile camera feed to a live object-detection backend, end-to-end, from a mobile app to a Python server and back. Getting the app to speak natural descriptions aloud automatically, with no button presses required — a meaningful accessibility-first design choice. Building a flexible backend that supports multiple modes (describing a scene, counting objects, filtering for a specific object, and identifying the most prominent object).

What we learned

How to structure a client-server architecture where two teams can build independently against a shared, agreed-upon data contract. Practical experience with real-time mobile networking constraints — IP addressing, CORS, and Wi-Fi network requirements that don't come up in typical local development. The importance of designing for accessibility from the start, rather than as an afterthought