Resonate

Inspiration

There are 43 million blind people and 295 million people with moderate to severe visual impairment worldwide. Unfortunately, much of our current infrastructure reflects systemic ableism, since it was not designed with people with disabilities in mind. Visual impairment is also more common than people think. For many, vision starts to decline in their 40s, and age-related conditions like cataracts can make everyday tasks harder over time, while others lose much more.

What is Resonate?

Resonate is an iOS app that helps blind and low-vision people get around on their own. It plans accessible walking routes, warns you about obstacles using the camera outside and LiDAR inside, and describes the room in front of you. You choose how it talks to you: voice, vibration, or tones.

Key Accessibility Design Choices

During our research phase, we talked to visually impaired people and learned how they use their phones. Many rely on VoiceOver to read text aloud, and things like big buttons, dark mode, and larger text make a big difference. We built all of these into our design. In addition, we added an AI agent that can control the app without the user having to touch any buttons using voice.

Tech Stack

App: Swift, SwiftUI
Obstacle Detection: YOLO, CoreML, LiDAR, RoomPlan
LLM APIs: Gemini API, Claude API
Routing: Valhalla, OpenStreetMap
Cloud: Stadia
Voice: Eleven Labs
Backend: Supabase, Auth0

Challenges we ran into

Our first version of obstacle detection picked up too many objects. Users were getting overloaded with alerts and ended up confused. To fix this, we added a distance threshold so only nearby objects trigger alerts, and we ignore anything outside the user’s path unless it’s moving into it.

Another challenge came up indoors. LiDAR kept detecting the same objects and announcing them over and over, even when the user was standing still. To fix this, we used the LiDAR data to build a map of the room as the user moves, so the app only announces objects it hasn’t seen before.

Important Lessons

We learned how much a well-trained model matters. The models we used weren’t trained on a large enough dataset, so they didn’t recognize a lot of everyday objects. The app could still warn users that something was in their way, but it couldn’t always tell them what it was, which made the alerts less helpful. Going forward, we want to train on a bigger, more varied dataset so Resonate can give users a clearer picture of what’s around them.