Be My Eyes

Be My Eyes What it is: A remarkable app that connects blind and low-vision users with a global community of sighted volunteers and company representatives through a live video call.

Comment

Inspiration

Our inspiration came from a simple, yet profound observation: for countless daily tasks, blind and low-vision individuals face small "visual interpretation gaps" that can hinder independence. Tasks like checking an expiration date, reading a thermostat, or picking out the right can from a pantry often require a moment of sight. On the other side of this gap is a global community of people with a fundamental desire to help one another. Our inspiration was to build a digital bridge across that gap—to channel the power of human compassion and technology into a tool that could deliver sight, on-demand, to anyone who needs it, anywhere in the world. We wanted to turn the abstract idea of a "global village" into a practical, immediate reality.

What it does

Be My Eyes is a free mobile application that connects blind and low-vision individuals with a global community of sighted volunteers and specialized company representatives through a live, one-way video call.

  • For the Blind/Low-Vision User: When a user needs visual assistance, they open the app and tap a single button to request help. The app connects them to the first available sighted volunteer. Through their phone's rear-facing camera, the user can show the volunteer what they need help with, and the volunteer describes what they see in real-time.
  • For the Sighted Volunteer: Volunteers receive a notification when a user needs help. If they are available, they accept the call and become that person's eyes for a few moments, providing friendly, direct assistance.
  • Specialized Help: We've partnered with companies like Microsoft, Google, and LinkedIn to provide direct, accessible customer support through the app, connecting users with trained company experts instead of volunteers for specific product issues.
  • Be My AI: Our newest feature integrates OpenAI's GPT-4. Users can send an image to our "Virtual Volunteer," which provides an instant, detailed AI-generated description of the image and can engage in a conversational Q&A about its contents.

How we built it

Be My Eyes is a robust, scalable mobile platform built with a focus on reliability, low latency, and accessibility.

  • Frontend: The mobile applications for iOS and Android were built natively using Swift and Kotlin, respectively. This allows for deep integration with the operating systems' award-winning accessibility features, including VoiceOver and TalkBack. The UI is designed with extremely high contrast, large tap targets, and a simple, unambiguous layout.
  • Backend: Our backend is built on a microservices architecture running on cloud infrastructure (like AWS or Azure). This ensures high availability and scalability to handle our global user base. We use managed databases for user profiles, call logs, and volunteer metrics.
  • Real-Time Communication: The core video-calling functionality is powered by the WebRTC (Web Real-Time Communication) protocol. This enables a direct, peer-to-peer, low-latency video and audio stream between the user and the volunteer's devices, which is critical for real-time interaction. A signaling server on our backend manages the initial connection handshake.
  • Notifications & Matchmaking: When a user requests help, our backend runs a matchmaking algorithm that pings thousands of volunteers who speak the same language, prioritizing those in a similar timezone. This is handled through Apple Push Notification Service (APNs) for iOS and Firebase Cloud Messaging (FCM) for Android.
  • AI Integration: The "Be My AI" feature leverages an API integration with OpenAI's GPT-4 Vision model. When a user submits a picture, our backend securely sends it to the API, receives the descriptive text, and streams it back to the user's device for the screen reader to announce.

Challenges we ran into

  1. Minimizing Latency: The single biggest technical challenge was ensuring a near-instantaneous video stream. Any significant lag makes real-time guidance impossible. We invested heavily in optimizing our WebRTC implementation and global server infrastructure to minimize connection times.
  2. Volunteer Availability ("The Midnight Problem"): How do you ensure a user in Pilani at 2 AM gets help when most volunteers in their timezone are asleep? We solved this by building a truly global volunteer base. The matchmaking algorithm intelligently routes calls across timezones, so a user in India might be helped by a volunteer in Canada who is just finishing lunch.
  3. Designing for Non-Visual Users: Every design decision had to be vetted through a "voice-first" lens. We had to unlearn many standard UI/UX patterns and work extensively with blind and low-vision users to create an interface that is completely intuitive to navigate using only a screen reader and gestures.
  4. Safety and Trust: Creating a safe environment was paramount. We implemented a one-way video (the volunteer can't be seen) and a robust reporting and blocking system to ensure the integrity of the community.

Accomplishments that we're proud of

  • Building a Global Community: We are incredibly proud to have built a community of over 7.5 million volunteers who are available to help over 600,000 blind and low-vision users in more than 150 countries and 185 languages.
  • The Power of Simplicity: We've kept the core user experience down to a single tap. This relentless focus on simplicity has been key to our adoption and success.
  • Successful AI Integration: The launch of "Be My AI" was a major accomplishment. It provides an instant option when a user doesn't want to talk to a person, handling a significant percentage of image description requests and freeing up human volunteers for tasks requiring more complex interaction.
  • Reliability: Maintaining a highly available service that people rely on for their daily independence is an accomplishment our engineering team is deeply proud of.

What we learned

  • Design with, not for: We learned that we could not have built this app without integrating the blind community into every step of our design and testing process. Their lived experiences and feedback were the most valuable resource we had.
  • Humanity is the Killer Feature: While the technology is complex, the core of Be My Eyes is human connection. We learned that people have an immense capacity for kindness, and our job was simply to provide an efficient channel for it.
  • AI and Humans are Better Together: The introduction of "Be My AI" taught us that AI is not a replacement for human connection but a powerful complement to it. The AI handles quick, factual tasks, while human volunteers excel at tasks requiring nuance, emotional intelligence, and complex problem-solving.

What's next for Be My Eyes

The future is about creating an even more seamless and context-aware visual interpreter for the world.

  • Integration with Smart Glasses: We are actively exploring deeper integrations with upcoming AR/Smart Glasses. Imagine a future where a user doesn't even need to pull out their phone. Assistance, both from AI and human volunteers, could be streamed directly to their field of view, enabling true hands-free interaction with the world.
  • Proactive AI Assistance: We envision an evolution of "Be My AI" that can run in a "live" mode, providing a continuous, ambient audio description of the user's surroundings through their phone's camera, alerting them to obstacles or points of interest.
  • Expanding Specialized Help: We plan to onboard more corporate partners in sectors like banking, healthcare, and transportation to provide expert, secure assistance for tasks that are too sensitive for the volunteer community.
  • Enhanced Community Tools: We want to build more tools for our volunteer community, allowing them to track their impact, connect with other volunteers, and share their experiences, further strengthening the network of global kindness we've built.

Built With

Share this project:

Updates