Vision

Inspiration

My inspiration for Vision comes from a powerful story in the Bible about Jesus healing a blind woman. Her joy in regaining sight moved me deeply and made me appreciate the gift of vision — the beauty of the world and the faces of loved ones. This motivated me to build a tool that helps people "see" again in their own way — not only to stay safe but to regain independence and confidence in their surroundings.

What it does

Vision is a mobile app that uses the device’s rear camera to capture snapshots of the user’s environment. These frames are analyzed using the Google Vision API to detect surrounding objects. The app then provides intelligent audio feedback based on:

Height classification of the object:
- Low (e.g., backpack)
- Waist-high (e.g., chair, couch)
- High (e.g., ceiling, fridge)
- Flat/ignorable (e.g., floor, walls)
Distance from the user, adjusting volume accordingly:
- Closer objects ⟶ Louder sounds
- Distant objects ⟶ Softer sounds
Object-specific naming, announced via text-to-speech.

New Features:

A dropdown that lists all specifically detected objects (not just main categories).
An accuracy score that factors in:
- Correctness of detection
- Volume scaling accuracy
- Completeness of obstacle detection
- Logical position & distance match

How we built it

Built using React Native (Expo) for cross-platform mobile support.
Captured frames via the device camera using Expo Camera API.
Sent images to the Google Vision API to detect and classify objects.
Estimated height of obstacles from bounding box coordinates.
Used bounding box size to approximate distance, then scaled volume accordingly.
Played custom audio feedback sounds based on object height.
Added a dropdown UI and calculated an accuracy percentage for each detection.
Ensured everything worked inside Expo Go without needing native code or prebuilds.

Challenges we ran into

Making the audio volume dynamic (louder for closer objects) without any external hardware or native modules — using only JavaScript and Expo tools.
Designing the logic to categorize objects into low, waist-high, default, or high using just 2D image data.
Preventing scrollable dropdown UI components from changing height and overlapping the rest of the interface when multiple objects are detected.
Working entirely within non-native tools since we couldn’t prebuild for iOS (no $99 Apple Dev Account or MacBook with Xcode).

Accomplishments that we're proud of

Successfully integrated the Google Vision API to get highly detailed object recognition results.
Developed a multi-level sound system to play different audio cues depending on obstacle height.
Programmed dynamic volume scaling based on how close the object is to the user — with no external hardware.
Created a real-time accuracy estimation system that factors in distance, detection correctness, and system response.

What we learned

Designing for accessibility means every choice matters — sound tone, delay, verbosity, etc.
Real-world environments are unpredictable — filters and fallback behavior are essential.
Expo Go is more powerful than expected when used creatively.
Even without native modules or hardware, it’s possible to create useful, real-time assistive tech.
Importance of clean, minimal UI for people who aren’t using screens visually.

What's next for Vision

🧠 Add spatial awareness — distinguishing if the object is to the left or right of the user using horizontal bounding box offsets.
📸 Implement consecutive auto-screenshotting, so the user doesn’t have to tap every frame.
🔊 Improve sound mapping to better match object type and size with more realistic audio cues.
🔭 Add zooming and far-distance detection for earlier obstacle alerts.
🌍 Add language support for multilingual users, and VoiceOver integration for blind users on iOS.
💡 Explore offline ML models to reduce dependency on external APIs and enhance speed.