Netra Assist AI

An AI-powered "Digital Eyes" super-app empowering visually impaired users with real-time hazard detection, object recognition, and voice assistance for independent living.

Comment

💡 Inspiration

According to the WHO, over 2.2 billion people globally have a near or distance vision impairment. While technology is advancing rapidly, visually impaired individuals still struggle with daily independence, safe navigation, and accessing written information. Inspired by the United Nations SDG 3 (Good Health and Well-being), I wanted to build more than just an app; I wanted to create "Digital Eyes" that act as an auto-pilot for safe, independent living.

⚙️ What it does

Netra Assist is an AI-powered, voice-first multimodal accessibility super-app. It empowers users through:

  • Live Vision Auto-Pilot: Uses the camera to narrate the real-world environment, detecting objects and warning about physical hazards (e.g., "Car ahead, move left").
  • Document & PDF Intelligence: Instantly reads, summarizes, and explains complex documents or physical text aloud.
  • Face & Emotion Scan: Helps users understand social cues by reading facial expressions (Happy, Sad, Angry) of people around them.
  • 100% Voice-Controlled UX: Designed entirely around a hands-free, voice-activated interface suitable for the visually impaired.

🛠️ How we built it

  • Frontend: Built a highly accessible, high-contrast UI using Flutter and Dart.
  • AI Engine: Powered by Google Gemini 1.5 Flash. I specifically chose the Flash model because real-time hazard detection requires ultra-low latency and fast multimodal reasoning (processing camera frames to text instantly).
  • Backend & Cloud: Used Firebase for secure data handling and cloud infrastructure.
  • Integrations: Implemented flutter_tts for continuous voice feedback and various native camera/file packages.

🚧 Challenges we ran into

The biggest challenge was latency. For a blind user walking on a street, a delay of 5 seconds in hazard detection is dangerous. I had to optimize the image compression and rely heavily on the speed of the Gemini Flash model to ensure near real-time auditory feedback. Designing a UI that doesn't rely on "looking" at the screen was another major UX challenge, which I solved by building a gesture and voice-heavy navigation system.

🏆 Accomplishments that we're proud of

I am extremely proud of successfully pivoting and refining the core architecture into a pure "Social Good" application under tight hackathon deadlines. Achieving near real-time object narration and emotion detection that genuinely feels like an AI companion for the blind is my biggest win.

📚 What we learned

I learned how to fully exploit the multimodal capabilities of LLMs (sending images and context together seamlessly). More importantly, I learned about the strict UX guidelines required for building true accessibility apps—realizing that less UI clutter and more voice interaction is the key to inclusive design.

🚀 What's next for Netra Assist AI

  • Wearable Integration: Expanding the app to work with smart glasses (sending camera feeds directly from the glasses to Gemini).
  • Offline Mode: Integrating smaller, on-device models to provide basic hazard detection even without internet access.
  • Multilingual Expansion: Adding support for more regional languages to reach rural populations.

Built With

Share this project:

Updates