INSPIRATION
My neighbor's grandfather stopped leaving his house last year. No accident. No drama. He just quietly stopped going outside. He memorized every step from his bed to his front door — fourteen steps — and counted them every morning just to feel safe in his own home.
He had a smartphone in his pocket the whole time. A device with a camera, processor, and internet connection. Completely useless to him because nobody built the missing piece.
285 million people are visually impaired worldwide. OrCam, Envision, eSight — remarkable technology, ₹3,00,000 ($3,600) minimum. For 70% of India's blind population living below the poverty line, that price is not a barrier — it is a closed door.
I went to the local electronics shop. Spent ₹540 ($6.50). Two sensors. One buzzer. One Arduino. The technology was already in his pocket. I built the bridge.
That bridge is VisionAssist AI.
WHAT IT DOES
Say "My Eye" followed by any command. The AI responds instantly. VisionAssist AI transforms a standard smartphone and ₹540 worth of components into a complete hands-free wearable assistive device — worn around the neck, camera always facing forward, fully voice-controlled. Works on any smartphone browser. Zero installation required.
1. Agentic Multi-Step Reasoning — Core Innovation The agent thinks before acting. Say "help me find my medicine." It checks saved local memory first — if the object location is already stored, it speaks the answer instantly and stops. No camera opened. No API call wasted. If not in memory, it opens the camera and scans the current scene. Still not found — it triggers a full 360° room scan, capturing frames as you slowly turn. Each completed step is stored in session memory so the agent never repeats work. A 3-second answer when data exists. A complete room search only when truly needed.
2. Real-Time Obstacle Detection — Fully Offline Two HC-SR04 ultrasonic sensors provide complete spatial awareness with zero cloud delay and zero internet required. Top sensor detects chest-level obstacles — walls, poles, people. Bottom sensor angled 30° downward detects ground-level hazards — curbs, steps, small objects. Arduino buzzer fires instantly: 3 short beeps for obstacles, 1 long beep for stairs. No latency. No dependency.
3. Traffic Light Detection with Dual Verification Two methods run in parallel — Groq AI vision identifies the light color while HSV color analysis via OpenCV confirms it independently. Both must agree before speaking. Red detected → "Stop, do not cross." Green detected → "Safe to cross now." If either method detects red, safety overrides. Response under 2 seconds. Live Mode runs continuously and only speaks when the light changes.
4. Indoor Navigation — Dual-Index System A custom algorithm fusing voice commands with real-time visual verification — works in blurry corridors where GPS fails completely. Zero infrastructure setup needed, unlike BLE beacons or RFID systems that require hundreds of physical installations per building. Say "Go to room 204" — the system guides step by step using pre-recorded routes.
5. Document Memory with n8n + Pinecone Blind users upload Aadhar cards, prescriptions, and ID proofs once. Say "My Eye... show my Aadhar number" — the AI searches the Pinecone vector database and reads the answer aloud instantly. n8n extracts text, splits into chunks, converts to OpenAI embeddings, stores in Pinecone. Retrieval is fully voice-driven. No carrying physical documents. No asking someone else to read for you.
6. Object Finder 360° Captures multiple frames during a slow rotation. Sends all frames together for full-room spatial analysis. Returns exact direction and distance. Always checks memory before scanning — skips the camera entirely if location was already saved. Ask twice → second answer from memory, zero API cost.
7. Food Identifier Names every item on the plate, estimates quantity, counts plates and glasses, states meal type. Live Mode only speaks when something on the plate changes — prevents voice fatigue so the user stays informed without being overwhelmed.
8. Medicine Reader Auto-detects when a medicine label appears in frame. No button press needed. Extracts name, dosage, and usage instructions — critical for blind users managing their own medications independently.
Additional features: face recognition, currency detection, scene description, stair safety, emergency SOS with live GPS via Twilio SMS + SMTP email simultaneously, outdoor routing via Leaflet.js with OpenStreetMap. Works in Hindi, English, and Hinglish.
HOW WE BUILT IT
Built entirely from scratch — Python and Flask for backend, HTML5, CSS3, and Vanilla JavaScript for frontend. No paid frameworks. No shortcuts.
AI Vision — Groq API (Llama 4 Scout 17B) as primary model, Google Gemini 1.5 Flash as automatic fallback. 3-key rotation system with exponential backoff — if one Groq key hits rate limits, the next takes over instantly. No request fails silently.
Computer Vision — OpenCV for frame processing and HSV color analysis, YOLOv8 for real-time object detection, DeepFace with Facenet for face recognition, Tesseract OCR for text extraction backup.
Agent Brain — Custom multi-step reasoning loop in Python. Each session holds an AgentSessionMemory object storing results from every completed step. Agent reads this before each decision — never re-runs completed work. Fully thread-safe with lock-protected reads and writes.
Speech — Web Speech API handles wake word detection, continuous voice recognition, and text-to-speech simultaneously in the browser with zero latency. Microphone pauses exactly when the agent speaks and restarts automatically — prevents feedback loops.
Document Intelligence — n8n cloud workflow + Pinecone vector database + OpenAI text-embedding-3-small. Documents uploaded once, retrieved forever by voice.
Hardware — Arduino clone (ATmega328P) + 2× HC-SR04 ultrasonic sensors + active buzzer. Powered entirely by the phone via OTG cable — no separate battery. Total hardware cost: ₹540 ($6.50). Complete device with smartphone: 280 grams.
Maps — Leaflet.js with OpenStreetMap for outdoor routing. Indoor map generated dynamically from a home walkthrough video using AI room detection.
Alerts — Twilio SMS + SMTP email triggered simultaneously on SOS with live GPS coordinates.
Deployment — Backend on Render free tier. No installation needed — runs in any smartphone browser.
CHALLENGES WE RAN INTO
Traffic light latency — AI takes 1.5–2.5 seconds per frame. Too slow for a live road crossing. Solved with Live Mode — runs continuously in the background, speaks only when the light changes. The answer is always ready before the user asks.
Traffic light accuracy in low light — Pure AI vision failed in shadows and glare. Added HSV color analysis as a parallel independent verification layer. Safety always overrides confidence.
Agent state across HTTP requests — Flask is stateless by default. Built a full session store with thread-safe locks so the agent remembers completed steps across multiple API calls without ever repeating them.
Indoor navigation on irregular layouts — AI returned unstructured text for unusual floor plans. Added a fallback parser that extracts room names from keywords and builds a valid navigation graph even from imperfect AI output.
API rate limits under heavy use — Exponential backoff with 3-key rotation. Each key gets cooldown time proportional to failure count and resets automatically.
Voice collision — TTS speaking while mic was listening caused feedback loops. Solved by pausing the microphone precisely when the agent speaks and auto-restarting after speech ends.
ACCOMPLISHMENTS WE'RE PROUD OF
Agent reasoning that stops early — memory check first, camera only when needed. Faster, smarter, feels like a real companion not a tool.
Dual Mode across every feature — Normal for accuracy, Live for continuous background awareness. No other accessibility app has this combination.
Dual verification for traffic lights — AI vision and HSV color analysis must both agree. Safety-first architecture validated in real conditions.
8+ features built solo — most accessibility apps ship 3–4 features with a full team.
Zero software cost — every component runs on free tiers. No credit card. Works on any smartphone globally.
Real user validation — a visually impaired tester said after the first session: "I don't need someone with me anymore. I can walk alone, read my prescription, know what I'm eating. That's freedom."
₹540 total hardware cost — 0.18% of the cheapest commercial alternative. Proven on a real user in a real home.
WHAT WE LEARNED
Safety must always override speed. Smart reasoning that stops early is better than always doing more. Voice-first design changes everything for users who cannot look at a screen. Free tiers are fully sufficient to build production-grade tools. Never trust AI alone for safety-critical decisions — always verify with a second independent method.
WHAT'S NEXT
Android app where each user sets up their own free Groq API key once — making the system truly free for every user with 1,000 requests per day. No shared server. No shared limits.
On-device TinyML (TensorFlow Lite) for offline object detection and OCR. Local encrypted document storage (SQLite + AES) replacing cloud Pinecone. Bluetooth module replacing OTG cable. Medication expiry alerts. More Indian regional languages. Crowdsourced hazard alerts from the blind community.
Scale vs commercial alternatives:
- 10 users: ₹5,400 vs ₹30 lakh for OrCam
- 100 users: ₹54,000 vs ₹3 crore for OrCam
"Technology should lift people up, not price them out."
BUILT WITH
Python · Flask · Groq API (Llama 4 Scout 17B) · Google Gemini 1.5 Flash · OpenCV · YOLOv8 · DeepFace · Facenet · Tesseract OCR · Web Speech API · Leaflet.js · OpenStreetMap · Twilio · SMTP · n8n · Pinecone · OpenAI Embeddings · Arduino · HTML5 · CSS3 · Vanilla JavaScript
TRY IT OUT
🌐 Live App: https://global-blind-device.onrender.com/ 🎥 Real Blind User Demo: https://youtu.be/LY4OkuAPpD8?si=RMLXmWiQU4U1KrV7 💻 GitHub: https://github.com/clod9344-cloud/Global_blind_device
"I don't need someone with me anymore. That's not convenience. That's freedom."
Log in or sign up for Devpost to join the conversation.