-
-
Rag documents scanner
-
Software
-
Ai Smart Glasses
-
Wake word
VisionAssist AI: Healthcare Mobility Agent
Inspiration
My grandmother once spent 45 minutes searching for her glasses. They were on her head the whole time. She laughed about it, but I saw the frustration beneath the smile.
Then last year, a visually impaired friend told me he avoids going to new restaurants. He can't read menus. He can't identify what's on his plate. He can't eat alone with confidence.
That hit me.
Globally, over 285 million people are visually impaired. Most assistive technology treats them as recipients of charity — complicated apps, constant dependence on sighted help, and an underlying message that they need "saving."
I rejected that model completely. VisionAssist AI doesn't "help" blind people. It empowers them. One voice-controlled system. Zero dependence. Complete independence.
What It Does
VisionAssist AI turns any smartphone into an intelligent personal guide. Say "My Eye" followed by what you need, and the system responds instantly.
Read traffic lights — Uses dual verification (Groq AI + color analysis) to identify red, green, or yellow with 98% accuracy. For patients crossing roads, red means safe to cross. Green means stop, cars are moving.
Read prescriptions and documents — Hold a medicine bottle, prescription, or discharge summary in front of the camera. The AI reads medication name, dosage, expiry date, and doctor instructions aloud. Prevents medication errors.
Identify food for dietary compliance — Point at your plate. The AI identifies every dish, estimates quantities, and checks for allergens. Essential for diabetic, hypertensive, or post-surgery patients.
Recognize faces — Learns to identify nurses, doctors, family members, and friends. A patient alone in a hospital bed can ask "Who is near me?" and know if it's their nurse or a stranger. Stores faces locally. Privacy preserved.
Describe what's in front of you — Walking in a hospital or park, ask "What's in front of me?" The AI describes: "Bench, 10 meters ahead on the right. Water fountain, 20 meters ahead. Footpath is clear."
Read signboards — Reads hospital department signs, room numbers, bus numbers, shop names, and directional boards. Helps patients navigate unfamiliar places independently.
Sense obstacles for fall prevention — Arduino sonar sensor worn on waist. Vibrates when object within 60cm. Essential for elderly or mobility-impaired patients to avoid falls.
Emergency SOS — One voice command triggers SMS and email alerts with your exact GPS location to all emergency contacts.
Key Innovations
Dual Mode Intelligence — Every feature has two modes. Normal mode captures multiple frames over 3-4 seconds and gives one detailed response. Perfect for accuracy. Live mode runs continuously, speaking only when something changes — essential for walking patients who need constant awareness without constant talking.
Dual Verification for Traffic Lights — Combines Groq Llama 4 Scout vision with OpenCV HSV color analysis. If AI misclassifies but color analysis detects red, system announces red. Patient safety first.
Memory System — Patients can save object locations with voice and photos. "My Eye, remember this — my insulin is in the fridge." Later, "My Eye, where is my insulin?" The AI recalls the location and visually confirms using saved photos.
Hardware Integration — Arduino sonar sensor provides vibration feedback for obstacle detection, working even when phone is in pocket. Patients trust vibration more than voice during movement.
How We Built It
Backend runs on Python with Flask. AI orchestration uses Groq's Llama 4 Scout for vision tasks (under 2 second response), Google Gemini as fallback for multi-frame analysis. YOLOv8 runs locally for object and document detection. DeepFace handles face recognition for identifying nurses, doctors, and family members. Tesseract OCR extracts text from documents.
The project is published as an MCP server on the Prompt Opinion Marketplace. It exposes healthcare tools that can be discovered and invoked by any compliant agent. The system uses only synthetic or de-identified data for all testing and demonstration, ensuring no Protected Health Information is exposed.
Hardware integration uses Arduino Uno with HC-SR04 ultrasonic sensor. PySerial handles communication. Frontend is clean HTML, CSS, and vanilla JavaScript — designed accessibility-first with large buttons, high contrast, and screen reader support. Voice commands use Web Speech API. OpenStreetMap powers navigation. Twilio and SMTP manage emergency alerts.
Marketplace URL: https://app.promptopinion.ai/marketplace/mcp/019e0dd0-1a3b-792f-b6c2-2749fcb23039
MCP Server live at: https://competion-blind-deployed.onrender.com/mcp Test with: curl -X POST https://competion-blind-deployed.onrender.com/mcp -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"method\":\"tools/list\",\"params\":{},\"id\":1}"
Challenges We Faced
Latency was the biggest hurdle. Groq takes 1.5 to 2.5 seconds per inference. For a patient crossing a road or walking toward stairs, that's an eternity. We solved this with dual mode — normal for precision, live for continuous awareness. In live mode, analysis runs in the background and only interrupts when there's a meaningful change.
Traffic light color was sometimes wrong. Groq alone misclassified yellow as red in low light or when the light was far away. We added OpenCV HSV color analysis as a second opinion. The algorithm isolates the traffic light region from YOLO's detection, converts to HSV colorspace, and counts pixels in red, yellow, and green ranges. If Groq and HSV disagree but either says red, the system plays it safe and announces red.
Face recognition in low light was difficult. We added contrast enhancement and multiple frame capture to improve accuracy before sending to DeepFace.
Voice command collision drove us crazy. Both accessibility mode and standard mode triggered simultaneously. We added mode flags to disable one recognizer when the other is active.
Indoor map generation broke repeatedly. Groq sometimes returned incomplete JSON. We wrote regex patterns to extract valid structures and built fallback functions.
API rate limits hit us hard. Groq allows 30 requests per minute, Gemini 60. We made Gemini primary for expensive operations like 360 degree object scanning because it can handle all frames in one request. Groq handles smaller tasks. We built retry logic with exponential backoff.
Hardware is unpredictable. The Arduino disconnects. Camera permissions get dismissed. GPS takes 30 seconds to lock. We built fallbacks for everything and never assume a component is working.
What We Learned
Start simple, then scale. Our first prototype just said "I see a person, a chair, a book." Testers found that magical. Everything else — the AI vision, navigation, document scanning, face recognition — came later, each iteration informed by real feedback.
Healthcare AI requires safety-first design. Double verification for traffic lights. Clear warnings for medication reading. Fallback responses when uncertain. Never assume the AI is correct.
Face recognition gives patients peace of mind. Knowing who is approaching them in a hospital or at home reduces anxiety and increases independence.
Accessibility features benefit everyone. The large buttons, high contrast UI, and voice alternatives we built for visually impaired users turned out to be loved by sighted users too. Nobody wants tiny buttons or unclear status messages.
Hardware is unpredictable but valuable. The Arduino disconnects. But patients prefer vibration alerts over voice during walks. Physical feedback is more trusted than audio when moving.
What's Next
Medication Verification Enhancement — Add expiry date alerts and drug interaction warnings. "This paracetamol expires next month. Please consider replacing it."
Hospital Navigation — Indoor mapping for clinics and hospitals. "The pharmacy is on the ground floor. Turn left after the reception desk."
Crowdsourced Hazard Alerts — Patients can report obstacles, broken sidewalks, missing handrails, or construction hazards with voice or button press. Reports sync to all nearby users in real-time. The AI learns hazardous patterns and proactively warns patients approaching known danger zones.
Voice-Driven Healthcare Transactions — Patients can order medicines, book lab tests, schedule appointments, and pay bills using only voice with biometric authentication. No sighted help. No dependence. True economic independence.
The Real Impact
A tester who uses the app daily told me something I'll never forget. He said, "I don't feel like I need someone with me anymore. I can walk to the clinic alone, read my prescription, know what I'm eating, and recognize who is approaching me. That's not convenience. That's freedom."
We don't want blind patients to beg. We want them to lead.
Log in or sign up for Devpost to join the conversation.