AccessibleAID
AccessibleAID is a privacy-first AI accessibility assistant designed to help disabled individuals better understand, navigate, and interact with the world around them entirely on-device and fully offline after the initial setup.
Unlike traditional accessibility tools that rely heavily on cloud processing, AccessibleAID keeps sensitive data local. Camera frames, microphone audio, pasted documents, and accessibility interactions never leave the device. All AI inference runs directly in the browser using open-source local AI models.
Inspiration
Accessibility tools today are often fragmented, internet-dependent, and focused on solving only one accessibility challenge at a time. We wanted to explore a different question:
What if AI could act as a realtime accessibility companion that helps disabled individuals navigate everyday environments more independently?
Our inspiration came from observing how classrooms, transportation systems, conversations, signage, and public spaces are often not designed with accessibility in mind. We wanted to create a system that reduces those barriers while respecting privacy, independence, and dignity.
Instead of building another cloud-dependent AI assistant, we focused on creating a fully local, offline-capable accessibility platform powered entirely by open-source AI models.
What It Does
AccessibleAID combines multiple accessibility-focused AI systems into a single mobile-first platform.
The application includes:
Understand World
AI-powered scene understanding and environmental narration using realtime camera input.
Scene Camera
Object detection, spatial awareness, OCR, and environmental understanding that help users interpret their surroundings.
Navigate
Accessibility-aware assistance that helps users better understand nearby obstacles, distances, and spatial positioning.
Live Captions
Realtime offline speech-to-text captions powered entirely on-device.
Sign Language Support
Gesture and communication assistance designed for accessibility-focused interactions.
Simplify Info
AI-powered simplification that rewrites difficult documents into easy-to-understand language with readability scoring.
The platform supports:
- blind and low-vision users
- deaf and hard-of-hearing users
- wheelchair users
- neurodivergent users
- elderly individuals
- users requiring cognitive accessibility support
Standout Features
- Fully offline-capable after first load
- Privacy-first local AI inference
- Realtime scene narration
- Spatial “what’s around me” guidance with clock-position descriptions
- OCR text reading and document understanding
- Medication identification through local retrieval pipelines
- Smart sound classification with vibration alerts
- Voice-only mode for blind users
- Cross-mode memory between AI tools
- Installable PWA experience
- WCAG-AA accessibility support
- Haptic feedback patterns for accessibility alerts
How We Built It
We built AccessibleAID as a modern AI-powered accessibility platform using React Vite Tailwind CSS and Progressive Web App architecture.
The AI stack uses fully local open-source models including Whisper Tiny for offline speech-to-text ViT GPT2 for image captioning DETR for object detection Depth Anything v2 for depth estimation TrOCR for OCR MiniLM for local embeddings and Qwen2 5 for local language simplification.
For local inference and acceleration we used transformers js ONNX Runtime WebGPU and WebAssembly allowing the entire AI pipeline to run directly on-device without external AI APIs.
We also focused heavily on accessibility-first UX by implementing:
- screen-reader optimized interfaces
- high-contrast accessibility modes
- adaptive typography scaling
- haptic accessibility feedback
- realtime spoken narration
- voice-first interaction flows
Challenges We Faced
One of the biggest challenges was balancing:
- realtime AI inference
- browser/mobile performance
- offline capability
- and accessibility-focused UX
Running multiple AI systems entirely inside the browser while maintaining smooth interactions required extensive optimization and lightweight model selection.
Another challenge was designing interfaces that remain:
- visually clean
- cognitively accessible
- highly readable
- and responsive to screen readers while still feeling modern and immersive.
We also had to carefully architect offline caching and local model persistence so the application could continue functioning without internet access after the initial setup.
What We Learned
Through building AccessibleAID we learned:
- how to optimize AI systems for edge and browser environments
- how accessibility fundamentally shapes UX architecture
- how multimodal AI systems can collaborate in realtime
- and how important privacy-first design is for assistive technologies
Most importantly we learned that accessibility should not be treated as an afterthought but should instead be built directly into the foundation of intelligent systems.
Future Vision
We envision AccessibleAID evolving into a fully offline-capable accessibility intelligence platform that helps disabled individuals navigate everyday life more independently and confidently.
Future directions include:
- richer spatial navigation
- advanced sign language understanding
- wearable integrations
- smarter environmental understanding
- personalized accessibility adaptation
- deeper offline AI support
Our goal is simple:
Build AI that makes the world more accessible without compromising privacy independence or dignity.
Disclaimer
AccessibleAID is a hackathon prototype intended as an accessibility-first proof of concept and is not a substitute for medical legal or professional advice.
Built With
- 5
- anything
- api
- app
- context
- css
- depth
- detr
- gpt2
- html5
- indexeddb
- javascript
- minilm
- onnx
- progressive
- qwen2
- react
- runtime
- service
- tailwind
- tiny
- transformers
- trocr
- v2
- vit
- vite
- web
- webassembly
- webgpu
- whisper
- workers
Log in or sign up for Devpost to join the conversation.