Inspiration Our project was inspired by the everyday communication challenges faced by deaf and hard-of-hearing individuals—especially in environments where audio announcements are unreliable, such as public transit, hospitals, and airports. Projects like RERC’s accessible voice assistant work (Project R3) and aiD’s AI-driven sign language avatars showcased the potential of inclusive tech to promote independence. Personal connections with Deaf community members and research on socioeconomic barriers in accessing assistive technology further motivated us to design a solution that is both affordable and user-driven.

What It Does The Real-Time Optical Marker Sign Decoder captures hand movements using a camera and interprets sign gestures by processing optical markers in real time. The system: Recognizes signs using computer vision and machine learning Converts gestures into text or voice output Provides visual feedback for clarity and accessibility Integrates with smart devices for hands-free control The tool aims to bridge communication gaps in public spaces, medical settings, classrooms, and daily interactions without requiring interpreters or expensive proprietary equipment.

How We Built It Tech Stack: Python + OpenCV for gesture tracking and optical marker detection TensorFlow for training and running the sign recognition models React Native for the mobile interface and AR-based avatar feedback WebRTC for telerehab-style video communication features Voice Assistant Integration (R3-inspired): Modified assistant APIs to support adjustable speech pitch, speed, and volume Enabled vibration alerts and real-time transcription for accessibility Telerehab Components (R1-inspired): Web-based video sessions with captions and visual cues Cloud deployment to support remote speech training Cochlear Implant UI (R2-inspired): Built dashboards with sliders for volume and frequency adjustments Added haptic feedback to confirm changes Ensured WCAG-compliant contrast, alt text, and readable layouts All components were integrated into a unified app for seamless interaction and smart home control.

Challenges We Ran Into Technical Challenges: Training models to accurately recognize diverse sign gestures in noisy or low-light environments Dataset limitations required heavy augmentation for reliable recognition Video relay systems introduced delays during fast-paced or emergency scenarios

Accessibility Challenges: Designing ASL-friendly interfaces was difficult without continuous Deaf community collaboration Public spaces often lacked supportive infrastructure (e.g., visual announcements, caption displays)

Resource & Ethical Barriers: Hardware costs made large-scale testing difficult Data privacy concerns around video input slowed deployment planning Limited interpreter access during usability testing reflected real-world inequities Despite these hurdles, iterative user testing and open-source resources helped us refine core features.

Accomplishments We’re Proud Of

Achieved accurate real-time decoding of optical marker gestures Developed a unified platform integrating sign recognition, voice assistant customization, and accessibility tools Created an app interface that meets key WCAG accessibility standards Successfully piloted remote-captioned telerehab sessions and user-adjustable hearing tech dashboards Built a low-cost prototype demonstrating the potential for scalable, inclusive communication tools

What We Learned

We learned that many accessibility issues stem from device defaults—such as high-pitched voices conflicting with hearing aids or insufficient visual feedback. Telerehabilitation research (Project R1) reinforced the importance of captioning, visual cues, and remote training options. User-driven cochlear implant programming (Project R2) highlighted the value of intuitive interfaces and daily autonomy. Most importantly, we recognized that intersectional issues—budget limitations, cultural considerations, and limited ASL-centered design—intensify accessibility barriers. Early and frequent user feedback is essential to designing culturally respectful, effective assistive technology.

What’s Next for Real-Time Optical Marker Sign Decoder

Expand gesture datasets to support regional sign variations and more complex phrases Integrate with AR glasses for hands-free, always-on sign recognition Partner with Deaf organizations to co-design improvements and ensure culturally aligned development Enhance smart home and IoT compatibility for broader independence Develop an offline version to support use in low-resource or emergency environments Explore collaboration with telerehab and voice-assistant platforms for wider accessibility integration