SmartGlass

Inspiration

Millions of people worldwide face challenges due to visual, hearing, or limb impairments, limiting their independence and interaction with the environment. Existing assistive technologies are often expensive, incomplete, or difficult to use. Inspired by the potential of AI and wearable technology, we set out to create SmartGlass—a device that combines real-time AI processing, haptic feedback, and multimodal interaction to enhance accessibility and independence.

What it does

SmartGlass is an AI-powered wearable system designed to assist individuals with visual, hearing, and limb impairments by integrating:

Obstacle Detection & Navigation: LiDAR & AI provide real-time warnings (e.g., "Pothole ahead, 2 meters"). Sign & Text Recognition: OCR extracts text from street signs, labels, and books, converting it into speech. Sign Language Translation: AI-powered gesture recognition converts sign language into text/audio. Haptic Feedback for Amputees: Smart gloves provide touch sensation, material recognition, and grip control. Emergency Sound Detection: Identifies alarms, sirens, and vehicle horns for users with hearing impairments.

How we built it

Hardware Components: Smart Glasses: Equipped with a camera, LiDAR sensor, and real-time AI processing unit. Haptic Gloves: Integrated with EMG sensors, vibration motors, and material recognition AI. Software & AI Models: YOLO (You Only Look Once) for real-time object and sign language detection. Tesseract OCR for text extraction and speech conversion. EMG & EEG Processing for muscle and neural signal interpretation in haptic gloves. Edge AI for on-device processing, ensuring fast response and data privacy.

Challenges we ran into

Real-time Processing: Ensuring AI models run efficiently on wearable hardware without latency. Gesture & Sign Language Recognition: Training YOLO to detect a wide range of hand signs accurately. Haptic Feedback Calibration: Fine-tuning EMG signals for accurate touch simulation without overstimulation. Privacy & Security: Implementing encryption & on-device processing to protect user data.

Accomplishments that we're proud of

The importance of multimodal accessibility—users benefit more from a combination of AI-powered visual, auditory, and haptic feedback. AI model optimization for real-time wearable use is key to achieving low latency and high accuracy. Ensuring user privacy and on-device processing builds trust and security in AI-driven assistive devices.

What we learned

What's next for SmartGlass

Expanding AI Training: Improve sign language recognition with more diverse datasets. Elderly Assistance Features: Integrate fall detection and vision magnification. Medical Applications: Adapt SmartGlass for surgeons to overlay medical imagery in real-time. Voice Control & Customization: Enable users to personalize voice commands and haptic sensitivity. Commercial Development: Refine the prototype and explore partnerships for large-scale manufacturing.

Built With

• tesseract
• yolo