About the Project: DreamSense
Inspiration
Globally, over 285 million individuals live with visual impairments, yet most assistive technologies provide only limited, linear interaction with their surroundings. DreamSense was conceived to reimagine accessibility — using the power of AI to deliver dynamic, context-aware environmental understanding through natural language and spatial audio.
Our goal was to develop a solution that empowers visually impaired users to perceive their environment more intuitively, enhancing mobility, safety, and confidence in daily life.
What I Built
DreamSense is an AI-powered assistant that interprets real-world visuals and conveys them through natural, spoken descriptions. Key components include:
- Object Detection: YOLOv8 identifies relevant objects and people in the environment.
- Contextual Scene Understanding: GPT-4 Vision generates detailed, human-like descriptions based on the visual input and detected elements.
- Speech Output: The generated description is delivered through real-time text-to-speech (TTS), enabling users to hear what's around them.
- Interface: The initial prototype is a desktop application built with Python (Flask, OpenCV), optimized for low-latency performance and accessibility.
Key Learnings
- Integration of multi-modal AI models for assistive technology.
- Advanced prompt engineering for scene-based description generation.
- Optimization of real-time vision pipelines for responsiveness and usability.
- Best practices in accessibility-centric design and user interaction.
Challenges
- Ensuring low-latency performance with multiple AI components.
- Balancing detail and clarity in spoken output for users with different needs.
- Designing an architecture adaptable to future wearable and mobile implementations.
Next Steps
- Expand to a mobile and Raspberry Pi-based wearable version with spatial audio.
- Add voice command capabilities using Whisper for hands-free interaction.
- Enable offline or hybrid inference for broader deployment in low-connectivity environments.
DreamSense represents a step forward in assistive innovation — transforming how users with visual impairments experience and engage with the world around them through accessible, intelligent, and human-centered AI.
Log in or sign up for Devpost to join the conversation.