SightLine

Inspiration

Navigating everyday environments can be challenging for people with visual impairments, especially when important information about their surroundings isn't easily accessible. We wanted to explore how advances in computer vision and AI could make the physical world more accessible through a simple, intuitive, voice-first experience.

What it does

Sightline is an AI-powered accessibility assistant designed to help visually impaired users better understand and navigate their surroundings. Using real-time visual information, Sightline identifies objects, interprets the environment, and provides helpful spoken guidance. Users can also interact with the system through natural voice commands, allowing for a hands-free experience.

How we built it

We built Sightline by combining vision-language models, computer vision, and speech technologies. Perceptron's vision-language models interpret visual information and generate contextual guidance, while ElevenLabs powers speech recognition and text-to-speech. We also integrated local wake-word detection and voice activity detection to support hands-free interaction. These components work together to create a pipeline connecting visual understanding with natural language interaction.

Challenges we ran into

Some of our biggest challenges involved coordinating multiple AI components, managing latency, and making voice interactions reliable. We also had to consider how to provide useful guidance without overwhelming users with unnecessary information, particularly when dealing with changing real-world environments.

Accomplishments that we're proud of

We're proud of building a system that brings together visual understanding and natural voice interaction to address a meaningful accessibility challenge. Developing a working hands-free audio pipeline and bringing multiple technologies together within a limited hackathon timeframe were important milestones for our team.

What we learned

Through Sightline, we gained experience integrating multimodal AI systems, working with vision-language models, and designing voice-first interactions. We also learned how important reliability, accessibility, and thoughtful communication are when building technology that people may depend on in their everyday lives.

What's next for Sightline

We envision expanding Sightline into a more robust assistive platform with improved real-time scene understanding, personalized guidance, and support across different devices. Future improvements could include more accurate spatial awareness, faster interactions, and additional accessibility features informed by feedback from visually impaired users.