SoundSight

Inspiration

For individuals who are deaf or hard of hearing, everyday sounds can carry important information that isn't always accessible. Whether it's someone calling their name, a knock at the door, or an alarm going off, missing these sounds can create challenges in everyday life.

We created SoundSight with the goal of bridging this gap through technology. Our vision was to build a digital hearing aid that makes both speech and environmental sounds more accessible, helping users stay connected to conversations and aware of their surroundings.

What it does

SoundSight is an accessibility application designed to translate sound into meaningful visual information and support two-way communication.

The application focuses on three core features:

Live Speech Captions: Transcribes spoken conversations into real-time text, making it easier for users to follow conversations.
Environmental Sound Detection: Recognizes important sounds, such as alarms, doorbells, knocking, and barking dogs, and presents relevant alerts.
Text-to-Speech Communication: Allows users to type responses that are converted into natural-sounding speech, making communication more accessible.

SoundSight is built as a mobile-friendly Progressive Web App (PWA), allowing users to access it through their smartphone without downloading a traditional native application.

How we built it

We built SoundSight using Next.js, React, TypeScript, and Tailwind CSS for the frontend, alongside Python and FastAPI for the backend.

To power its core functionality, we used several existing technologies:

Deepgram Nova-3 for real-time speech recognition and transcription.
TensorFlow YAMNet for environmental sound classification.
ElevenLabs for natural-sounding text-to-speech communication.

Our application brings these technologies together through a mobile-friendly interface, with the goal of making audio information accessible in real time.

Challenges we faced

One of our biggest challenges was bringing several different technologies together into a cohesive application. Real-time speech transcription, environmental sound detection, and text-to-speech each have different processing requirements and communication flows.

We also had to consider latency, accuracy, and responsiveness, particularly when identifying important sounds without overwhelming users with unnecessary alerts. Designing around mobile browser limitations while working within a tight hackathon deadline required us to prioritize the most essential features.

What we learned

Building SoundSight helped us gain practical experience with real-time audio processing, integrating external APIs, backend communication, and Progressive Web App development.

We also learned how important it is to consider accessibility when designing software, especially when the application is intended to help users interact with their surroundings and communicate more easily.

What's next for SoundSight

We hope to continue improving SoundSight by expanding its sound recognition capabilities, reducing detection latency, and introducing customizable alerts based on users' needs.

Future possibilities include sound-direction detection, more advanced background sound monitoring, and improved personalization.

Our long-term goal is to make SoundSight an affordable and accessible digital hearing companion that helps deaf and hard-of-hearing individuals navigate everyday life with greater awareness and independence.