Home light mode
Listen light mode
Teach light mode
Sounds light mode
History light mode
Login light mode
Home dark mode
Listen dark mode
Teach dark mode
Sounds dark mode
History dark mode
Login dark mode

Inspiration

Here when it matters. Hearing what matters.

Everyday sounds carry important information, from a knock at the door to a smoke alarm or glass breaking. For individuals who are deaf or hard of hearing, these sounds may go unnoticed.

Our team wanted to explore how artificial intelligence could make environmental awareness more accessible. With backgrounds spanning computer science, statistics, sociology, and digital arts, we combined our technical and creative perspectives to develop a practical, accessibility-focused application.

One team member's two years of American Sign Language (ASL) coursework also encouraged us to think more deeply about communication and accessibility.

What It Does

Hear for You is a web application that identifies important environmental sounds and provides real-time visual alerts.

Users can:

Detect everyday sounds such as dog barking, door knocking, glass breaking, and alarms.
Teach custom sounds by recording examples of sounds specific to their environment.
Customize alerts by enabling or disabling individual sound categories.
Review detection history to see previously recognized sounds.
Use the application without an account, with local storage for history and preferences.

Our goal is to help users stay informed about their surroundings without depending exclusively on hearing.

How We Built It

We developed Hear for You using React, TypeScript, Vite, and Tailwind CSS for the frontend, with a Python FastAPI backend responsible for audio processing and AI integration.

The browser's Web Audio API captures microphone input and processes it into 16 kHz, mono, 16-bit PCM WAV audio.

We integrated Google's Gemini API, using its multimodal embedding capabilities to represent and compare environmental sounds. The system supports predefined sound categories and user-taught sounds through example recordings.

We also used SQLite for backend sound data, browser storage for guest preferences and history, Figma for interface design, and GitHub for collaborative development.

Challenges We Faced

One of our greatest challenges was balancing recognition sensitivity and accuracy. While overly sensitive detection generated false alerts from typing and conversations, restrictive filtering could cause important sounds, such as door knocking, to be missed.

We also encountered challenges with microphone permissions, exact audio recording durations, browser compatibility, and integrating our frontend with an evolving AI backend.

These problems required repeated testing, debugging, and close collaboration across our team.

Accomplishments We're Proud Of

We built a functional web application that combines real-time environmental sound recognition with customizable alerts and the ability to teach new sounds.

We're especially proud of creating an interface that users can access without creating an account, while supporting personalized sound preferences and detection history.

What We Learned

This project strengthened our experience with full-stack development, audio processing, AI integration, accessibility-focused design, and collaborative software engineering.

More importantly, we learned that building accessible technology involves more than implementing features. It requires considering how reliably those features work in real-world situations.

What's Next

Our long-term goal is to develop Hear for You as a mobile application with vibration-based alerts and notifications, allowing deaf and hard-of-hearing users to stay aware of important sounds without constantly checking their screens. Ideally, the application would support continuous background listening, even when the phone is locked or the user is using another application.

We also envision expanding Hear for You into a smart home accessibility system by incorporating small, interconnected sound-detection devices placed throughout the home. These devices could improve recognition of sounds occurring in other rooms and send alerts directly to the user's phone, helping users remain aware of their surroundings regardless of where they are in their homes.

Additionally, we hope to explore alternative AI models for audio recognition to improve detection accuracy, reduce false alerts, and recognize a wider variety of environmental sounds. We would also love to collaborate with deaf and hard-of-hearing individuals to gather feedback and ensure our technology addresses their everyday needs.

What's next for Hear For You

Our long-term goal is to develop Hear for You as a mobile application with vibration-based alerts and notifications, allowing deaf and hard-of-hearing users to stay aware of important sounds without constantly checking their screens. Ideally, the application would support continuous background listening, even when the phone is locked or the user is using another application.

We also envision expanding Hear for You into a smart home accessibility system by incorporating small, interconnected sound-detection devices placed throughout the home. These devices could improve recognition of sounds occurring in other rooms and send alerts directly to the user's phone, helping users remain aware of their surroundings regardless of where they are in their homes.

Additionally, we hope to explore alternative AI models for audio recognition to improve detection accuracy, reduce false alerts, and recognize a wider variety of environmental sounds. We would also love to collaborate with deaf and hard-of-hearing individuals to gather feedback and ensure our technology addresses their everyday needs.