Aware — Making the World More Accessible for Deaf and Hard-of-Hearing People

Inspiration

Imagine someone knocking on your door, your friend calling your name, or a fire alarm going off, and you don't even realize it's happening.

For millions of deaf and hard-of-hearing people, these are everyday challenges. So much of our world communicates through sound, yet not everyone has equal access to it.

That made us wonder: What if we could turn everyday sounds into something people could see and feel?

That's how Aware was born.

We wanted to build something simple but meaningful, an AI-powered companion that recognizes important sounds and turns them into vibrations and visual alerts. Whether it's a crying baby, a car honking, or someone calling your name, Aware aims to make everyday moments safer and more accessible.

Because sometimes, the smallest things make the biggest difference.

Awareness shouldn't depend on hearing.

What it does

Aware is an AI-powered accessibility companion that turns sounds into awareness for deaf and hard-of-hearing people.

It listens to surrounding sounds and alerts users through vibrations and visual notifications, helping them stay connected to what's happening around them.

Aware can recognize everyday sounds such as doorbells, knocks, crying babies, car horns, dog barks, and emergency alarms. It can also alert users when their name is detected in recognized speech.

Beyond sound detection, Aware supports the on-device Gemma 3 4B model to summarize recent sound detections and speech transcripts, helping users understand what's happening around them.

Our goal is simple: make the world more accessible, one sound at a time.

How we built it

We built Aware as a native Android application using Kotlin, Jetpack Compose, and Material 3. We used Kotlin Coroutines and Flow to handle audio processing and live interface updates, while Android DataStore saves users’ preferences locally.

For environmental sound detection, we capture microphone audio through Android’s AudioRecord API and process it locally using Google’s YAMNet model through MediaPipe Audio Classifier. We map 36 model labels into 15 user-facing sound types, including doorbells, car horns, crying babies, and emergency alarms. Our alert engine uses confidence thresholds, repeated detections, and cooldowns to filter uncertain results and avoid excessive alerts.

We connect these detections to Android’s vibration and notification APIs, using distinct vibration patterns for different sound categories. An Android foreground service keeps detection running in the background. For live captions and name-call detection, we use Android’s SpeechRecognizer, selecting on-device recognition where supported. Google ML Kit Translation provides translation using downloaded language models.

For contextual understanding, we integrated optional Gemma 3 3B, using a quantized model through MediaPipe LLM Inference. It combines recent sound events and recognized speech into short situation summaries on the device. Basic alerts continue working independently of the LLM, with template-based summaries available when the model is unavailable.

Challenges we ran into

Long app build times: Compiling a single iteration sometimes took 30 minutes, slowing down development and making it harder to test changes quickly.
Creating the 3D signing avatar: Building an accurate, natural-looking avatar was challenging, especially when refining hand shapes, finger positions, and signing movements.

Accomplishments that we're proud of

Offline functionality: Making the app run entirely offline on users’ devices. This helps protect the privacy of their everyday conversations and ensures that an internet connection is never a barrier.
A 3D signing avatar: Representing live speech through a 3D avatar with 64 core word and phrase signs, support for 141 word forms, and fingerspelling for all 26 English letters and digits 0–9. Unsupported words and names automatically fall back to fingerspelling.
Environmental sound detection: Mapping 36 model labels to 15 user-facing sound types, including sirens, smoke alarms, doorbells, car horns, crying babies, and barking dogs.
A complete mobile app: Building a full-fledged Android application with a clean, intuitive UI and UX.
On-device contextual analysis: Running a local large language model (Gemma 3B) to help users understand what is happening around them.

What we learned

The challenges deaf and hard-of-hearing people face in their everyday lives.
Edge computing, including the challenges of running AI models efficiently on mobile devices.
Android development and designing accessible mobile experiences.

What's next for Aware

In the next version of Aware, we plan to extend the app to smartwatches and Meta glasses. Visual input from the glasses could improve contextual analysis, while wearable alerts could help users stay aware of their surroundings without taking their phones out.

We also plan to explore automatic spoken responses to support everyday conversations. For example, if someone says, “Hey Alex, how are you?”, Aware could recognize the greeting, generate an appropriate response, and speak it aloud. This could help deaf users who have difficulty speaking participate more easily in spoken conversations.