Synesthesia

Inspiration

We were inspired by Minecraft’s Sound Subtitles and many video games’ visual sound effects, which show players where important sounds are coming from through on-screen indicators. We realized that this same idea could be used outside gaming to help people who are Deaf or hard of hearing understand the sounds happening around them, and stay more connected to the people they are with.

What We Built

Synesthesia is a prototype AR-style sound-awareness system that helps users “see” nearby sound. A siren can appear as a red-and-blue alert on the side it came from, a knock can become a focused pulse in front of the wearer, and a person saying a saved name can trigger a clear visual notification and vibration.

By turning sounds into visual, spatial cues, Synesthesia can help users notice when someone is trying to get their attention and participate more confidently in conversations, homes, classrooms, and social spaces.

How It Works

A Raspberry Pi Pico reads three microphone channels and estimates the rough direction of a sound based on relative sound levels. An optional rear-facing sensor provides an additional warning for sounds that may be behind the user.

The Pico sends direction events over UART to an ESP32, which forwards them to the phone through Bluetooth Low Energy.

On the phone, the web app uses the device microphone and an on-device YAMNet audio-classification model to recognize sounds such as alarms, sirens, horns, knocks, speech, dogs, doorbells, and more. The app pairs the recognized sound with a nearby direction event and displays an animated visual cue using color, shape, iconography, and motion.

Users can also save their name and nicknames. Where supported, local speech recognition detects them and creates a more personal alert. The phone camera can provide a live background, creating an AR-style overlay that places sound cues into the user’s view of the real world.

Why AI Is Essential

AI transforms raw sound into useful meaning. A simple volume sensor could show that something happened, but it could not distinguish a car horn from a knock, recognize speech, or identify when someone says the user’s name.

Synesthesia uses on-device AI sound classification and local speech recognition to turn nearby audio into context-aware visual alerts. This is what makes the system useful for both safety awareness and everyday human connection.

Privacy

Privacy was a major design goal. Audio is not transmitted over Bluetooth or uploaded to a server. The ESP32 sends only small direction packets, while sound classification and supported name recognition run locally on the phone.

Challenges We Faced

The largest challenge was direction detection. The Pico does not perform true audio triangulation; it compares microphone loudness. This can be affected by echoes, microphone gain differences, distance, overlapping sounds, and background noise.

To make the system more honest and useful, we added ambient-noise calibration, event thresholds, clipping detection, an ambiguity score, and a rear-only warning mode. When the system is unsure, it avoids claiming an exact direction.

Another challenge was combining the sensor’s direction event with the phone’s sound classification. They do not share a clock, and multiple sounds can happen at once. We built a timing-based fusion layer that assigns a direction only when there is a clear nearby match. Otherwise, the app still shows the sound type without presenting a potentially incorrect bearing.

What We Learned

Through this project, we learned about MicroPython firmware, analog microphone sensing, UART communication, Bluetooth Low Energy, browser hardware APIs, local machine-learning inference with MediaPipe and YAMNet, Canvas animation, and accessibility-focused interaction design.

Most importantly, we learned that accessibility technology should communicate uncertainty clearly. Rather than treating every detection as certain, Synesthesia distinguishes between a confident cue, an unclear direction, and a possible sound behind the user.

Looking Forward

Synesthesia is still a prototype, not a safety-certified navigation device. However, it demonstrates how a familiar idea from games can become a meaningful real-world accessibility tool.

In the future, we hope to bring Synesthesia to AR glasses, using their microphones and displays to make sound awareness feel more immediate, immersive, and naturally integrated into everyday life.