Heard: Speech recognition that learns you

Inspiration

We've reached a point where you can ask your phone to solve a math problem or explain quantum physics, but for millions of people with speech differences, getting that same phone to understand a simple sentence can be incredibly difficult. With family members who experience speech difficulties, this problem felt personal to us. We also spoke with an occupational therapist who explained how communication barriers can affect someone's everyday independence. It made us think about all the little interactions we take for granted, like ordering food or texting a family member. We kept coming back to one question: Why should people have to adapt to technology instead of technology adapting to them? That's how Heard was born.

What it does

Heard is a speech recognition app that learns how you speak rather than expecting you to speak a certain way. Users record themselves saying a phrase three times, and Heard learns to recognize their pronunciation. From there, they can hold a button, speak naturally, and have their words displayed on a large card, spoken aloud, or sent through iMessage. But here's our favorite part: Heard knows when it doesn't know. Instead of confidently guessing the wrong phrase, it gives users three picture-based options to choose from. And every time someone confirms a phrase, Heard learns from that recording, improving its recognition over time. Think of it as a voice assistant that actually gets better at listening.

How we built it

We built Heard as a mobile-first web application, using Python and FastAPI for the backend and HTML, CSS, and JavaScript for the frontend. The biggest technical challenge was figuring out how to recognize speech that conventional speech-to-text models struggle with. Rather than relying entirely on a large pretrained model, we built a personalized audio-matching system. When someone records a phrase, we use FFmpeg to clean and standardize the audio. We then extract Mel-frequency cepstral coefficients (MFCCs), which essentially capture important characteristics of the sound. Next, we use dynamic time warping (DTW) to compare recordings, even when the same phrase is spoken at different speeds. This allows Heard to match someone's speech against examples they've recorded previously. We also built a confidence-checking system. If two phrases have similar scores, Heard asks the user to clarify instead of automatically choosing one. For spoken output, we integrated ElevenLabs, with browser-based speech synthesis as a fallback. We also connected Photon to support sending confirmed messages through iMessage. Finally, we implemented a feedback loop that saves confirmed recordings, allowing the system to improve its personalized matching over time. All of this runs through a simple interface designed around large buttons, visual cues, and as few unnecessary steps as possible.

Challenges we ran into

Our biggest challenge was realizing that building speech recognition for people whom existing systems struggle to understand requires a completely different approach. Traditional speech-to-text models weren't enough, so we turned to personalized audio matching. But people don't say the same phrase exactly the same way every time, which made accounting for speaking speed, background noise, and pronunciation differences tricky. We also had to figure out when Heard should trust its predictions and when it should ask for clarification. Connecting audio processing, speech synthesis, and iMessage into one smooth experience was another challenge. And, of course, we experienced the classic hackathon tradition of fixing one bug only to discover three more.

Accomplishments that we're proud of

We're proud that Heard became more than just an idea. We built a working application that can learn someone's phrases, recognize them later, speak them aloud, and help communicate them through messages. But what we're especially proud of is how simple we made the experience. Behind one big button is an entire system of audio processing, feature extraction, and personalized matching. We also designed Heard to recognize uncertainty rather than confidently communicate the wrong thing. Speaking with an occupational therapist reinforced why this matters. At the end of the day, the most important part of Heard isn't the algorithm. It's the person using it.

What we learned

One of our biggest lessons was that the most technically impressive solution isn't always the most useful one. Initially, the idea of recognizing every possible word sounded exciting, but we realized that reliably understanding a smaller set of personally important phrases could be far more valuable. We also learned a lot about audio processing and just how difficult it is to recognize the same phrase spoken in different ways. Most importantly, we realized that accessibility isn't something you add at the end. It shapes every design decision, from button sizes to what happens when the system makes a mistake. Good technology shouldn't expect everyone to interact with it the same way.

What's next for Heard: Speech recognition that learns you

One of our biggest lessons was that the most technically impressive solution isn't always the most useful one. Initially, the idea of recognizing every possible word sounded exciting, but we realized that reliably understanding a smaller set of personally important phrases could be far more valuable. We also learned a lot about audio processing and just how difficult it is to recognize the same phrase spoken in different ways. Most importantly, we realized that accessibility isn't something you add at the end. It shapes every design decision, from button sizes to what happens when the system makes a mistake. We also want to continue to improve the backend infrastructure and voice understanding.