Sonomod

A real-time conversation accessibility app that lets you replay what you missed without missing what comes next.

Comment

Sonomod started with a friend.

One of our team members has a friend with Auditory Processing Disorder (APD). He described how frustrating conversations can become when he misses one word or part of a sentence. He may ask someone to repeat themselves several times and still not catch it. Eventually, it can become uncomfortable to keep asking, so sometimes he just continues the conversation without fully understanding what was said.

That made us ask:

What if you didn't have to interrupt a conversation just because you missed five seconds of it?

APD inspired Sonomod, but we quickly realized this problem is much broader. People who are hard of hearing, have difficulty processing speech, struggle in noisy environments, or simply miss an important moment can experience the same thing:

“Wait, what did they just say?”

That's why we built Sonomod.

Sometimes you don't need the entire conversation repeated.

You just need those five seconds back.

🎧 What it does

Sonomod is a real-time conversation accessibility app designed to help people stay in the conversation.

It provides live captions and lets users replay the previous 5, 10, or 20 seconds of a conversation.

With Bluetooth earbuds, users can press the physical media button on their earbuds to privately replay the last five seconds—without pulling out their phone or asking the speaker to repeat themselves.

Most importantly:

Sonomod keeps listening while replay happens.

The microphone and live transcription continue, so catching up on what you missed doesn't mean missing what comes next.

For longer conversations, Sonomod also uses AI to turn transcripts into useful information such as summaries, questions, decisions, and next steps.

Hear it again. Read it. Understand what mattered.

🛠️ How we built it

We built Sonomod primarily with Flutter and Dart, with native Kotlin for deeper Android audio and Bluetooth integration.

Sonomod captures 16 kHz mono PCM audio from the phone microphone. That audio is simultaneously:

  • Streamed through our backend using WebSockets to ElevenLabs for real-time transcription
  • Stored in a rolling audio buffer for instant 5, 10, or 20-second replay

We use the Gemini API to transform longer transcripts into summaries and useful takeaways.

For hands-free interaction, we used native Android MediaSession and media APIs so the physical button on Bluetooth earbuds can trigger the five-second replay.

Behind one simple interaction:

Press button → hear the last five seconds

...is a system coordinating live microphone capture, WebSockets, transcription, audio buffering, Bluetooth playback, Android media routing, Flutter, Kotlin, ElevenLabs, and Gemini.

🧩 Challenges we ran into

Our biggest challenge wasn't making each feature work.

It was making everything work at the same time.

Initially, replaying recent audio could interrupt microphone capture and live transcription.

But if replaying the last five seconds makes you miss the next five seconds, we haven't solved the problem.

We worked through Android audio focus and Bluetooth routing until Sonomod could replay recent audio while continuing to listen and transcribe.

Then we ran into an unexpected problem:

Spotify kept receiving the earbud button instead of Sonomod.

We had to dig into Android MediaSession routing and change how Sonomod establishes its playback presence so Android would route the physical button to our app.

We also added duplicate-event protection so:

One press = one replay.

🏆 Accomplishments that we're proud of

We're most proud that replay became a real physical interaction, not just another button on a screen.

A user can:

Miss something → press their earbud → hear the previous five seconds → keep listening.

And throughout that entire interaction, Sonomod continues listening and transcribing what happens next.

We're also proud that each part of our system solves a specific piece of the problem:

  • 🎙️ ElevenLabs — What was said?
  • ↶ Replay — Can I hear that again?
  • ✨ Gemini — What was important?

To the user, all of that technology disappears into one simple moment:

“I missed that.”

Press.

Hear it again.

🧠 What we learned

We learned that accessibility isn't just about providing more information.

It's about providing the right information, at the right moment, without creating another burden for the user.

Technically, we learned a lot about:

  • Real-time audio capture and WebSockets
  • Rolling audio buffers
  • Bluetooth playback and Android audio focus
  • MediaSession routing
  • Native Kotlin-to-Flutter communication
  • ElevenLabs and Gemini integration

But our biggest takeaway was simpler:

Technology doesn't always need to change the conversation.

Sometimes it just needs to help someone stay in it.

🚀 What's next for Sonomod

We don't want Sonomod to end when Hacklanta ends.

Next, we want to:

  • 🔊 Improve performance in noisy environments
  • 👥 Explore speaker-aware functionality
  • 🎛️ Add more personalized accessibility controls
  • 🎧 Expand hands-free earbud interactions
  • 🔒 Increase on-device processing for privacy
  • 📱 Fully test and support iOS

Most importantly, we want future versions of Sonomod to be shaped by the people who would actually use it—including people with APD, people who are hard of hearing, and people with other listening or auditory-processing difficulties.

Sonomod started because we listened to one person's experience.

Now we want to listen to more.

Because sometimes you don't need someone to repeat the entire conversation.

You just need those five seconds back.

Built With

Share this project:

Updates

Submission history