Hearo.

Inspiration

Hearo started with the problem of hearing fatigue.

Hearing devices can make communication easier, but constantly processing amplified sound can become exhausting in loud or overstimulating environments. Someone may want to remove their hearing device at an airport without missing a boarding announcement. A parent may need a quiet moment while still knowing if their baby starts crying.

We wanted to create a way for people to rest without disconnecting from what’s happening around them.

What it does

Hearo lets users describe exactly what they don’t want to miss.

Airport example: “Tell me when Flight 312 starts boarding.”

Café example: “Let me know when order number 47 is ready.”

Hearo monitors both spoken announcements and environmental sounds. It can recognize events such as a doorbell or siren. When something relevant happens, Hearo sends a notification to the user’s phone and a discreet haptic alert to their Apple Watch.

This allows users to step away from an overstimulating soundscape while staying aware of the moments that matter.

How we built it

We built native iPhone and Apple Watch apps using Swift and SwiftUI. Hearo then processes the environment through two real-time detection lanes.

Speech detection: The iPhone streams live audio to our FastAPI backend through a WebSocket connection. Deepgram Nova-3 converts that audio into real-time transcripts.
Intent understanding: Gemini turns a request like “Let me know when Dana is called” into structured intent. It identifies spoken variations and the surrounding context that could indicate a match.
Environmental sound detection: Apple Sound Analysis recognizes non-speech events such as a doorbell or baby crying.

Our custom matcher evaluates finalized speech against the Gemini-generated intent. This keeps alerts fast and predictable. When either detection lane finds a match, the iPhone sends a standardized alert to the Apple Watch and triggers the appropriate haptic feedback.

SQLite and SQLAlchemy preserve requests and alert history across sessions.

Challenges we ran into

Processing live speech

Live speech doesn’t arrive as a clean sentence. We had to separate unstable interim words from finalized speech. Waiting too long made alerts feel delayed, while evaluating too early caused incorrect matches.

Understanding spoken variations

Names and numbers can be transcribed in several ways.

For example, 312 might appear as “three twelve” or “three hundred twelve.”

Gemini helped us generate these variations before matching them against Deepgram’s transcript.

Connecting multiple detection systems

Spoken announcements and environmental sounds require different detection methods. We had to combine Deepgram’s transcription pipeline with Apple Sound Analysis without producing conflicting alerts.

Delivering alerts across devices

An event travels from the microphone to the backend before returning to the iPhone and Apple Watch. Coordinating that pipeline while keeping haptic feedback responsive required careful testing.

Accomplishments that we're proud of

We built a complete experience that works across a Python backend, an iPhone, and an Apple Watch.

Hearo isn’t restricted to a few hard-coded situations. Users can describe a custom event in everyday language. The same system can recognize an airport announcement during one session and a café order during the next.

We also connected Apple Sound Analysis to real Apple Watch haptics. Important events don’t stop at a transcript or dashboard notification. They reach the user through immediate physical feedback.

Most importantly, Hearo addresses hearing fatigue without asking users to give up awareness of their surroundings.

What we learned

Our biggest lesson was that hearing everything isn’t the same as understanding what matters.

A transcript can capture every word correctly and still fail the user if it can’t identify which moment deserves their attention. At an airport, dozens of announcements may play before Flight 312 is called. In a café, nearby conversations don’t matter when someone is waiting for order 47.

Gemini helps Hearo understand the user’s intent. Our matcher then looks for that meaning in real-world speech.