Inspiration

Babies talk to us all day long, but not in words. A wiggle, a pointed finger, a change in pitch, a face that crumples a second before the cry: the signals are there, and they're easy to miss. Parents end up in one of two modes. Either they watch constantly and wear themselves out, or they look away and wonder what they missed.

We wanted a third mode: be present without hovering. Hand the watching to a model, and let the parent glance at their phone only when it matters. We kept coming back to the research on serve-and-return: a baby "serves" (points, babbles, reaches) and a responsive adult "returns" it. A baby gets the most out of those moments when the parent notices them. BabySpeaker is our attempt to make sure fewer of those serves go unanswered.

What it does

BabySpeaker watches and listens during parent–child together time and turns what it sees and hears into one of three lock-screen states:

State	What it means	Example
All calm	Nothing needs you	Playing quietly, content
Needs your attention	Your baby is reaching out to you	Pointing at a plane: "Mom, look!"
A bit fussy	Restless, but not crying	Fussing heard, picture unclear

The app is designed to stay out of the way. You don't have to open it. Everything important arrives as a native iOS notification, written in your baby's voice ("I might be having a hard time. Check on me?"). Tapping it opens a few seconds of the moment that triggered the alert, along with the cues the model actually observed, so the parent can see why it fired.

It also keeps highlights: the happiest morning of the week, the first time pointing at a plane. These are small moments a busy parent would otherwise never get back.

How we built it

Perception. We send short windows of video and audio to the Perceptron Mk1.5 API. It returns observable cues (cry, coo, fuss, pointing, gaze, limb movement, background talk/TV) plus a state estimate for each modality.

Fusion backend (FastAPI). A /api/analyze endpoint combines the audio and video results for each window into a single label:

agree: both modalities report the same state
compatible: the two states are consistent
conflict: the two states disagree
single_modality: only one modality is usable
insufficient: not enough evidence

From that label it derives a state and a confidence level. A safety layer then decides whether to notify the parent, and fixed templates turn the state into a first-person "baby says" line.

We check a window of length every 10 seconds. To avoid false alarms, a single distressed window isn't enough.We alert only when sound and picture agree across consecutive windows. Weaker evidence (a single modality, or low confidence) maps to A bit fussy instead of Needs your attention.

Frontend. We designed the app for Expo/React Native on iOS. Alerts use iOS's built-in notifications (Time Sensitive for the "Needs your attention" state), with Live Activities as the next step. For the demo, we built a clickable web prototype of the lock screen. The notification and the app view are populated with the real outputs from a recorded run of our backend, and each notification opens the matching clip of the baby feed, so judges can see the experience end to end.

Challenges we ran into

Two senses that disagree. Audio might hear fussing while video sees a calm, sleeping baby. Background TV and adult conversation look a lot like baby sounds. Treating disagreement as a separate state, instead of forcing a single answer, was the key design decision in the backend.
Media plumbing. Perceptron ignores a video's soundtrack unless you explicitly enable it. We lost time to "why does audio never show up?" before we started sending audio separately (WAV/MP3/FLAC) alongside the video.
False alarms vs. missed moments. Every extra second of required agreement means fewer false alarms but slower alerts. The 20-second rule is a deliberate tradeoff, not a default we stumbled into.
Lock-screen constraints. A locked iPhone app can't keep checking the backend for new results, so the backend has to push notifications. That changed how we thought about the whole architecture.
Saying no to features. We started with a nap monitor and a "Baby Dictionary" that learned your baby's sounds. Both were interesting, and both made the product heavier. We cut them so the whole experience fits on the lock screen.

Accomplishments that we're proud of

A working multimodal pipeline that detects baby cues from real video and audio, explains each decision with the evidence behind it, and holds back when the evidence is weak.
A product that reduces a parent's workload instead of adding to it: three states, one glance, no app to babysit.
Notifications in the baby's voice, which make an alert read like a message from your kid rather than a monitoring system.
Being honest about uncertainty: "Can't tell" is a real state, and every alert shows the cues that triggered it.

What we learned

Accurate cue detection is the hard, valuable part to build first. Turning cues into words is only trustworthy if the cues are right.
In multimodal systems, how the modalities agree or disagree is a signal in itself, not just noise to average away.
For anything that interrupts a parent, design is mostly subtraction. Each screen we cut made the product better.
Building on an AI model requires reading the API's defaults carefully: the soundtrack setting is the obvious example.

What's next for BabySpeaker

Cue → word generation. Today our baby-voice lines come from fixed templates. Next we'll map detected cues (pointing + looking up + excited babble) to generated, context-aware phrases like "Look! Plane!".
Serve-and-return coaching. Notice when a baby "serves" and gently prompt the parent to return it, then track those exchanges over time.
Co-viewing. Understand what the baby is looking at on a shared screen or book, and suggest things to talk about.
Live Activities and Dynamic Island for a continuous, glanceable state during together time.
Highlights reel. Automatically save first-time moments and best days, entirely on the device.
Evaluation. Build a labeled dataset to measure precision and recall for each state, and to tune $\Delta$ and the agreement threshold properly.