Playback

Inspiration

One of us walked out of a call with a grandparent and realised he'd asked one question in ten minutes, talked over them twice, and let "your grandmother really misses you" go by without a word. He didn't notice any of it while it was happening. Nobody does. The people we love most are the ones we listen to worst, and there has never been a mirror for that conversation.

Every online source we tried was a chatbot you describe the conversation to afterwards. It can't hear you. It doesn't know when you started talking. It takes your word for what happened. We wanted something that measures the conversation first and only then lets a model open its mouth.

What it does

Playback is a replay for your conversations. A mic on a pair of glasses and your phone's camera record a real conversation; afterwards you get the moment you talked over someone, the thing they brought up that you let drop, and the one follow-up message that fixes it.

Record. Tap once on your phone. The glasses mic streams audio to Muse Voice Transcribe in real time with speaker diarization; the phone camera watches the person you're talking to. Nobody types a name — if you say "Hi, Grandpa" or they say "I'm Priya", Playback picks it up.
Lip lock. The camera runs a face landmarker at 8 fps on the phone and votes, per line, whether the mouth in frame was moving (theirs), the close mic was loud (mine), or neither (somebody at the next table). That's how it knows which diarized voice is you, and which lines to throw out as background — without ever uploading video.
Measure. Deterministic NLP over the transcript: questions and whether they built on what the other person just said, cut-ins (their sentence stops mid-word, yours starts), hedges ("I just kind of…"), short replies, missed bids (they raise something, your next line moves on), commitments you made, topics they kept coming back to.
Coach. Muse Spark gets the transcript and the measurements, and has to answer inside a strict JSON schema: three takeaways (try this instead / keep doing this / ask about this next time), a follow-up message, and an "advice check" that says whether each piece of advice is backed by what the camera saw. Any takeaway that quotes a line not in the transcript is thrown out.
Replay on your Mac: the recording with a live face mesh, a timeline of every question, missed moment, cut-in, promise and on-camera moment, a word heatmap over the transcript, and the follow-up draft. Click any moment and the video jumps there.
Over time: a before/after strip against your last conversation with the same person, a pre-meeting brief ("last time she mentioned the move to Austin"), and a Home page that tracks streaks, goals and follow-ups you haven't sent.

It handles English and Spanish, including switching mid-sentence.

How we built it

System architecture

Layer	Technology	Role
Capture	Glasses-clipped mic, phone camera	Close audio from the wearer, video of the other person
Phone app	AudioWorklet, MediaPipe Face Landmarker, MediaRecorder	Audio leveling, lip-lock cues, local video recording
Bridge	Node 24, ws	Paces audio to Muse, reconnects without breaking the timeline
Transcription	Muse Voice Transcribe	Realtime transcript with speakers, English and Spanish
Measurement	Rule-based NLP (no model)	Questions, cut-ins, hedges, missed bids, promises, names
Coaching	Muse Spark + reviewer pass	Grounded takeaways and a follow-up draft
Server and storage	Express 5, node:sqlite	Transcripts, cue tracks, history per person
Front end	Vanilla JS, one shared WebSocket hub	Phone app, Mac Home / Live / Replay, projector stage
Replay	MediaPipe on the saved video	Smiles, nods, facing, and the face mesh

LIVE RECORDING

Glasses mic                        Phone camera
(wearer, close)                    (faces the other person)
      |                                    |
      v                                    v
+---------------------------------------------------+
| Phone app (browser)                               |
| Audio: evens out both voices, streams it          |
| Lip lock: watches their mouth -> cue track        |
| Video: recorded locally                           |
+---------------------------------------------------+
      |                                    |
  audio + cue track                   video file
  (WebSocket)                         (Mac only, never uploaded)
      |                                    |
      v                                    v
+----------------------------+      +--------------+
| Node bridge + WS hub       |      | Mac          |
| Paces audio to Muse        |      | Replay page  |
| Reconnects + stitches      |      +--------------+
+----------------------------+
      ^          |
      |          +--> SQLite (transcript + cue track)
      v
+----------------------------+
| Muse Voice Transcribe      |
| realtime, with speakers    |
+----------------------------+

AFTER THE CONVERSATION

Transcript + cue track
             |
             v
Lip lock: each line tagged me / them / background
             |
             v
Rule-based NLP (no model)
questions, cut-ins, hedges, short replies,
missed bids, promises, recurring topics, names
             |
             v
Muse Spark (fixed JSON format)
takeaways + follow-up draft + advice check
             |
             v
Every quote checked against the transcript
Reviewer pass drops advice the numbers don't back
             |
             +--> Spark fails? rules-only fallback
             |
             v
Coaching report --> Mac Replay / Home

REPLAY ON THE MAC

Saved video --> face tracking --> smiles, nods, where they're looking
                                          |
                                          v
                            lined up with each line of the transcript
                                          |
Transcript + measurements + coaching ---->+
                                          |
                                          v
           Timeline, word heatmap, face mesh, follow-up draft,
           before/after vs. your last conversation with them

The details

Hardware: A small mic clipped onto a pair of glasses, plugged into the phone. The phone's camera points at the person you're talking to. That's the whole rig.

Audio: The person wearing the mic is always way louder than the person across the table, so the phone evens the two out before anything gets sent. Without that, the quiet side of the conversation barely showed up in the transcript. Audio streams to our server over a WebSocket. Video stays on the device and only ever ends up on the Mac.

Talking to Muse: Muse's realtime transcription hangs up if you send audio faster than it can handle, which is exactly what happens after a network hiccup when a backlog dumps all at once. So our Node server sits in the middle and feeds the audio out at a steady pace. If Muse drops the connection anyway, we open a new session and stitch it back onto the same timeline so the transcript doesn't break.

Figuring out who's talking: Diarization can tell two voices apart, but it has no idea which one is you. So we use the camera. If the other person's mouth is moving, the line is probably theirs. If the mic is loud and their mouth is still, it's probably you. If it's neither, it's someone at the next table and we drop it. Whichever voice lines up with "you" most often wins. No setup, no tagging your own voice.

Measuring the conversation (no AI here): Before any model sees anything, plain code goes through the transcript and finds the moments worth talking about. A cut-in is when their sentence trails off mid-thought and you jump in right away. A missed bid is when they bring up something that matters and it never shows up in your reply. It also counts questions, hedges, short answers, promises you made, and topics they kept coming back to. Names get picked up from how people actually talk, like "I'm Priya" or "how are you doing, Grandpa?"

Reading reactions: After the conversation, the Mac goes back through the saved video and tracks smiles, nods, and where the other person was looking, then lines it all up with the transcript. For anything you said, you can see how they reacted. We stick to what a camera can actually see. A nod is a nod. We don't guess at feelings.

Smooth face mesh: The mesh on the replay is filtered so it doesn't jitter when someone's sitting still but still keeps up when they move.

Keeping the AI honest: Muse Spark only ever gets the transcript together with the measurements, and it has to answer in a fixed format. Every piece of advice has to point to a real line from the conversation, and we check every quote. A second reviewer pass throws out anything the measurements don't back up. If Spark fails entirely, a rules-based fallback fills in so you never get a blank page.

Stack: Node, Express, SQLite, and plain JavaScript with no build step. The phone, the Mac, and the projector demo screen all talk through one shared WebSocket hub.

Challenges we ran into

Two voices, one mic. Diarization tells voices apart but not which one is you. Asking the wearer to tag their own voice every conversation killed the flow. The camera turned out to be the answer — but only after we got the mouth-motion baseline and the "close loud voice" cue to agree, and learned to throw out lines where they don't.

"Audio processing backlog too large." Muse's realtime endpoint hangs up (code 1008) if you send audio faster than it can process, which is exactly what happens when a client dumps buffered audio at it after a tunnel stall. We lost a whole recording to it mid-demo. The pacer and the mid-recording reconnect came out of that afternoon.

The quiet grandparent. One-sided audio: the wearer is loud, the other person is far. We tuned the leveler across real recordings until both sides were transcribable, and then had to undo clipping from an over-eager head-start gain.

Not being a wrapper. Early versions let Spark decide what went wrong; it praised a rambling monologue as good listening. The fix was architectural: Spark never sees the raw conversation without the measurements, and can only write about moments the detectors found.

Names. "Hi, Grandma." worked; "How are you doing Grandpa?" didn't, because the vocative grammar expected the name at the start of the line or after a comma. Family words are now a closed list that counts anywhere in a line — unless they're talked about ("Grandma really misses you").

Three audit passes. After a big front-end rework we ran parallel reviews with disjoint scopes, plus a headless-Chrome run of every flow. It found a frame that could crash the server, a timer that drifted after a reload, and a page that showed a stranger's pre-meeting brief against any unnamed conversation. Tests went from 241 to 286.

Accomplishments that we're proud of

A coaching note you can click: every takeaway lands on a second of video, with the face mesh and the numbers that justified it.
Zero typing. Who you're talking to, which voice is yours, who's background — all picked up from the mic and the camera.
Spanish/English code-switching works end to end, including the NLP.
Audio is never stored. Video never leaves the Mac. Only the transcript and the cue track persist.

What we learned

Measure first, generate second. Every time we let the model lead, it was confidently wrong. Every time we made it work from evidence, it was useful.
Realtime audio APIs have opinions about pacing. Respect them in the bridge, not in the client.
"Observable cues only" is a stronger product position than "emotion detection", and an honest one. A nod is a number, a feeling is not.
A feature you can't point at on screen doesn't exist to a judge. We lost the on-camera moments in a design port and only noticed when someone asked "where did the behavior analysis go?"