Inspiration

You can't hear your own speaking habits in the moment. You find out you rushed, rambled or said "like" forty times only afterwards, if anyone tells you at all. Feedback that arrives a day later doesn't change what you do mid-sentence. We wanted a coach that is there during the conversation and stays out of the way.

What it does

Undertone is a speech coach you wear on your wrist.

  • Listens all day. A Vexo band captures your conversations automatically. There is nothing to start or stop.
  • Coaches you live. A silent buzz on your phone tells you to slow down or drop the filler words, without anyone else noticing.
  • Reads your hands. The band's motion sensor tells still hands from gesturing and fidgeting.
  • Debriefs every conversation. One thing to try next time, what went well, the transcript and the recording.
  • Remembers people. It recognises voices across conversations, labels relationships such as friend, boss or coworker from how you talk, and maps the people, topics, habits and promises in your life.
  • Practice with your coach. The coach asks you questions out loud, you answer, and it gives you a score and simple goals to work on.
  • Builds coaches from a sentence. Type a goal like "help me sound confident in interviews" and it builds a coach for that.

How we built it

Vexo band and open kit

The Vexo band is the whole input side of Undertone. Using the Vexo open kit, we stream its microphone audio and its accelerometer and gyroscope data to a laptop over Bluetooth Low Energy.

  • The microphone feeds always-on capture: the backend decides by itself when a conversation starts and ends.
  • The motion sensors feed a classifier that labels each second as still, gesturing or fidgeting, after a 15-second personal calibration. We combine that with who is speaking, so "fidgeting while they talk" becomes a signal.
  • Because it sits on the wrist, it works in real conversations where a phone on the table or a laptop mic would not.

Deepgram

Deepgram is how Undertone hears.

  • Real-time speech-to-text through Pipecat, with word-level timings, live partial transcripts and speaker labels. Word timings are what let us measure pace, pauses, interruptions and talk share.
  • We chose the Nova model that keeps filler words such as "um" and "uh" in the transcript, because those are exactly what we coach on, and we pass each coach's own vocabulary to it.
  • Deepgram Voice Agent powers "Practice with your coach": one connection that listens, thinks and speaks, so the coach can interview you out loud.
  • Deepgram text-to-speech reads the spoken recap after a conversation.

Google Gemini

Gemini is the judgement behind the coach. Every call returns structured output that our code validates.

  • Live judge: scores each turn as it happens against the active coach's questions, fast enough to trigger a nudge mid-conversation.
  • Debrief: writes the one thing to try and what went well, from the measured numbers.
  • Coach builder: turns a one-sentence goal into a full coach with its own rules, questions and nudges.
  • Relationships and topics: infers how you relate to each person and what you talked about.
  • Practice scoring and daily recap: turns measurements into a score and plain-language goals. Every Gemini feature has a rule-based fallback, so the product keeps working if the model is slow or offline.

The rest of the stack

  • Backend: Python and FastAPI, SQLite, and a WebSocket bus that pushes live events to the phone and the website.
  • Voice memory: on-device speaker embeddings (TitaNet via sherpa-onnx) so it knows your voice from other people's across conversations.
  • Phone app: Expo and React Native, for haptic nudges and a live view.
  • Website: Next.js, with Live, Insights, Conversations, Coach and Settings pages.

Challenges we ran into

  • Telling the wearer's voice from everyone else's with a single wrist microphone.
  • Making always-on capture split speech into sensible conversations instead of fragments.
  • Keeping the live judgement fast enough to nudge in the moment.
  • Nudging without distracting: we limit buzzes so they help rather than throw you off.

Accomplishments that we're proud of

The whole loop runs for real: from the band, through transcription and judgement, to a buzz on the phone and a debrief you can read afterwards.

What we learned

One clear thing to fix beats a wall of numbers, so the coach speaks in plain sentences. And a wrist is a surprisingly good place to listen from.

What's next for Undertone

Showing the end-of-day recap in the app, stronger speaker recognition in noisy rooms, and proper privacy controls and sign-in.

Share this project:

Updates

Submission history