Inspiration

Conversations with someone new tend to die in the awkward middle. Once "what's your major?" and "where are you from?" are used up, answers get shorter, pauses get longer, and both people reach for their phones. Research by Kardas, Kumar & Epley shows that people expect deep talk with strangers to be more awkward than it really is, and they underestimate how interested the other person is. Small talk isn't a preference. It's a miscalibration.

We were inspired by dating-show formats where a neutral third party occasionally cuts in to spark the conversation. Then we asked: what if that third party actually listened? It would know both people, follow the rhythm of the talk, and only speak up when the moment needs it.

What it does

Breadcrumbs is an AI companion for two people meeting in person. A laptop sits on the table, listens, and stays silent while the conversation flows. When a pause runs long, it drops a crumb out loud: a question, a thread you both share, or a grounded fun fact.

Just talk (the default): no phones and no forms. Both people agree on one consent card and start talking. Breadcrumbs learns about each person from the conversation itself and suggests follow-ups. At the end, it shows a trail of threads to pick up next time.
With profiles: each person records a 60-second voice profile on their phone. The table maps their voices and opens with something they have in common. At the end, each person privately chooses whether to swap contacts.
It knows when not to talk. Say "wait, let me show you this" and it holds back. Voices away from the table, like a bystander, a TV or room noise, don't reset its timer.
Nothing is stored. Transcripts and the live memory stay in server memory and are wiped when you're done, when you ask, or after 2 hours.

How we built it

Listening: the browser streams microphone audio over a WebSocket to a FastAPI server, which relays it to Muse Voice Transcribe in realtime diarization mode to get speaker-labelled turns.
Conversation engine: a Python state machine tracks turns, pauses, staleness and "hold" intent, and picks one of three escalating pause thresholds:

T{Tnormal,Tlong,Texc}={1.5s, 3.0s, 4.5s},Thold=2T

Noise rejection: a turn only moves the pause clock if it comes from a seat at the table. We classify each turn using the diarization label, the voice-activity level and how much of the turn had speech. Steady background noise is absorbed by a noise floor that latches. A sound only counts as new speech if it rises clearly above that floor:

LturnLfloor+6dB

Crumbs: Claude (claude-opus-5-5, low effort, fast mode for the time-critical calls) writes crumbs as strict JSON. A validator then checks length and safety. We prefetch and cache the next crumb, so most are ready before the pause ends. ElevenLabs speaks them.
Frontend: React 19, TypeScript, Vite and Tailwind. The orb is drawn with raw WebGL. An AudioWorklet handles voice activity detection and echo-aware barge-in.
Testing:
- 762 pytest tests.
- 41 scripted conversation simulations that replay noisy rooms, bystanders and interruptions.
- An end-to-end smoke test in mock mode.
- A usage simulator that measures API calls and voice credits per session.

Challenges we ran into

Background noise froze the timer. Room chatter kept resetting the pause clock, so crumbs never fired. We rebuilt the turn model so only turns from the table count, and added the noise-floor and rise rule.
The app interrupted itself. Its own voice coming out of the speakers registered as someone talking. Echo-aware barge-in fixed that.
Latency: a crumb that arrives 8 seconds late is worse than no crumb. Prefetching, caching, fast mode and only voicing the next crumb brought steers down to about 3 s into a silence.
Diarization labels reset on reconnect. Speaker labels like "A" and "B" only last for one session, so mapping voices to people had to survive dropped connections.
Provider fallbacks: our Gemini key hit a billing error mid-hackathon. The fallback chain (Claude → Gemini → Muse Spark for text; ElevenLabs → Gemini TTS → browser speech for voice) kept the demo running.

Accomplishments that we're proud of

Crumbs land about 3 seconds into a lull. In our usage simulation, 7 of 8 were served from the cache instantly.
A setup with no phones and no forms that learns about both people from the conversation itself.
Privacy by design: no transcript is ever written to disk or logged.
Every check passes: 762 unit tests, 41 of 41 simulated conversations, 46 of 46 end-to-end checks and 170 web tests.

What we learned

Timing matters more than content. The best question at the wrong moment still kills a conversation.
Knowing when to stay silent is the hard part of building a conversational AI.
Real rooms are much noisier than test benches. Building simulations early saved us hours.

What's next for Breadcrumbs

Shorter crumbs that sound like a friend chiming in, rather than a facilitator.
No calibration step: the first crumb is just "Introduce yourselves," and Breadcrumbs learns who's who from the names they say. (This is half-built on a branch.)
Better voice isolation using a per-person voice fingerprint.
Tuning the noise thresholds in real, loud rooms like career fairs, dorm lounges and cafés.

Built with

python fastapi uvicorn pydantic websockets httpx react typescript vite tailwindcss framer-motion webgl web-audio-api audioworklet anthropic claude elevenlabs muse-voice-transcribe gemini cloudflare-tunnel ffmpeg pytest node.js

AI tools used

Tick Anthropic and ElevenLabs.

Gemini: it's wired into the code as a fallback, but the key hit a billing error and we never verified it live. Only tick it if you're comfortable with that.
Muse Voice Transcribe: add it under Other.

Did you implement a generative AI model or API?

Yes.

Anthropic Claude (Opus 5.5) writes every crumb in real time: questions, shared threads and grounded fun facts. In profile mode it also builds each person's profile from their spoken interview and finds what the two have in common. In Just talk mode it keeps a live memory of what each person has shared. We use low effort and fast mode on the time-critical calls, because a crumb has to land within about 3 seconds of a pause to feel natural.

ElevenLabs gives Breadcrumbs its voice, so it sounds like a third person at the table rather than a notification.

Muse Voice Transcribe provides realtime transcription with speaker diarization.

Gemini is wired in as a fallback for text and voice.

We used generative AI because the core problem, saying the right thing at the right moment to two specific people, can't be solved with a fixed deck of question cards.

The live demo is at https://logs-representing-healthy-satellite.trycloudflare.com, and the repo is https://github.com/amalsameel/breadcrumbs.