Who said what, through one pair of glasses. AR glasses hear a whole conversation through one microphone, so every voice arrives mixed together. Picavoice separates that single stream into speakers, works out who each one is, and turns what they said into facts and a short summary. You see the result in Mind Palace XR: each person you meet becomes a room, with what they told you floating as objects and every conversation summarised on a timeline wall. The problem At a hackathon or conference you meet dozens of people in a day, and by the next morning most of them are a blur. Smart glasses are already on your face and already hear the room, so they should be able to remember people for you. But one microphone makes that hard, in three ways:
- Segmenting: when does each person speak? Everyone is recorded through the same glasses. The wearer is close and loud, everyone else is quieter, and people talk over each other in a noisy room. Transcription alone gives you words, not speakers.
- Identifying: who is that? Speaker labels like "Speaker 1" reset every conversation, so they can't tell you that today's Speaker 1 is the person you met yesterday. And a name said out loud can't be trusted on its own.
- Understanding: what mattered? A full transcript is too long to use mid-conversation. You want the few things worth remembering: what they do, what they're building, where you met, and the gist of what you talked about. And it has to happen with permission: nobody's voice should be remembered unless they agree. What it does Picavoice is one pipeline from the glasses' microphone to a memory you can walk into:
- Hear. Mentra Live glasses stream raw microphone audio, not just text, so the rest of the pipeline can tell voices apart. Any single microphone works the same way, so a laptop mic or a recorded conversation can stand in for the glasses.
- Separate. Every line is labelled with who spoke it (Speaker 0, Speaker 1…), live, from the single mixed stream.
- Ask, then recognise. When a new voice appears, you ask out loud, "Is it OK if I remember your voice?" A yes links that voice to a person. Next time, their saved voiceprint names them automatically. "Name only" keeps your notes but never matches their voice.
- Understand. What each person says is boiled down to facts (work, studies, places, likes, projects) and a short summary with keywords for each conversation.
- Remember, in XR. In Mind Palace XR, each person gets a room generated from their voice. Their facts float as 3D objects, and each conversation is a card on the timeline wall. It runs in a VR or AR headset, or on a laptop as a headset simulator. Two more views use the same people: the Picavoice phone app shows who is talking right now and what they told you last time, and Voice Garden grows each person into a tree beside their head in a webcam AR mirror. Deleting someone removes their voiceprint, transcript lines, room and tree. Everything stays on the laptop. How we built it Each of our team's voice projects became one stage of the pipeline. They share one database and one API on the laptop. Stage Project What it adds Built with Hear FieldAssist (glasses miniapp + laptop server) Raw 16 kHz microphone audio from Mentra Live to the laptop, posted every 250 ms over plain HTTP MentraOS Miniapp SDK, Bun, TypeScript Separate FieldAssist + Deepgram A live transcript where every line carries its speaker, saved with the recording Deepgram nova-3 speaker labels, SQLite Ask, then recognise Picavoice app + XR-Voice-Profiles Spoken consent, linking a voice to a person, and naming returning speakers from saved voiceprints Bun web app, SpeechBrain ECAPA voiceprints (Python) Understand Mind Palace XR Facts pulled from each sentence, plus a summary and keywords for each conversation Rule-based fact extractor, extractive summariser Remember, in XR Mind Palace XR, Voice Garden A room per person with facts and a timeline wall; a tree per person in an AR mirror Three.js, WebXR, MediaPipe Face Landmarker Small bridges connect the XR projects to the pipeline: everyone who agreed in Picavoice gets a room and a tree, and new lines from the glasses appear in their room as people talk. Mind Palace XR also works on its own with a laptop microphone, so the understanding and memory steps can be shown without glasses. To test without glasses, bun run replay plays any recorded conversation into the pipeline exactly as the glasses would. What we measured We recorded four short phone clips with three people and ran them through speaker separation and ECAPA voiceprints. • Separating two people in one recording worked. On a 105-second conversation the system picked two speakers on its own, and every long turn was labelled correctly. The only mistakes were one-word replies at turn changes, like "Okay." • Recognising the same voice in another recording worked for phrases over about 2 seconds. The same person scored 0.45–0.56 similarity across recordings; different people scored 0.25 at most. A cut-off around 0.35 separates them. • Very short answers can't be attributed. "Yes." and "And then…" were too short to match anyone reliably. • A spoken name isn't proof. In one clip, "This is Maya" was said by a different person, so a name should only be accepted when the voice matches.
Challenges we ran into Our Mentra Live glasses stopped working reliably during the hackathon, so the live demo uses the laptop microphone in their place. It has the same problem the glasses do: one microphone hearing everyone. Mind Palace XR carries the demo because it runs live on a laptop with no headset, and the glasses pipeline is shown with a recorded conversation, played in exactly the way the glasses would send it. Setup: the laptop runs the Picavoice server, with Mind Palace XR open in Chrome (microphone allowed) and the live transcript in a second tab. A volunteer stands at the laptop.
- The platform moved under us. Mentra hardware and SDK support was missing so we are unable to use the glass
- The phone couldn't hold a WebSocket connection to the laptop, so audio, transcripts and every screen run over plain HTTP.
- One microphone, many voices. Everyone is recorded through the same glasses, and the wearer is much louder than everyone else. Speaker labels also reset every conversation, so recognising someone later needs voiceprints.
- Short replies are hard. "Yes" or "Okay" is too short to tell who said it. -** Our team is down to 1 , two separated and one member wasn't feeling well
Accomplishments that we're proud of -audio all the way through to a speaker-labelled transcript and a person's profile.
- Consent built into the flow, and a delete that really removes everything, including the tree.
- On our own sample recordings, two people talking were separated cleanly, and the same voice matched across different recordings. Same-person phrases longer than 2 seconds scored 0.45–0.56 similarity; different people scored at most 0.25.
- Two team projects joined into one: Voice Garden now grows from the glasses conversation.
What we learned
- Telling voices apart within one conversation is easy. Recognising the same person tomorrow is the real problem, and it needs a few seconds of clear speech.
- A name said out loud can't be trusted on its own. In our tests, "This is Maya" was said by someone else, so a name should only be accepted when the voice matches.
- Asking for consent out loud fits the moment better than a form.
Built With
- deepagram
- mentra
- python-package-index
- speakers
- speechbrain
- voiceprints)
- xr-voice-profiles-(python