Inspiration

In 1953, Colin Cherry studied air-traffic controllers hearing several pilots at once through a single loudspeaker and named the puzzle the "cocktail party problem": how does a listener in a noisy room follow one voice and discard the rest? His experiments showed that people can attend to one stream and remember almost nothing of the other, yet still notice their own name from across the room. Most of us do this without thinking.

Everyone loses that filter sometimes. A restaurant where you lean in and nod at words you did not catch. A trade-show floor, a warehouse, a factory line, an open office with three meetings in earshot. A parent trying to follow one child in a playground of forty. For people with hearing loss or auditory processing differences, every room is that room. And in search and rescue, the stakes flip: the one sound that matters is a voice under rubble or across a field, buried in wind, engines, and shouting. We wanted a device that cancels out everyday noise and does the selecting for you: show who or what is making sound and where, let you choose it, and bring that one source forward. Cocktail is our attempt to give the cocktail party effect to anyone who needs it, in any room.

What it does

Cocktail turns a Meta Quest 3S into assistive ears. The headset streams its microphones, head pose, and camera to a laptop hub. The hub splits the room's sound into separate sources, labels each one (speech, music, other), works out its direction from how levels shift as your head turns, and tracks each one with a stable identity. In passthrough, you see a picker of who is talking, point with your hand, and pinch to choose. The chosen source gets a rectangle and label; if they are out of view, an arrow at the edge of your vision points to them. The hub sends that voice back to the headset, with the rest of the room turned down.

How we built it

Headset (Unity, Meta XR SDK): raw mic capture at 48 kHz, head pose and camera streaming over WebSockets, a hand-tracked HUD (picker, pointing, pinch selection, edge arrows), and a developer mode with fake data, a test panel, and an on-device event log.
Hub (Python, PyTorch): MossFormer2 for live voice separation, an AudioSet classifier for labels, a head-motion-based direction estimator, a tracker that keeps one identity per sound through separator slot swaps and pauses, and a two-layer audio return (chosen voice plus room).
Process: a frozen wire protocol between the two halves, a mock headset and session replayer so we could test without the device, simulated scoring for separation and direction, and one PR per feature with hardware, mocked, or untested evidence labels on every claim.

Challenges we ran into

The microphones. The Quest 3S delivers four channels, two of which are empty, and the two live ones differ by 15 to 32 dB with weak coherence. Our direction model assumed two matched ear-like mics, so it confidently reported the wrong side. We measured the real behaviour on recordings and built opt-in calibration and gates that withhold a bearing rather than show a wrong one.
Quiet voices. Raw, unprocessed capture puts speech at -56 to -77 dBFS, below the gates that decide a source is "speaking." It took a live session with both halves instrumented to find the two thresholds that were silently dropping every talker.
A stack overflow on the headset. When the real hub sent an empty source list, two init methods called each other forever and the app died with a segfault. Fake data had hidden it all week.
Logistics. One headset shared by four people, dropped USB debug links after every reinstall, the room boundary pausing the app mid-test, and a build signed with a different key refusing to install.

Accomplishments that we're proud of

A live loop end to end: real audio from the headset, separated and labeled on the hub, located, tracked, and drawn on the HUD, selectable with a pinch.
A HUD that stays honest: nothing is chosen automatically, a source with no direction gets a dot rather than a guess, and bad data from the hub (NaN, duplicates, floods) is cleaned at the door.
Measurement over assumption: every "it works" in our docs says whether it was hardware-tested, mocked, or simulated, and the mic facts in our architecture notes come from recordings, not spec sheets.
Four people, one weekend, 68 pull requests, no merges to main without review.

What we learned

Test on the real device early. Fake data let us build fast, and it also hid the crash and the quiet-mic problem until the last hours.
Direction from head motion works, but only with a mic model that matches the hardware. Calibrate per device before trusting any number.
Every threshold is a hypothesis. Two separate -55 dB gates, each reasonable alone, together meant "never."
Small, labeled, reviewable changes kept four parallel tracks from colliding, even under deadline pressure.