OneVoice

Inspiration

At a campus research fair, Arnav couldn't hear the presenter properly. So he started nodding along because he did not want to seem rude. So an idea struck him . What if we made a product that intelligently focused on the noise of a person that we want to hear . What if there's no need of the social awkwardness of pretending to understand a person in a crowded room. Enter OneVoice . Instead of treating the whole room as one sound, we wanted to give the listener a choice. This person, this conversation. One tap, and the rest of the room gets quiet. That experience became our starting point on day one of VT Hacks. Instead of treating the whole room as one sound, we wanted to give the listener a choice: this person, this conversation. One tap, and the rest of the room gets quiet.OneVoice lets you choose who to hear. A small camera on a glasses frame watches the room. You tap a face on the screen, and OneVoice lifts that person's voice above everyone else's. That is happening live through wired earbuds. It is intelligent noise cancellation that you steer with a tap.

How we built it

Hardware. A Logitech C270 camera on a 3D printed glasses frame, a laptop, and wired earphones.

Listening. Face detection and tracking (MediaPipe and RetinaFace) find and follow every face. When you tap one, an audio-visual speech separation model (Dolphin, running on PyTorch with a CUDA GPU) uses that face to pull out the matching voice from the microphone mix.

*The app. *A Python backend runs the session and serves a plain JavaScript interface that only listens on the local machine. Captions use ElevenLabs realtime speech to text, and summaries and explanations use Gemini.

The story. Our landing page is a Three.js journey through an alpine world. As you scroll, the camera dives into the lens of the glasses and hands you straight into the app. The story site is hosted on Vercel, and the listening app runs on our demo laptop.

The team. Shubhankar handled the GPU machine and live demo. Raghav built captions and summaries. Arbaas built the 3D landing page. Arnav built the app interface and joined the pieces together.

What it does

A few extras help when you want them:

Live captions of the person you picked, off by default.
Recordings that get a short summary and key points when you stop.
"Show me what they mean" to explain a moment using the camera image and recent captions.
An experimental alarm mode that can bring room audio back if it hears an alarm. It is a safety feature. Because you don't really want the fire alarm's noise to be cancelled do you ?

No GPU? A preview mode lets anyone try the whole interface with three simulated people. No isolated audio plays in preview, and the app says so.

Challenges we ran into

Only GPU laptops can run the isolation. Most of our machines couldn't, so we built a preview mode with simulated people. That let us build and test the whole interface without a GPU.
Connecting a website to a local app safely. The story page has to open an app on your own laptop. Browsers blocked our first handoff, so we added a narrow check that only answers our own pages and kept the app locked to the local machine.
Four people, one branch. We hit real merge conflicts, and at one point conflict markers showed up in the live interface. After that we tested after every merge.
A heavy 3D scene. The main model is about 10 MB, so we built a loading bar that shows real progress instead of a spinner.
Privacy. Captions and summaries send data to ElevenLabs and Gemini. So captions start off, and the app tells you when a recording will be sent.

Accomplishments that we're proud of

Live noise isolation working on a real laptop with the Logitech camera and earphones.
A landing page that flows straight into the working app, with no page in between.
An interface that is honest about what it is doing: preview versus live, experimental versus ready.
An automated test suite for both the app and the website.
Shipping all of it in one weekend as a team of four.

What we learned

Seeing the speaker is a strong hint. The face tells the model exactly whose voice to follow.
Build the fake version first. Preview mode let four people work at once before the hardware was free.
Say what you have and have not measured. We show illustrations as illustrations, and we have not yet measured the before and after sound, so we say so.
Cutting features makes the product calmer. We removed spoken summaries and a settings panel to keep the app simple.

What's next for OneVoice

Move from tethered to worn: test the printed frame on more faces and try a smaller camera.
Measure the sound before and after isolation with proper metrics.
Do more processing on the device and less in the cloud.
Try wireless earphones and lower delay.
Make alarm awareness reliable enough to trus