VoiceLock

Inspiration

Some of the people closest to us are autistic. We've seen firsthand how overwhelming everday environments can become when many conversations happen at once. We want those close to us to have the independence to connect with others on their own terms. Environments aren't usually designed with their needs in mind, and barriers are created that others may not even see. We wanted to build something to bridge that gap.

What it does

The app lets the user focus on one person's voice, even when multiple people are talking at the same time. Hold your phone up, and the sound locks on to he person you want to talk to and plays it in your earbuds, while everything else goes quiet. Unlock other systems, this system is personalized and tailored for each user via AI.

How we built it

We built the mobile app using Expo, React Native, and TypeScript, with a Python and PyTorch audio-processing pipeline. A pretrained MossFormer2 model separates overlapping speech, while speaker embeddings match the separated voices against a two-second enrollment sample to lock on to the voice with louder volume.

Our team coordinated development through GitHub and Notion using multiple AI agents. We evaluated the system using synthetic speech mixtures, including equal-loudness conditions.

Challenges we ran into

The biggest challenge we had was an under-explored research domain, which is extracting a lower volume speaker from a higher volume environment. We tested many methods on our GPUs distilling a teacher model for fast accurate responses, but the results were unsatisfying, the student model was not able to learn enough. After that we tried fine tuning with audio clips of us but it still produced a weak model. We then tried a projection head onto the pretrained model, but it unexpectedly caused a lower accuracy. We learned a lot and we found new paths that can be explored more given longer time, but we concluded that the general model outperformed every other model in general use given the time constraints and hardware limitations.

Accomplishments that we're proud of

VoiceLock achieved 99.2% correct-speaker selection in clean, equal loudness synthetic tests using two-second enrollment. While it needed more training to work in more noisy environments, we are proud that we measured the results systematically and going through empirical methods with real research instead of just make it look good.

What we learned

We learned how to conduct meaningful research under intense time constraints, spending countless hours studying existing approaches, experimenting with model architectures, and running GPU-intensive training sessions. By tackling an underexplored problem, we experienced both promising breakthroughs and unexpected setbacks. Most importantly, we learned to adapt our approach, make informed decisions under pressure, and recognize that meaningful research is measured not only by results, but also by the knowledge gained along the way.

What's next for VoiceLock

We want to improve performance when the target speaker is quieter than surrounding voices, solve target-absence detection, and reduce audio latency. Our next steps include additional model training and testing in real environments. Ultimately, we want VoiceLock to become an accessible everyday tool that gives people greater independence and control over what they hear.