Human Sound

logo
UI
UI_2

Inspiration

When thinking about the theme, Tech for Tomorrow, AI is an obvious development in technology that has provided numerous benefits. However, it is also incredibly important to take a look at how such advancements are used in criminal fields and how we can defend ourselves against them. Crimes like voice phishing are becoming increasingly relevant as AI makes it easier to impersonate people with synthetic voices similar to the real thing.

What it does

Human Sound checks whether or not speech is real or AI-generated. You give it an audio file (WAV, M4A, MP3), and it returns a verdict of real, fake, or inconclusive, along with a confidence score. Ambiguous results are flagged as inconclusive instead of forcing a binary judgment, because a false accusation can be worse than an honest "unsure".

How we built it

It was developed by using Python and PyTorch to build and train a small classifier based on the Fake-or-Real dataset, on top of Meta's wav2vec2, a speech model pretrained on unlabeled speech. wav2vec2 is frozen, meaning its weights and biases are locked, so we have it work as an extractor that turns audio into embeddings (768 num arrays). The classifier analyzes the patterns in the numbers and outputs a judgment.

Challenges we ran into

Hardware limitations: (no GPU, non-AI-supported CPUs) Instead of our original idea of training a spectrogram CNN or tuning wav2vec2, we chose to train a lightweight classifier on cached embeddings.
Accuracy: Our first model got 99% on validation but around 67-69% on testing.
Dataset: We also used the for-2sec sub-dataset of Fake-or-Real instead of the general one, limiting the training data to 2-second-long clips of audio. Overall, it is faster, but it definitely caps our accuracy.

Accomplishments that we're proud of

Improving Accuracy: Our first run got us hanging around 67-69% accuracy and a 1.5-2 test loss. By selecting which layer of the wav2vec2 to run it on, and averaging it, we managed to achieve around 80%+ accuracy and ~0.4 test loss.
Optimization: We designed the project around speed and efficiency based on our resource limitations.

What we learned

The last layer of a model isn't always the best. It turned out low-level audio details worked out better for detecting generated speech.
Transfer learning is very useful. Instead of training a model from scratch, borrowing wav2vec2 very much sped up the process.