ProofVoice

Three seconds of speech can clone a voice. ProofVoice detects AI-generated speech in files and live streams, shows the evidence behind every verdict, and says when it isn't sure.

Comment

Inspiration

Three seconds of someone's voice is now enough to clone it, and cloned voices are already being used in "grandparent" scams, fake-CEO wire fraud and bank-alert phishing. Recognizing a voice used to be one of the oldest trust signals between people. I wanted to answer one question: how quickly can we know that the voice we're hearing is synthetic, and what evidence supports that? Generation is becoming real-time, so verification has to become real-time too.

What it does

ProofVoice is an evidence layer for synthetic speech. One forensic core powers four surfaces:

  • Forensic Lab: upload any audio (or click one of 11 held-out demo clips) and get a verdict ("98% synthetic"), a per-window trust timeline you can listen along to, suspicious regions, evidence cards from each forensic technique, and a branch log explaining which techniques ran and why.
  • Live Trust: stream from your microphone (or stream a clip digitally) and watch a rolling probability update every 0.5 s, including time-to-confidence.
  • Grok Red Team: Grok Voice speaks a scam line, and the exact same audio bytes stream through the same detector in real time while you listen.
  • NSA HEARSAY batch mode: a Docker container that scores the 1,671-clip held-out set offline and writes the prediction TSV.

Every result carries explicit uncertainty: likely human, likely synthetic, inconclusive (route it to a human analyst) or insufficient evidence.

How I built it

  • Primary detector: a fine-tuned XLS-R (wav2vec 2.0) self-supervised front-end with attentive statistics pooling, calibrated with Platt scaling. Models were selected strictly on the official organizer metric, ASVspoof 5 minDCF (false alarms cost 4× misses), wrapped around the organizer's scorer with a bit-level parity test.
  • Data: organizer DiffSSD (10 generators) + LJ Speech, plus public LibriSpeech as extra real speech, because 6 of the 10 generators clone LibriSpeech speakers. Otherwise a detector learns "not this one voice ⇒ fake". Splits are group-disjoint by speaker, chapter and sentence.
  • Forensic branches from distinct families: container/metadata, spectral (LFCC, flatness, band limits), prosody (F0, jitter/shimmer, pauses), compression/transcoding, splice/discontinuity detection, and signal quality. A deterministic router runs the extra branches only when context calls for it (e.g. near-threshold decisions, lossy input) and logs the reason.
  • Backend: FastAPI + WebSockets, PyTorch on CUDA (23 ms per 4 s clip on a laptop RTX 4060, ~345 ms on CPU).
  • Frontend: Next.js + TypeScript with Web Audio (AudioWorklet) for live capture, custom SVG charts, and accessible, reduced-motion-aware UI.

Results (validation, lower minDCF is better)

System minDCF EER AUC
Organizer AASIST checkpoint, zero-shot 1.000 44% 0.638
Hand-crafted forensic features alone 0.506 — —
ProofVoice XLS-R (selected) 0.066 1.5% 0.999
  • 6/6 live Grok Voice clips flagged, even though Grok was never seen in training.
  • Robustness measured under MP3, Opus, AAC, μ-law telephone, noise, gain and clipping.

Challenges I ran into

  • Shortcut learning. With only one real speaker in the organizer data, early models learned speaker identity instead of synthesis artifacts. Adding real LibriSpeech speakers and group-disjoint splits fixed the headline number, but a per-speaker audit still shows one cloned speaker is much harder than another (subgroup minDCF 0.24). I report it instead of hiding it; one of the demo clips is that known miss.
  • More augmentation isn't automatically better. A v4 model trained with real codec transforms and harder generators scored worse (0.080) and was not shipped. Fusing the hand-crafted branches into the score didn't beat the neural detector out-of-fold, so they stay as explainable evidence, not score inputs.
  • Replay is a different threat. Playing an AI voice from a phone speaker into a laptop mic adds room echo and speaker distortion the model never saw, and results become unreliable. I measured this, turned off the browser's call-processing filters for forensic capture, added a digital "stream a clip" path, and documented the limitation.

Accomplishments that I'm proud of

  • Going from 1.000 → 0.066 minDCF against the organizer baseline, measured with the organizer's own scorer.
  • An interface that explains instead of just scoring: timeline, evidence cards, a branch log, ground truth on demo clips, and an honest "inconclusive" state.
  • A real-time loop where a frontier voice model attacks the detector live, byte for byte.

What I learned

Pooled metrics hide failure modes; subgroup and channel audits matter more than one headline number. Calibration and a well-designed "I don't know" state are as important as accuracy for anything an analyst acts on.

What's next

  • Real recordings of the cloned speakers for identity-balanced training.
  • Replay/room-acoustics augmentation for over-the-air calls.
  • Leave-one-generator-out evaluation.
  • A browser extension or phone-call integration.

Built With

Share this project:

Updates

Submission history