Inspiration Our first detector hit 98.6% on the validation split — and then confidently called a Gemini voice "human" when played through a laptop mic. That failure was the real starting point: it forced us to ask what the model had actually learned.
What it does Voice Shield / Calliope analyzes a call in near real time (~100× faster than real time, no GPU) and returns whether the caller is synthetic, with a calibrated confidence score.
How we built it We found the model wasn't detecting voices: it was detecting recording channels. The human and synthetic calls in our data came through different audio paths, so a single spectral feature alone could separate the classes with 0.967 AUC. We proved this with a counterfactual: adding only room noise and reverb to synthetic calls (no change to the voice itself) flipped confident "synthetic" verdicts to "human."
To fix it, we:
Applied five channel simulations (telephony, mobile, mic-in-room, VoIP, handset replay) equally to both classes, so channel cues carry no label information. Replaced raw cepstral features with post-normalization deltas that describe voice movement, not the recording path. Rebuilt our timing features around consistency rather than speed, since fast speech-to-speech models (like Gemini Live) actually respond quicker than humans. Wrote an automatic check that flags and drops features that identify our own data generator rather than the actual class. Re-ran evaluation honestly: a grouped, stratified train/test split with no leakage back into the reported numbers.
Results 99.3% accuracy on a held-out test set never touched during training, with meaningful gains on out-of-distribution channels (e.g. mic-in-room accuracy rose from 92.4% to 98.5%).
Challenges we faced The hardest part wasn't building a classifier: it was learning to distrust our own metrics. A 100% validation score turned out to be evidence of a leak, not success. Diagnosing why the model was right for the wrong reasons, then designing experiments that isolated channel from voice, took far longer than training the model itself.
What we learned A high accuracy number is only as trustworthy as the evaluation behind it. The most valuable work in this project was adversarial: trying to break our own system before someone else did.
