Inspiration
Voice cloning technology has gotten good enough to fool a human ear in a phone call — and banks are one of the first places that matters. A call center authenticating a customer by voice is exactly the kind of high-trust, high-stakes interaction that synthetic voices are starting to slip through. We wanted to see whether we could build something that catches that, using nothing but the kind of stereo call recording a real bank call center would already have: the caller on one channel, the automated agent on the other.
How we built it
We started from the assumption that no single detection signal would be enough on its own, so we designed the system around three independent angles:
- Acoustic detection — spotting the artifacts a text-to-speech vocoder leaves behind, using pretrained anti-spoofing architectures (AASIST, RawNet2) alongside hand-crafted spectral/temporal features.
- Conversational behavior — the things a cloned voice doesn't naturally get right even when it sounds convincing: how long a caller waits before responding, whether they interrupt or talk over the agent, the rhythm of turn-taking across a real conversation.
- Semantic analysis — what's actually being said, and whether it reads like spontaneous human speech or something scripted.
Each of these became close to its own model, with the plan to fuse their outputs into a single verdict rather than relying on any one of them alone.
Challenges we ran into
The dataset was small — around 300 labeled calls — which shaped almost every decision we made. Our first serious attempt (fine-tuning AASIST and RawNet2, then stacking XGBoost on top) looked strong under standard cross-validation, but failed badly on anything outside its training scope, defaulting to "human" on new audio.
Digging into why became the real turning point of the project. We ran a leave-one-cluster-out validation: since we didn't have explicit labels for which TTS engine generated each synthetic call, we clustered the synthetic training samples unsupervised, as a proxy for "engine family," and repeatedly held one whole cluster out to see how the model handled a "new" voice generator it had never seen. Most clusters generalized fine — but one cluster failed hard, misclassifying 30 of 33 held-out calls as human.
We dug further and found that this cluster's acoustic embeddings sat closer to the human centroid than any other synthetic cluster in our feature space, and the model was scoring it confidently human rather than borderline. That ruled out a calibration fix — it was a genuine gap in what our acoustic features could see, most likely because that particular TTS engine produces a more naturally-sounding voice than the others in our dataset.
That sent us back to listening to the actual audio by ear, and it paid off: the synthetic callers in that cluster consistently paused unusually long before responding and frequently talked over the agent instead of yielding the floor — cues that live entirely outside acoustic feature space. We turned those observations into concrete conversational-timing features and added them to the model, which measurably improved detection on exactly the cluster that had been failing.
What we learned
The biggest lesson was that high accuracy under a naive random split can hide a model that has memorized narrow characteristics of its training data rather than learned something general — and that the fix isn't always "try a different classifier." Sometimes the model is fine and the features are the problem. We also learned that listening to the raw data by hand, after the numbers pointed us at where to look, surfaced signal that no amount of re-tuning existing features would have found on its own.
What's next
We're integrating a teammate's semantic-analysis model with our acoustic-and-behavioral model through score-level fusion, so that when one signal is blind to a particular kind of synthetic voice, the others can still catch it.
Log in or sign up for Devpost to join the conversation.