Aqua | Devpost

Inspiration

With the rise of synthetic voice systems capable of imitating human speech almost perfectly, bank call centers have become an easy target for automated fraud at scale: a single attacker can run hundreds of synthetic "callers" in parallel. We were inspired to build Aqua, a detector that runs at the moment of the call, without relying on recognizing a specific voice, but instead on patterns that any synthesis system leaves behind — no matter how good the TTS engine is.

What it does

Aqua takes the audio of a bank customer-service call (stereo WAV, channel 0 = caller, channel 1 = agent) and responds, via a real production API, whether the caller is a real person or a synthetic voice — with a calibrated confidence score, not just a boolean. The whole pipeline runs deployed and reachable over HTTP for live evaluation.

How we built it

We designed 50 features across three fronts, betting on signals that generalize to speakers never seen before (the evaluation split is speaker-disjoint):

Acoustic — jitter/shimmer, spectral flatness, MFCCs, short breaths within turns, background noise floor between turns. Conversational timing — response latency after the agent speaks, genuine overlapping speech (backchannels), and how the caller reacts when interrupted. An pipeline tends to have more uniform latencies and reacts "too cleanly" to interruptions. Semantics via ASR — genuine confusion from a human when asked something nonsensical, versus fluent confabulation from an LLM.

Challenges we ran into

Our biggest challenge was the size of the dataset: only 282 training calls (113 human / 169 synthetic) and 71 validation calls. With so little data, the real risk wasn't "Aqua predicting wrong" but memorizing specific voices instead of learning generalizable patterns — and the challenge's own design punishes exactly that, evaluating on speakers the model never saw. We fought this with:

Feature engineering deliberately agnostic to the speaker's identity (spectral texture and timing, not vocal timbre). A strict speaker-disjoint split to measure ourselves honestly from day one. Explicit model calibration, so reported confidence reflected real uncertainty instead of false confidence from too few examples.

With that, Aqua reached 0.958 accuracy and 0.989 AUC on validation — a result that surprised us given how limited the dataset was, and confirms that the strongest signal (spectral_flatness, ~79% of the model's importance) does generalize well even with few examples.

That with small datasets, discipline in feature design and validation matters more than model complexity. We also learned, in the final stretch, to solve a very different problem: taking Aqua from a desktop prototype to a real cloud-deployed endpoint, with error handling that never lets an exception escape as an HTTP failure (because in this challenge, a timeout counts as an incorrect response, just like a wrong prediction).