NORA

• 

  Github repository

• 

  ElevenLabs API requests

• 

  ElevenLabs API success rate

• 

  ElevenLabs API latency

• 

  Interface

• 

  Interface

💡 Inspiration

Voice cloning and conversational AI agents have gotten good enough to hold a full phone conversation as a stand-in for a real person — including calling a bank's customer service line. That's a serious opening for fraud and social engineering, and it's also exactly the scenario we were handed: given a recorded call between a caller and a bank's AI agent, decide whether the caller on the other end is a real person or an autonomous system doing the calling. We wanted to see how far a small, disciplined feature set could go against that question, without leaning on any signal that would quietly stop working the moment the voice on the other end was one we'd never heard before.

🛡️ What It Does

The system exposes a single endpoint, POST /detect, that takes a stereo 8kHz recording (caller on one channel, agent on the other) and returns a verdict — is_synthetic plus a calibrated confidence score. Under the hood, it's a logistic regression classifier reading two independent kinds of evidence: how the conversation behaves (who interrupts whom, how consistently the caller responds, how much of the call each side talks) and how the caller's voice sounds (energy stability, spectral flatness). Combined, those signals reach 95.8% accuracy and a 0.967 AUC on our validation set — and because train and validation never share a caller's voice, that number reflects generalizing to unheard voices, not memorizing familiar ones.

🛠️ How We Built It

The backbone is conversational and acoustic: response latency percentiles, how often and how persistently each side talks over the other, speech ratio, and turn counts — all derived from our own WebRTC VAD rather than the gold-standard turn annotations that came with the training set, because those annotations don't exist at judging time. We deliberately calibrated our VAD against the gold labels offline and then threw the gold labels away for anything the model would actually see live, so the accuracy we measured is the accuracy we'd actually get graded on.

On top of that, we added two acoustic features — energy stability and spectral flatness — that a teammate validated with real rigor: features were frozen before measuring AUC, checked against a bootstrap confidence interval, and explicitly tested for whether they were secretly just proxies for how much the caller talked rather than genuine voice-quality signals. Only the two that passed every check made it into the final model.

We also pushed further, using ElevenLabs Scribe to transcribe both sides of every call and layering in text-based signals (filler words, lexical diversity, speaking-rate variability), and started building a Gemini-based semantic check to catch a caller confidently "confirming" something the agent had invented, or misremembering information it was asked to repeat back. The transcription layer worked well; the simple text heuristics didn't move the needle enough over the conversational + acoustic model to justify the added latency and cost, so we measured that honestly and kept the leaner model as the one we ship.

🚧 Challenges We Faced

The one that shaped every other decision: the gold conversation-turn timestamps we were given for training are never available to the live endpoint, only raw audio is. It would have been easy to build a model that looks great offline and quietly falls apart the moment it has to segment audio itself — so a lot of the actual engineering effort went into making sure training and serving used exactly the same segmentation path.

Past that, we spent real time on: confirming our acoustic features weren't just re-encoding caller talkativeness, being honest with ourselves about a validation set small enough (71 calls) that a one- or two-call swing looks like a meaningful accuracy jump when it isn't, and a fair amount of environment friction — a C-compiler dependency that doesn't exist by default on Windows, PowerShell's different command syntax tripping up Unix-style examples, and an API provider's anti-abuse system flagging a legitimate batch job as suspicious mid-run.

📚 What We Learned

Behavioral signals — how a conversation unfolds in time — hold up better against unseen voices than raw acoustic ones, precisely because they don't depend on any particular voice at all. Pre-registering features and checking them against a plausible confound before trusting them catches false leads that a bare AUC number would have let through. And a signal that sounds intuitively right (people who are faking it should use fewer filler words, right?) still has to earn its place with an actual number — ours didn't clear that bar, so it stayed out.

Built With

• css
• elevenlabs
• fastapi
• html
• javascript
• python
• uvicorn