ChallengeAltur_ByCrowTeam

Inicio
Espectro del sonido
Detección
Descripción técnica
Plus

What Inspired This Project The challenge was compelling: distinguish between human callers and AI-generated voices in phone conversations with a bank's customer service system. This isn't just a detection problem—it's about understanding the subtle acoustic and conversational patterns that betray synthetic speech. When we read about synthetic voice quality reaching near-human levels, the academic question became personal: Can we catch the difference?

The dataset—stereo calls in Mexican Spanish with both the caller (channel 0) and agent (channel 1)—provided a unique angle: we don't just listen to isolation; we analyze interaction dynamics. How does the caller respond to the agent? What's the latency? Does the conversation flow naturally?

What We Learned Synthetic voices are eerily consistent. While humans vary pitch, add hesitations, and stumble over words, AI often produces:

Lower shimmer (frame-to-frame amplitude stability) More uniform spectral entropy (energy spread across frequencies) Hyper-regular response latency (bots pause predictably) Conversation is a fingerprint. Raw audio features alone aren't enough. The turn-taking pattern—silence gaps, speech ratio, overlap—reveals behavior. Humans interrupt, speak over each other, get confused. Bots wait, calculate, and respond with clockwork precision.

Ensemble beats any single model. We tested RandomForest, GradientBoosting, LightGBM, and SVC individually. But weighting them together (soft voting: RF×3, LGBM×2, GB×2, SVC×1) and calibrating probabilities gave us the edge needed for balanced accuracy on unseen speakers and voices.

Speed matters under pressure. At 30 seconds per call, every millisecond counts. Switching from pYIN to YIN for pitch extraction cut inference from ~30s to ~2–3s without sacrificing accuracy—a 10x speedup.

How We Built It Our pipeline follows a clear train-infer separation:

Phase 1: Feature Engineering (features.py) We extract approximately 50 dimensions per call:

Spectral: RMS, zero-crossing rate, flatness, centroid, bandwidth, rolloff, contrast Harmonic: 20 MFCCs plus delta-MFCCs (capturing temporal dynamics), chroma, pitch (F0) Prosody: Shimmer (amplitude stability), jitter (pitch jitter), spectral entropy Conversation: Turn counts, speech ratio, response latency variance, silence metrics Interaction: Caller vs. agent differences (energy, pitch, entropy, timbre) The total feature count combines Caller features, Agent features, Differential features, and Turn Dynamics.

Phase 2: Model Training (train_model.py) Built a VotingClassifier ensemble with soft voting. The probability of synthetic is calculated as:

P(synthetic) = (3 × P_RF + 2 × P_LGBM + 2 × P_GB + P_SVC) / 8

RandomForest (400 trees): Robust for tabular data, captures non-linearities LightGBM (300 iterations): Fast, handles imbalance natively GradientBoosting (300 iterations): Sequential refinement SVC (calibrated, isotonic): Boundary definition and probability calibration Validation strategy: 5-fold StratifiedKFold on train set, threshold optimization on val set (maximizing F1 over 181 grid points).

Phase 3: Inference and Deployment (main.py) FastAPI endpoint (POST /detect) respecting the judge's contract LRU cache by MD5 hash (avoid reprocessing identical audio) Fast mode (YIN plus 1024-frame STFT) by default for less than 3s latency Threshold from training bundled with model; confidence scores calibrated for ROC-AUC Tech Stack Audio: librosa (MFCCs, pitch, spectral features), soundfile (WAV I/O) ML: scikit-learn (ensemble, calibration, cross-validation), LightGBM API: FastAPI and Uvicorn Dashboard: Streamlit (real-time analysis, error diagnostics) Challenges We Faced Distribution Shift: Train does not equal Inference The dataset uses pYIN (slow, high-quality) for ground-truth pitch during feature extraction. But at evaluation time, we have 30 seconds per call. Solution: Built a unified fast-mode (YIN plus reduced frame length) across train and inference to eliminate distribution shift.

Class Imbalance and Calibration The dataset may not be 50/50 human/synthetic. Random predictions would fail. Solution: class_weight='balanced' in RF and LGBM, explicit isotonic calibration on SVC.

Silence Isn't Empty—It's Informative Long pauses between turns are different for humans (confusion, thinking) versus bots (processing). Solution: Added silence-per-minute, response latency variance, and turn density features.

Stereo Channel Choreography The agent's responses (channel 1) are the context for what the caller does (channel 0). Naive single-channel models miss the interaction. Solution: Extracted both channels, then computed differential and ratio features.

30-Second Timeout: Every Optimization Counts pYIN alone took 30 seconds. Solution: Multi-faceted approach—YIN for inference, smaller frame size (1024 versus 2048), ThreadPoolExecutor for parallel I/O, LRU cache to skip duplicates.

Unseen Speakers and Voices at Evaluation Time Speaker-disjoint splits mean generalization is hard. Solution: Feature engineering that captures acoustic patterns (shimmer, entropy, pitch distribution percentiles) rather than speaker identity, plus ensemble voting to reduce overfitting to any single decision boundary.

Metrics and Results On the validation set:

ROC-AUC: approximately 0.94 (cross-validated 5-fold) F1 (optimized threshold): approximately 0.89 Balanced Accuracy: Primary metric for unseen callers Latency: 2–3 seconds per call (well under 30s limit) The error analysis (error_analysis.py) breaks down performance by call duration, acoustic quality, and conversation density to identify blind spots.

Key Insight The strongest signal isn't any single feature—it's the ensemble of micro-patterns: A synthetic voice rarely ages (shimmer drops), its pitch never wavers (low jitter), its entropy never spikes (uniform spectrum), and its response timing is too perfect (zero latency variance). Humans are beautifully erratic; algorithms are elegantly predictable.

This project synthesizes classical signal processing with modern ML best practices—and proves that even in the age of realistic TTS, human and artificial voices still dance to different rhythms.