Inspiration

Voice cloning now needs only a few seconds of audio. Scammers use it to call people as a grandchild in trouble, or as their bank, asking for money or a code, fast. Caller ID checks the number. Nobody checks what is being said.

Most call-protection tools ask you to trust where your call audio goes. We wanted scam protection that runs entirely on your own devices, and whose privacy you can check yourself instead of trusting a policy.

What it does

Tell listens to the caller's side of a call and gives two separate answers, live, about every 3 seconds:

  • Is the voice human or AI?
  • Is the conversation a scam pattern? Impersonation, urgency, secrecy, or requests for gift cards, wire transfers or one-time codes.

A scammer can be a real person, and a harmless appointment reminder can be an AI that says it is one, so those land in different places on a 2×2 card, with the reasons in plain language and an honest "how sure". A single scam sign never raises an alert: an alert needs agreeing signs over consecutive updates, and a red alert stays up until you dismiss it. When the call ends, Tell scores the last few seconds once more before it stops.

Everything runs on your own computer: no servers, no account. Caller audio is analysed in memory and wiped when the call ends. Nothing is stored by default; an opt-in alert history is encrypted (AES-256-GCM) with the key in the OS keychain.

How we built it

  • Engine: Python + FastAPI on loopback only (127.0.0.1), streaming over a local WebSocket.
  • Voice check: Pellav2, an open synthetic-voice detector, on Apple Silicon (MPS), scoring 4-second windows; plus an open wav2vec2 anti-spoof model and interpretable acoustic tells (pause rhythm, pitch movement, noise floor, phone-band audio).
  • Words: faster-whisper transcribes locally; pattern rules flag scam intent and AI self-disclosure; a small local LLM (qwen2.5:3b via Ollama) reads the transcript for scam patterns. The model interprets; code decides.
  • Live policy: a stabiliser with warm-up, hysteresis and alert latching, so the card doesn't flicker.
  • Where it listens: on a Mac, a Core Audio tap hears the caller's side of FaceTime, Zoom, WhatsApp and Teams and starts on its own when a call is answered. iPhone calls work through the Mac (Calls on Other Devices), because iOS gives apps no call audio. An Android app (Capacitor) streams to the engine over USB; a web page and a menu-bar app show the verdict.
  • Privacy you can check: caller audio is blocked from leaving by code (a test scans the analysis path); the engine reports the OS's own count of its outbound connections since start (0 in our demo, with Wi-Fi off); a per-install API token protects every data route (401 without it); models load offline.
  • Red team: we built our own attacker, voice-talk, a scam bot that talks back live in voices cloned with consent (local Qwen3-TTS + a local LLM), with a staged fictional script and a TEST ATTACKER label. We also cloned our own voices in Gradium to generate attack calls, and Tell speaks its verdict in a Gradium voice, rendered ahead of time, so nothing is sent during a call. Gradium only ever received our own text.

Challenges we ran into

  • Getting call audio. iOS gives third-party apps no call audio. On our Samsung test phone, Android silences the microphone for other apps during a real call, even on speaker. So live calls run through the Mac, which works.
  • New clones beat the voice check. With the Qwen3-TTS engine held out of tuning, the shipped voice check caught 1 of 10 Qwen clones. That is our blind spot, and it is why the words are checked separately: a scam read in any voice can still be caught by what it asks for.
  • Phone audio is bad audio. On a very lossy simulated channel (6 kbps Opus with packet loss), one voice model misfired on 7 of 12 real human memos, though none crossed our alert line.
  • Three models on one laptop. Whisper, Pellav2 and a 3B LLM run together; the LLM answers in about a second, and the card updates about every 3 seconds.

Accomplishments that we're proud of

  • The whole pipeline runs with Wi-Fi off, and we show the OS's own connection count during the call.
  • We attacked our own app with a cloned voice of a teammate, and we publish what it misses instead of hiding it.
  • It flags a scam script read in a real human voice, so protection doesn't depend on the voice detector being right.
  • We trained models that do better on voices they never heard (a learned layer on Pellav2: held-out Qwen catch 0.55 → 0.90 on clean audio in our tests). We kept them out of the live engine until they hold up on real calls.

What we learned

  • Synthetic-voice detection degrades on phone-quality and replayed audio; published numbers don't transfer to calls.
  • Each cloning engine leaves its own fingerprint, so a detector has to be tested on engines it never saw.
  • A privacy claim you can demo is far more convincing than one you have to promise.
  • Two honest answers beat one blended score: users need to know whether the voice or the words raised the alarm.

What's next for Tell

  • Real phone-call captures and more consented voices, to fix the replay blind spot.
  • Ship the learned voice layer once it passes on real calls.
  • A family plan: an adult child protects a parent, with optional alerts to a trusted family member, with consent.
  • Smaller models so more of it runs on the phone itself.

Measured on small sets; counts, not rates (pm/AUDITS.md, docs/HEAD-FINETUNE.md, docs/LEARNED-TELLS.md). Tell can be wrong: if a call asks for money or a code, hang up and call back on a number you already have.

Built With

Share this project:

Updates

Submission history