UnderTow

Inspiration

Every bank security layer checks who is paying. None of them check who else is talking. Phone scams work because the real customer willingly types in the real code and every check passes. Americans 60 and older lost $7.75 billion to fraud in 2025, and phone calls produced the highest per-victim losses of any contact method. We wanted to close the air gap between the phone and the bank.

What it does

Undertow is an on-device acoustic sensor that sits between a phone call and a bank session and watches for three attacks:

Spoken secret. The bank sends a one-time code. If that code is heard spoken aloud while another voice is active, Undertow burns it on the server before the scammer can type it in. The attacker gets “code invalid.” The user gets a calm notice and a fresh code.

Dictated payee. A new payee account number that was spoken by an unknown voice before the user typed it doesn’t go straight through. It lands in the Tide Pool, a locked Nessie sub-account, where a pre-chosen trusted contact can approve or block the release.

Instructed outflow. A large cash withdrawal or a purchase at a crypto kiosk or gift-card merchant, made while an unknown voice is active, also holds in the Tide Pool. This catches the cash leg of kiosk and gold-courier scams.

The trusted contact has no access to balances and no ability to move money unprompted, which mirrors FINRA Rule 2165 and Rule 4512’s model for protecting vulnerable adults.

How we built it

Sensor daemon (Python): raw microphone input, Silero VAD, Vosk digit ASR with a restricted grammar, SpeechBrain ECAPA speaker verification. No audio or transcript leaves the device. The sensor emits HMAC hashes of digit windows over a localhost WebSocket so the backend can match without ever seeing the digits.

Backend (FastAPI + PostgreSQL): a pure rules engine that evaluates sensor events against armed targets, OTPs issued and payee entries typed, and writes every decision to a hash-chained append-only evidence log.

Nessie (Capital One API): real account balances, real transfers, real Tide Pool holds. The outbox pattern ensures every Nessie call is idempotent and reconcilable on startup.

Frontend (React + Vite): the /stage view puts Maya’s bank, the Sonar instrument and the attacker console on one screen for the live demo. Sonar shows the live Taint Thread, a visible line from the moment a digit was spoken to the moment it appeared in a payee field. Atkinson Hyperlegible Next throughout, built for low-vision readers.

Eval harness: synthetic scam calls rendered with Piper TTS, passed through a telephone band-pass and room noise at multiple SNR levels, run through the real sensor in file mode and measured for detection rate, false holds and latency.

Challenges we ran into

Browser echo cancellation erases the scammer’s voice off the speakers before the mic picks it up. The sensor had to run as a native process reading the raw mic, not through getUserMedia. Getting this right for a shared-screen Gather demo needed careful audio routing and a Replay Mode fallback.

Fuzzy digit matching had to be tolerant of spoken variations (“oh” for zero, “double five”, chunked reading, self-corrections) without producing false matches. We built a symmetric-delete neighborhood hash scheme that proves false-match probability mathematically per OTP length, and committed the threat model.

Speaker verification on short utterances under noisy telephone-band conditions needed real calibration, and we show the EER curve honestly on the /lab page rather than hiding it.

Accomplishments that we’re proud of

The live demo works: a judge plays the scammer, reads the code aloud, types it in and gets rejected on screen. The attack fails in real time, in front of them, before the scammer finishes typing.

No raw audio or transcript ever leaves the device. The only thing the backend sees is hashes.

Every money movement has a Nessie transaction ID and an entry in the hash-chained evidence log. make verify-chain checks it.

The /lab page shows real eval numbers from real sensor runs, with honest false-positive rates and an explicit “what Undertow can’t catch” panel.

What we learned

The hardest part of building sensor-based financial protection isn’t the ML, it’s the policy layer. Getting the asymmetric cost right (a false burn costs one fresh code, a false hold costs a reversible delay, only a true positive stops a scammer) was the design decision that made everything else sensible.

Acoustic fuzzy matching at telephone quality is genuinely hard. The normalization table for spoken numbers is longer than we expected.

Designing for someone who is actively frightened is different from designing for someone who is calm. Every intervention had to calm first and explain second.

What’s next for Undertow

A native mobile SDK so any banking app can wire in the sensor without routing audio through a server. Enrolling the trusted contact’s voice as a known-safe voice, so a daughter calling her mother doesn’t trigger R2. Extending R3 to detect gift-card purchase patterns in transaction history, before a call even starts. Working with a credit union or community bank to pilot on real accounts with real older adults.

Built With

• capital-one-nessie
• fastapi
• pipertts
• postgresql
• python
• react
• silero-vad
• speechbrain
• typescript
• vite
• vosk