SafeKeep

Inspiration

Every phone scam is a heist. There's a crew, a target and a script, and the script barely changes: someone you trust is in trouble, it has to happen right now, don't tell anyone, and pay with gift cards or a wire. People who lose money to these calls aren't foolish. They're being worked by professionals who've run the same play a thousand times, usually against people who are home alone and polite enough to stay on the line.

We kept coming back to one idea: if the scammers are running a script, a program can learn the script, and step in before the money moves. So instead of planning a heist, we built the crew that stops one. The Anti Heist is our app, SafeKeep, a scam shield for credit union members that listens to a suspicious call, recognizes the con as it unfolds, and locks the vault before anything leaves the account.

What it does

Listens to the call (with consent). Put a suspicious call on speaker and tap Listen. SafeKeep first plays a recording notice out loud. That's legally required, and it's also a feature: a scammer who hangs up the moment they hear "this call is being monitored for fraud prevention" has already lost.
Spots the tell. Every line is checked against the four tactics scammers rely on (trust, emotion, urgency and isolation) by a fast rules engine and by Gemini, fused into a single live risk score. The background slowly turns from calm green to red as the risk climbs.
Predicts the next move. SafeKeep knows how each scam script unfolds (fake grandkid, fake bank, fake IRS, fake tech support), shows where the caller is on a heist board, and forecasts their next move. The board is a branching timeline: the main line is what actually happened, and every fork is a forecast we made, marked as called or missed.
Speaks up and gets help. When the risk crosses the line, SafeKeep warns out loud, vibrates, and rings a warning bell without hijacking the screen mid-call. One tap opens the warning sheet: call someone you trust, call the official number, cancel the payment, or (only with a press-and-hold) ignore it.
Joins the call as a third caller. On the protected line, the member merges SafeKeep's AI agent (built on ElevenLabs Agents) into the call. The agent says the recording notice into the call itself, stays silent while our backend analyses every turn, and speaks one short, scam-specific warning that both people hear if the call turns into a con. The member's app follows along live: same risk gauge, same heist board, same warning bell. Nothing is captured on the member's phone.
Locks the vault. A risky payment waits behind a cooling-off hold until someone from your crew confirms it's real. When a scam is stopped, you watch a 3D vault door seal shut.
Keeps evidence without keeping your life. No audio is ever saved. Text is only kept once a call crosses the alert threshold, and even then it's redacted, encrypted with a per-case key, and chained so it can't be quietly edited. After the call you get a plain-language report of what the caller was after, quoting their words and never yours.

How we built it

Frontend: React 19 + TypeScript on Vite, styled with Tailwind CSS 4, framed as a phone so it demos anywhere and installs as a web app.

Motion: GSAP timelines drive the cinematic moments: the steel vault doors that split down the middle on launch and the vault lockdown. Framer Motion handles the dock, lists and the rolling risk counter, and Lenis smooths scrolling.
3D and shaders: three.js / React Three Fiber for the vault door and the 3D heist board, plus WebGL shaders (via OGL and React Bits) for the risk-reactive background and the liquid-metal logo. We adapted React Bits components to share GPU resources and to behave inside a scaled phone frame.

Backend: FastAPI on Python 3.14, with a WebSocket that streams microphone audio (or typed text) into a call pipeline.

Listening: Gemini Live transcribes, and a rules engine plus Gemini Flash-Lite classify each line.
Scoring: a fusion layer turns both signals into one risk score, the blueprint engine tracks the scam script and checks predictions, and ElevenLabs voices the warnings (with the browser's built-in voice as backup).
Protected line: an ElevenLabs agent handles the phone side (answering, transcribing, speaking). Its "Custom LLM" points at our backend, which speaks the OpenAI chat-completions protocol over server-sent events: each turn, we analyse the new lines and answer with silence (the skip_turn tool) or, once, a fixed warning line. A signed post-call webhook (HMAC-SHA256) closes the case, and a WebSocket streams the live call to the member's app. During development an ngrok tunnel exposes the backend to ElevenLabs.
Data: Tiger Data (TimescaleDB) stores anonymized metrics and the evidence chain: hypertables, continuous aggregates, append-only triggers and retention policies.
Security: Fernet encryption with per-case keys (delete the key and the content is gone for good, while the chain still verifies), SHA-256 hash chaining, HMAC'd phone numbers and FRE 902(13)/(14) certification packets for self-authenticating evidence.

Testing: 282 backend tests plus an evaluation harness of 30 scripted calls (20 scams, 10 legitimate). With Gemini, Safe Keep flagged all 20 scams with zero false alarms on the 10 real calls, alerting after about two caller turns on average. Rules alone still catch 18 of 20, with zero false alarms.

Challenges we ran into

Phones won't let apps hear phone calls. Android hands third-party apps silence during a call, and iOS interrupts recording. We designed around it twice: speakerphone mode on a second device, and the protected line, where an AI agent joins the call as a third caller.
Getting a bot onto a real call. Our first plan was a Twilio bot, but Twilio's trial doesn't hand out phone numbers. ElevenLabs Agents (with the hackathon's credits) let us keep the phone side off our servers entirely, but its Custom LLM expects a chatbot. Teaching it to stay silent on purpose, speak exactly once, and never analyse its own recording notice took careful protocol work. Our network also blocked Cloudflare Tunnel's port, so we moved to ngrok.
Legal vs. useful. Two-party consent laws mean the recording notice can never be skipped. We turned it into part of the defense, and made sure we only count it as a "win" when the caller actually hung up after hearing it.
Privacy vs. evidence. We wanted a record that would hold up, without hoarding anyone's conversations. Threat-gated transcripts, redaction before encryption, crypto-shredding and a hash chain that still verifies after content is destroyed took several redesigns to get right.
A report that doesn't embarrass anyone. Gemini writes the post-call report, but every quote is checked against the stored transcript, the member's own words are never quoted, and blaming language sends the report to a safe template.
Too much WebGL. An early shader button effect wanted one WebGL context per button. Forty of them blew past the browser's limit and killed the 3D vault, so we rebuilt it around one shared context. The scaled phone frame also broke every canvas measurement until we switched to unscaled sizes.
Infrastructure surprises. Async PostgreSQL doesn't run on Windows' default event loop, and our networks (even a phone hotspot) blocked the database ports, so the app falls back to an in-memory store and keeps working.
Testing animations we couldn't see. A hidden browser pane doesn't draw frames, so we verified our animations by driving headless Edge over the DevTools protocol and freezing GSAP's clock frame by frame.

Accomplishments that we're proud of

It catches the con early: 20/20 scripted scams flagged, 0/10 false alarms, typically within two lines of the caller.
"Called it" is real: SafeKeep forecasts the scammer's next move and the heist board visibly proves it right, which turns a scary moment into a "we've got this" one for the person on the phone.
A real call, end to end: a live ElevenLabs agent call was transcribed, scored, flagged, warned on and closed with a signed webhook, all through our own backend.
Privacy is built into the architecture: no audio saved, no text kept unless there's a threat, and evidence that's tamper-evident and destroyable at the same time.
It feels like a heist movie: vault doors, a heist board, kill streaks, a vault that seals when you win. Protection people actually want to open.

What we learned

Scams are remarkably consistent, which is exactly what makes them detectable. Modelling them as heist "beats" made both the detection and the explanation clearer.
Combining a deterministic rules engine with an LLM beats either alone. Rules give speed and explainability, Gemini catches the paraphrases.
Designing for privacy changes the whole system, from what gets stored and when, to who holds the keys, down to what a report is allowed to quote.
Motion design is engineering too: shared GPU contexts, transform-only animation, reduced-motion fallbacks and layering all mattered as much as how it looked.
Legal requirements make good product features when you design with them instead of around them.

What's next for Safe Keep

A phone number for the protected line: connect the agent to a real number over SIP (Telnyx or Twilio), so members can merge it into any call, or forward unknown callers straight to it.
Real credit union integration, so the vault hold applies to actual transfers, plus a secured staff dashboard behind Cloudflare Zero Trust.
More scripts and languages: romance and investment scams, and multilingual warnings.
A pilot with members and their families to tune the alert threshold and the crew check-in for real-world calls.