Scam My Grandma

An AI call firewall for older adults: it listens to every call, cuts off scammers mid-sentence, and iMessages the family the moment something sounds wrong.

Comment

Inspiration

Every one of us has an older relative who has picked up a call from "the IRS," "Medicare," or "your grandson in jail." Americans over 60 lose billions of dollars a year to fraud, and the phone is still the scammer's favorite tool. Caller ID can't help: it only knows a number, not what the caller is about to ask for. We wanted something that works the way a protective family member would: answer the call, listen, and hang up the moment it turns into a scam.

What it does

Guardian is a call firewall for older adults. Meet Linda, 78:

  • Family and known contacts (her son, her doctor, her pharmacy) ring straight through, never screened or recorded.
  • Known scam numbers are rejected before her phone ever rings.
  • Everyone else is screened live. Linda answers while Guardian listens to every sentence and scores the call for scam risk in real time.
  • When a call starts to look suspicious, Linda's family gets an iMessage heads-up. The moment it crosses the line, Guardian cuts the call in its own voice, saves the transcript as evidence and texts the family, who can reply BLOCK or SAFE.

On our site you can call Linda from your browser and try to scam her yourself.

How we built it

  • Voice: Linda is an ElevenLabs conversational voice agent. The browser is the phone line: the backend hands the page a signed session URL, so no API keys reach the client.
  • Listening: after each caller sentence, the page sends the exchange to our FastAPI backend. The scorer gives the whole conversation so far to Claude Haiku, which returns structured JSON: a risk $r \in [0,1]$, the scam stage (hook → authority → urgency → isolation → payment), and short red flags. Keyword rules run alongside as a backup. An obvious scam on the rules alone ends the call instantly, without waiting for the model.
  • Decisions: $r \ge 0.4$ sends the family heads-up, and $r \ge 0.7$ cuts the call.
  • Number reputation: every number gets a 0–100 scam score shared across all Guardian users:

$$ S = \operatorname{clamp}{[0,100]}\Big(45\,r{\max} + \min(10f,\,30) + 10\,\mathbb{1}{\text{5+ users/24h}} + 10\,\mathbb{1}{\text{3+ short calls}} + 5\,\mathbb{1}{\text{night}} - 40\,\mathbb{1}{\text{verified}}\Big) $$

Here $f$ is the number of families who flagged the number. Linda also gets a weekly safety score: $100 - 10U - 5C - 20B$, where $U$ is unverified callers connected to her, $C$ is scam calls screened, and $B$ is banking-app alerts.

  • Family alerts: a Node service on Photon Spectrum sends the iMessages and routes replies back to the backend.
  • Data: we built a 2,500-call labelled dataset (1,500 scam / 1,000 legit). It combines 800 real-world transcripts with 1,700 calls we generated with Claude, written specifically as conversations with a screener, so the model sees how scammers dodge questions. We used it to fine-tune Qwen2.5-1.5B on Veri.
  • Infra: Postgres/TimescaleDB on Tiger Cloud, the backend and messenger in one container on Railway, the site on Vercel.

Challenges we ran into

  • Our first scorer was blind. The keyword-rule version scored "I'm calling from the tax department… you have to pay 2,000 or you'll be fined" as 0.0 risk and connected the call. Switching to an LLM that reads the whole conversation took the same call to 0.90.
  • Don't trust the voice agent to call your tools. At first, the voice model was supposed to call a scoring tool after every answer, and sometimes it simply didn't. We moved scoring out of the model's hands: the page reports every exchange itself, so Guardian always listens.
  • Telephony compliance. Our SMS provider blocked sending until a business verification was approved, which pushed us to iMessage through Photon. The result is better for families anyway.
  • Hosting long-lived connections. Live call state, a server-sent event stream and a persistent iMessage connection don't fit serverless platforms, so we split static hosting (Vercel) from an always-on container (Railway).
  • Latency vs. accuracy. A scorer that takes too long makes the call feel robotic. Rules-first short-circuiting plus a small, fast model kept decisions to about one second.
  • Privacy by design. California requires all-party consent, so every call opens with a recording notice. Analysis stops at 90 seconds, and only scam calls are kept, for 90 days.

What we learned

  • A caller's intent only shows up across several turns. Context beats keywords.
  • Production voice AI means planning for the model not doing what you asked, and keeping critical logic in your own code.
  • The best safety tool is one that loops in a human: Guardian doesn't replace family, it alerts them.

What's next

  • Serve our fine-tuned model as the primary scorer.
  • Route real phone numbers through Guardian.
  • Add a family dashboard, and let all Guardian users share scam-number intelligence.

Built With

Share this project:

Updates