callcanary

Scam calls don't sound like scams. They sound like the IRS, a bank's fraud department, or a grandchild in trouble who needs bail money right now. Older adults lose billions of dollars a year this way, and the warning signs (gift cards, "don't tell anyone," "read me the code") are easy to spot from the outside and hard to spot when someone is scared and on the phone.

Miners used to carry a canary underground because it noticed danger before people could. We wanted that for phone calls: a small, friendly lookout that goes quiet when everything is fine and sings out when something is wrong.

What it does

Live call protection: with the call on speaker, CallCanary listens, writes live captions, scores warning phrases, confirms with AI, and then the canary says out loud why it thinks the call is a scam.
Screen a caller: the canary answers unknown callers in its own voice, asks who they are and why they're calling, and runs layered checks: the FTC complaint list, what the caller said, warning phrases, your trusted contacts, and an AI verdict.
Check an email or link: catches look-alike domains (rnicrosoft.com vs microsoft.com), links that go somewhere other than where they say, fake senders, typos and pressure tricks.
Android app: becomes the phone's caller ID & spam app, checks every non-contact call against about 196,000 FTC-reported numbers offline, and blocks matches before the phone rings. A Connect tab lets you allow call access and connect Gmail.

How I built it

Web app: Next.js 14, TypeScript and Tailwind with shadcn/ui, Motion and Phosphor icons. It's designed for older users: buttons at least 56 px tall, large type, and the Atkinson Hyperlegible font (made for low-vision readers). Deployed on Vercel.
Speech: browser SpeechRecognition gives fast provisional captions, ElevenLabs Scribe gives the accurate transcript, and ElevenLabs text-to-speech gives the canary its voice.
Judgment: Google Gemini reads the transcript in context and has to quote evidence before it can call something a scam.
Android: Kotlin and Jetpack Compose, using Android's CallScreeningService role, with the FTC list bundled into the app.

Scoring a call. Each warning rule has a weight. Over a sliding 40-second window, each distinct rule counts once.

Here, the negation factor can reduce the score for phrases such as "I will not send money" and for conversations that are talking about scams rather than actually being scams. When the score reaches 35, CallCanary sends the recent audio for a full transcript and an AI check. The score only starts an investigation; it never decides the verdict.

Looking up a number. The FTC list ships as a sorted 1.1 MB file, so a lookup is a binary search. With about 196,000 numbers, it takes at most 18 comparisons. That's fast enough to decide before the phone rings, with no database or network.

Catching look-alike domains. We compare domains using edit distance after normalizing look-alike characters, such as changing "rn" to "m," "1" to "l," and "0" to "o." Without that step, the distance between "rnicrosoft" and "microsoft" is 2, which looks like an ordinary typo. With normalization, the distance is 0 and the imitation is obvious.

Challenges we ran into

Keywords caused false alarms: "I got you a gift card for your birthday" was flagged as a scam.
The AI could be confidently wrong: Gemini sometimes reported typos that weren't in the email, or rated a dangerous link as fine.
Slow, overloaded models: At peak times some AI calls hung for 20+ seconds, which is useless in the middle of a live call.
The canary heard itself. Its spoken warning came out of the speaker, went into the microphone, and set off another alarm.
Broken audio uploads: Gluing recorded chunks together made invalid files, and long clips went over Vercel's request size limit.
Websites can't touch your phone: A website can't block a number or delete an email, and early buttons looked like they did.

Accomplishments that I am proud of

Real blocking on a real phone: the Android app stops FTC-reported numbers before they ring, fully offline.
An AI that has to show its work: a scam verdict needs quoted evidence, and checks done in code can't be overruled by the model.
Built for the people scammers target: large type, a readable font, big buttons, and a canary that explains in plain words instead of just showing a red X.
Honest by design: every action either really happens or tells you exactly how to do it yourself. Demos are labelled DEMO.

What I learned

Context matters more than keywords: Counting each rule once, detecting negation and "talking about scams," and using the score only as a trigger removed most false alarms.
Don't let the AI overrule facts: Typos it can't quote are dropped, and the AI can raise a link's risk but never lower it.
Audio needs careful state handling: I released the microphone before the canary speaks and hold phrase triggers while it talks.
Respect formats and limits: I rotated independent 20-second clips and keep every upload under the size cap.

What's next for CallCanary

Warnings during normal calls: route unknown callers through a cloud phone number so the canary can warn you mid-call, without speakerphone.
Automatic inbox protection: read connected Gmail through Google's Gmail API and move scam emails out of the inbox automatically.
Family alerts: let a trusted family member get a notice when the canary sings.
Play Store release: publish the Android app so anyone can install it in one tap.