Canary AI

• 

Inspiration

Existing scam protections act before a call (blocking numbers) or after (damage control). Nothing protects victims during the call, when high-pressure tactics are at work. Our core inspiration comes from online scam baiters, scam educators, and hands-on experience seeing how elderly individuals are targeted. Older adults often struggle to decipher what is real or fake during the conversation and face steep barriers navigating technology.

What it does

Canary AI sits in the background during unknown or suspicious phone calls, using call forwarding to bridge calls in real time.

Dual-Path Detection Engine: Evaluates every finalized sentence against high-risk instant triggers (like gift card demands, verification codes, or remote access requests) while accumulating scores for broader pressure tactics (authority claims, urgency, and secrecy).

Parallel AI Opinion: Runs Google Gemini in parallel alongside custom rule scoring to evaluate underlying intent and contextual nuances.

Immediate Intervention & Human Loop: Plays an audible warning directly into the call to interrupt scammers, sends an instant alert with the exact trigger phrase to a trusted contact, and updates a live dashboard featuring real-time risk scores, call states, and transcripts.

How we built it

Telephony & Audio Streams: Twilio Voice using Conference bridges and Media Streams to silently route calls and isolate dual-channel audio streams.

Speech-to-Text Processing: Deepgram for ultra-fast, real-time speech transcription.

Voice Generation: ElevenLabs to generate clear, high-quality audio warnings that play into active calls when high-risk tactics are detected.

Scam Detection Engine: Custom rule scoring engine running concurrently with Google Gemini for intent analysis.

Backend & Database: Node.js, TypeScript, and Fastify, powered by LibSQL (SQLite) deployed on Railway.

Frontend & Emergency Alerts: React with Vite and Tailwind CSS for the live contact alert page and monitoring dashboard, integrated with Twilio Messaging for SMS dispatch.

Challenges we ran into

Score Accumulation Bug: We encountered an issue where trigger words were correctly detected by our scanner, but the underlying risk score failed to increment properly. Resolving state synchronization across live evaluation streams fixed the pipeline.

SMS Alert Dispatch & Carrier Limits: Setting up the text message alert system required navigating Twilio trial account constraints and US carrier registration requirements (A2P 10DLC) to ensure emergency alerts reach trusted contacts mid-call.

Streaming Audio Precision: Live speech-to-text continuously updates mid-sentence. To prevent misheard or incomplete words from triggering false alarms, we engineered the engine to only evaluate finalized sentences.

Balancing Thresholds: Fine-tuning detection thresholds so realistic scam scripts land accurately above the alarm cutoff without flagging legitimate calls (like clinic appointment reminders or security alerts warning users not to share codes).

Accomplishments that we're proud of

True In-Call Intervention: Delivering an audio warning directly into the call and dispatching alerts within seconds of a scam request, while the fraudster is still on the line.

Actionable Evidence Alerts: Displaying the exact triggering quote (e.g., "asked for gift card numbers") in alerts rather than an unexplained risk score, giving family members immediate clarity.

Precision Against False Alarms: Rigorously testing against 64 synthetic scam scripts across 8 categories (government, bank, tech support, romance) while keeping the system completely silent on legitimate calls.

Seamless End-to-End Loop: Connecting live phone lines, real-time speech processing, AI intent scoring, ElevenLabs audio generation, SMS messaging, and web dashboards into one functioning workflow.

What we learned

False Alarms Destroy Trust: A single false positive leads users to ignore future warnings, making silent precision on normal calls just as critical as catching real scams.

Evaluate Intent, Not Delivery: Judging speech patterns, accents, or grammar causes false alarms; real-time protection must focus strictly on the specific request and pressure tactics.

Real-Time Systems Punish Assumptions: Streaming live audio taught us the necessity of sentence-completion checks so dynamic transcription updates don't trip triggers prematurely.

Transparency Over Labels: Presenting evidence and letting family members make the final decision is far more effective and responsible than assigning automated "safe" or "verified" labels to callers.

What's next for Canary AI

Direct Push Notifications: Upgrade from SMS to instant mobile push alerts for faster response times.

Email Alert Dispatch: Expand emergency notifications to include instant email alerts with detailed evidence summaries.

Bank & Enterprise Expansion: Partner with banks and credit unions to offer a white-label API that proactively intercepts financial fraud for vulnerable account holders.

Built With

• deepgram
• elevenlabs
• googlegemini
• node.js
• react
• sqlite
• tailwindcss
• twilio
• typescript
• vite