Scam Shield Agent

Inspiration

Phone scams work because they happen in real time. By the time someone realizes the "bank" or "IRS agent" on the line isn't real, they've often already read out a code or agreed to a transfer. Most existing protection, like carrier spam labels and blocklists, only looks at the number before you pick up. Once you're talking, you're on your own. We wanted something that listens alongside you and steps in during the conversation, when the manipulation is actually happening.

What it does

Scam Shield Agent is real-time scam call interception. A Twilio number sits in front of your real phone and forwards calls through to you. While you talk, ElevenLabs Scribe transcribes both sides of the conversation live.

Every turn, the caller's behavior gets scored. A PRISM guardrail pass runs first to catch known red flags (urgency pressure, requests for gift cards or verification codes, impersonation of institutions), then an LLM through Nicrron evaluates the conversation as a whole for subtler manipulation patterns. If the combined risk score crosses the threshold, the system plays a short audio warning into your ear and drops the caller from the line.

Built for Money Talks 2 (Block Convey), Trust & Risk track.

How we built it

The call flow runs on Twilio, which routes incoming calls through our service before connecting them to the user's phone. Audio from both legs streams to ElevenLabs Scribe for transcription.

The frontend, consisting of the ElevenLabs interface and the internal guardrail mechanisms, was built by William Spangenberg. The Nicrron scoring pipeline and the PRISM guardrails were built by William McGuire.

Scoring is layered on purpose: PRISM handles fast, rule-based checks, and the LLM handles context that rules miss, like a caller slowly building false trust before making a request.

Challenges we ran into

Latency was the main constraint. Transcription, guardrails, and LLM scoring all have to finish within a single conversational turn, or the warning lands too late to matter.

We also ended up with two layers of guardrails: the internal ones on the frontend and the ones we built with PRISM. Running both currently leads to some false positives, since a phrase that's harmless in context can still trip one of the layers. Tuning how the two interact is ongoing.

Accomplishments that we're proud of

We got a working end-to-end pipeline: a live call comes in, gets transcribed, scored turn by turn, and the scammer gets cut off with an audible warning to the user. Intervening mid-call, rather than just flagging a number beforehand, was the core idea, and it works.

What we learned

Scam detection is as much about conversation flow as keywords. Single phrases are rarely damning on their own, so the LLM's ability to weigh the whole exchange made a real difference over rules alone. We also learned a lot about the timing constraints of real-time voice pipelines and about working with Twilio media streams.

What's next for Scam Shield Agent

• Reduce false positives by consolidating the two guardrail layers and tuning thresholds against real scam call transcripts
• Softer intervention options, like a warning without dropping the call, so users stay in control
• A post-call summary explaining why a call was flagged
• Support for trusted contacts, so a family member can be alerted when someone vulnerable is being targeted.

Built With

• nicrron
• prism
• python