Inspiration
Callback scams (also called TOAD — telephone-oriented attack delivery) are one of the fastest-growing fraud vectors, and they're built to slip past every defense a security team already has. The payload isn't a malicious link or attachment that can be easily scanned — it's a phone number, sitting in an otherwise-plain text email. By the time a victim is on the phone with the "IT Helpdesk" that's asking them to install remote desktop software, no security tooling is watching that conversation at all.
What it does
The Callback Scam Screener is a CALL-E-powered pipeline that takes a suspicious email, extracts the phone number and claimed reason, and has CALL-E place one screening call to that number. It does so transparently, telling the receiver it is as an AI assistant verifying the requested call. The transcript is analysed using an LLM and the call is scored against a checklist of real scam behaviours (remote-access-software requests, gift-card/crypto/wire payment asks, urgency escalation, evasiveness about company identity). The result is a verdict, the triggered signals, and the full transcript.
Critically, the agent is built so it has nothing to give away: no real account numbers, no ability to authorize a payment, no ability to install anything.
The aim of the tool is not to try to clear anything as 100% safe, but rather to help the security team triage the investigation of potentially suspicious messages. A 'likely_scam' verdict could be used to automate taking a response like deleting an email from an end-user's inbox, whereas a 'likely_legitimate' or 'inconclusive' verdict could fast track that email for human investigation. The tool reduces workload on the obvious, high-confidence malicious cases, it does not replace review of the rest.
It's also bring-your-own-everything: no bundled credentials, and the LLM layer is provider-agnostic by design. I built it against Google Gemini and can add further integrations if requested.
How we built it
- Signal catalog first. Before writing any call logic, we defined a tiered checklist of scam behaviors (critical / high / medium) with explicit scoring rules, so every verdict is explainable — a SOC analyst can see exactly which signal fired and read the supporting quote themselves.
- Goal-driven, not scripted. CALL-E's actual model is a
task(a goal) plus real-time adaptation, not a fixed dialogue tree — which matched what we wanted from the start: give the Screener agent an aim and let it have a natural conversation, while the hard safety boundary lives in what data/tools it has access to, not in what it's told not to say. - Real, live, end-to-end calls. We placed a real CALL-E call, with real Gemini-based transcript tagging, all the way through to a verdict.
- Hard guardrails in code: A dev/test number allowlist, one-screening-call-per-number, a call cap matching the free tier, and a tuneable $/day LLM spend cap tracked from real token usage. This ships with no bundled credentials; every user brings their own CALL-E and LLM keys.
Challenges we ran into
- CALL-E refused our first call plan outright. We initially designed the Screener to "adopt the persona of the person who received the scam email" — CALL-E's own guardrails rejected that as impersonation before it would even place the call. We had to redesign the framing to be transparently AI. A later revision hit the same guardrail again from a different angle — quoting the scam's specific payment claim back in the call goal also got flagged — so the final goal is deliberately neutral, which is also better security design as it doesn't hand a potential scammer information about what we already know.
- Our initial keyword-based signal tagger failed on real speech. Running it against the real call's actual transcript scored zero signals despite real red flags being present. This led to implementation of an LLM-based tagger. -Ensuring phone number redaction for privacy
- A real answering-machine pickup that scored as likely_legitimate before the detection was fixed.
Accomplishments that we're proud of
- A PR merged into the official repo.
- A safety design where the agent can't leak anything or take any action is enforced by what the agent has access to, not by what it's told.
- LLM provider-agnostic "bring your own key" design.
What we learned
I was really impressed with how responsive CALL-E was and how it was able to hold what felt like a real conversation. I didn't realise that was possible yet.
Keyword/heuristic-based scam detection doesn't generalise. Every real phishing email tested used different phrasing than the last.
What's next for Callback Scam Screener
Adding support for additional LLM providers. Further changes will depend on feedback.
Log in or sign up for Devpost to join the conversation.