Inspiration
Americans over 60 lost $7.75 billion to phone scams last year. That's 201,266 complaints to the FBI, up 59% in one year. The usual answer is awareness material - pamphlets, seminars, lists of red flags. There's research showing it isn't enough. In one study, 240 people were told a scam call was coming, and many of them still gave up sensitive information. Knowing the rule and following it while someone is pressuring you are two different things. Only one of them ever gets practiced.
What it does
You get a call. It sounds like your bank's fraud team, and it works the way real ones do - a deadline, a case number, a small question first, then the real ask. When it ends, we ask four questions. How sure are you that you'd hang up and call your bank on the number from your own card? Then we show you what you did. For example, you said you were certain you'd verify, but at 1:12 you read out your account number. Every number on the screen points back to something you actually said, with the timestamp. Round two goes after whichever habit failed, and shows whether it got better. The results go to the family, not the person who took the call.
How we built it
React on Vercel, FastAPI and SQLite on Render. ElevenLabs runs the caller in the browser, so we capture the transcript with timestamps as it happens. Nemotron scores it. We split the scoring on purpose. Python does the arithmetic - the confidence percentages, the biggest gap - so the headline number isn't something a model made up. Nemotron does the part only it can do: reading a conversation and judging behavior against a written rubric, with a quote and a timestamp for every call it makes. We check its output before anyone sees it. Quotes have to appear word-for-word in the transcript, and anything outside the rubric gets dropped.
Challenges we ran into
ElevenLabs blocked our agent, and they were right to. Our first prompt told a voice agent to impersonate a bank and ask for account details. No classifier can tell that apart from a real vishing tool. We rewrote it as an explicitly consented drill - better framing, and it passed. A race condition that produced convincing nonsense. We started scoring the moment the call ended so the wait would overlap with the questions. But the answers arrived after that, so the scorer read an empty quiz and returned zero confidence on everything. The debrief still rendered. It just meant nothing. We moved scoring after the answers and added a guard so it can't fail quietly again. The caller kept breaking character at the worst moment. A rule meant to protect distressed users was firing when someone said "I'm scared" - which is exactly when a real scam is working. We narrowed it to explicit requests to stop.
Accomplishments that we're proud of
We tested our own scoring instead of assuming it worked. Three transcripts, four behaviors each, labeled by hand before the model saw them. It agreed with us 75% of the time. The disagreements all ran the same direction, and the model gave the same answer every time we ran it - so it wasn't unreliable, it was consistently wrong in one specific way. That turned out to be our fault. Our definition of "partial" didn't cover someone who says they'll call the bank back and then doesn't. We rewrote it and agreement went to 89%. We're also glad it refuses to guess. If nothing in the transcript supports a judgment, the score comes back empty instead of invented.
What we learned
Testing a model is cheap, and it changes what you build. Four hours of labeling turned a vague feeling that the scores "looked about right" into a specific problem we could fix. Silent failures are worse than crashes. A function that returned zero when a lookup failed looked identical to someone answering "not at all sure," and it cost us most of a night. And being blocked on safety made the product better. It forced us to say clearly what we were actually building.
What's next for Scam Gym
Real calls. Right now the drill runs in a browser, so you know it's coming. The version that matters arrives unannounced on an enrolled person's phone, with disclosure. That's the only way to test surprise. More scenarios. The engine doesn't care what the scam is. Tech support, government impersonation, grandchild in trouble - those are prompt files, not new code. Credit unions. They already run fraud programs for member families. A monthly drill with a score that goes up is something an institution can measure, and keep paying for.
Log in or sign up for Devpost to join the conversation.