-
-
Home Page
-
Parcel Delivery Scenario
-
Bank Scenario
-
Quiz
Inspiration
Scam calls work through pressure: fake authority, urgency, secrecy, and a request that sounds reasonable in the moment. Most people only learn what that feels like when it is real. ScamDrill is a safe place to practice resisting it.
What it does
An AI plays a fictional scammer in a chat-style call. You answer freely and, at four points, a decision card asks what you do: give your details, stay on the line, read out a one-time code, or move your money. Afterwards a debrief shows each key moment (what you chose, why it was safe or risky, what to say instead), which pressure tactics worked on you, and three rules to remember. A short quiz before and after lets you check what stuck. There are two scenarios, a fake bank fraud call and a fake parcel-fee call. Everything is fictional.
How we built it
The engine decides and the model phrases. Scenarios are authored JSON: five stages, four decision cards, and feedback text for every option. A deterministic state machine controls the flow and scores your decisions. Because we generate the scammer, we know which tactic each stage uses, so the debrief never has to guess what happened. The model only writes the caller's words.
- Backend: Python and FastAPI, stateless (the browser holds the call state).
- Frontend: React, Vite and Tailwind CSS.
- Safety: input masking (digit runs, emails, phone-like strings), an output filter on every caller line (URLs, emails, phone-like strings, real brand names, internal labels) with a scripted fallback, and rate limits.
- Deployed with Docker on Render.
Where we used Nemotron and Nebius Token Factory
Every caller line is written by NVIDIA Nemotron 3 Nano (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B) through Nebius Token Factory's OpenAI-compatible API. The model sees the stage goal and the recent conversation and replies in plain text. We did not use other Nebius services such as Serverless Endpoints or Jobs.
Challenges and what we measured
By default the model spent hundreds of hidden reasoning tokens before every short reply. In our first full evaluation (58 caller turns, scripted players) median latency was 22.9 s, 95th percentile 36.5 s, and 44.8% of turns timed out and fell back to a scripted line. Turning thinking off per request (chat_template_kwargs: {"enable_thinking": false}) brought median latency to 1.3 s and 95th percentile to 1.7 s, with no fallbacks, no unsafe lines, all 32 scored decisions correct and all 10 debriefs correct in the same 58-turn run.
Our first debrief design asked a larger model for structured JSON. In a small test (6 calls) half the responses fell back, so we replaced it with deterministic content built from the authored scenario data. That made the debrief instant and always consistent with what the player chose.
What we learned
Small models are good enough for short, in-character replies once their reasoning is controlled. Letting a state machine own the structure and the model own only the wording made the app predictable and easy to test.
Limits
ScamDrill is a practice tool, not a study, and we make no claim that it reduces fraud. Our evaluation uses scripted players and measures structure, safety, speed and scoring. It does not measure how convincing the caller sounds. English only. The caller sometimes invents details or repeats a request, and in one manual run the call ended before the last decision card appeared (we could not reproduce it). The project was built from scratch during the submission period.
What is next
More scenarios (family emergency, investment pitch), more languages, and an optional model-written coach note on the debrief.
Log in or sign up for Devpost to join the conversation.