Inspiration

Most scam tools only say “scam” or “not scam” after the message has already landed. The people who lose the most money are often the ones least able to judge that answer: a grandparent who gets a “Hi Mom, new number” text, or anyone rushed by a fake bank alert. T-Mobile’s challenge asked for protection “in ways we haven’t thought of yet,” with detection as a core piece, not the whole product. So we built SCAMFIRE: like a fire drill, you practice before the emergency, and when a real suspicious message arrives, you get a clear answer, an honest confidence level, and concrete next steps.

What it does

Drill mode shows realistic texts, emails, and voicemail transcripts, and you choose Scam or Safe. It then reveals the truth, highlights the red flags inside the message, and shows the AI’s confidence next to yours (“The AI was only 62% sure; here’s why it’s tricky”). It adapts to the categories you miss, and each round ends with a Scam Resistance Score and your top 3 habits to build.

Check mode takes a pasted message and returns a verdict (scam, safe, or unsure), calibrated confidence, the scam type, highlighted warning signs, and clear next steps, like “Call your bank using the number on your card. Forward it to 7726 (SPAM).”

Privacy by design: phone numbers, emails, names, card and account numbers, and addresses are removed locally before anything could reach an LLM, and you can see the redacted version. Pasted messages are never stored, and drill progress lives only in your browser.

How we built it

Stack: Python and FastAPI on the backend. React, Vite, and TypeScript on the frontend, mobile-first with a text-size control and large tap targets. Data: T-Mobile’s dataset wasn’t available, so we wrote a generator for about 500 synthetic messages (60% scam, 40% legit) across 11 scam types, including near-misses like real fraud alerts and a friend asking for $20. We also hand-wrote two small “hard” sets in new wording. Detector: a TF-IDF and logistic-regression baseline, plus an improved detector that blends a text model with a signals-only model (urgency, payment requests, impersonation, look-alike links, credential requests) and calibrates the output. It can answer “unsure” and only says “safe” when very sure. Nemotron agent: a multi-step tool-calling agent over NVIDIA’s OpenAI-compatible API, with tools such as redact_pii, classify, lookup_taxonomy, and generate_drill. It explains verdicts and writes new practice messages for a user’s weak spots. A guard blocks any request containing an unredacted email, SSN, card number, or phone number, and the agent can raise “unsure” to “scam” but never to a confident “safe.” Status: the code is written but not yet tested against live Nemotron. Built with: Python, TypeScript, FastAPI, Uvicorn, Pydantic, scikit-learn, NumPy, pandas, SciPy, joblib, React, Vite, NVIDIA Nemotron. No hosted deployment and no database; the demo runs locally.

Challenges we ran into

Synthetic data makes easy tests. The baseline scored a perfect 1.000 on the random split because test messages share templates with training ones, so we added hand-written hard sets and report them separately. Our first improved detector under-called scams in unfamiliar wording. Instead of hiding that, we changed the product behavior to an asymmetric “unsure” zone that favors caution. Honest drill confidence. The detector was trained on the messages shown in the game, so we precomputed out-of-fold predictions so every card is scored by a model that never saw it. Near-identical messages. A gift-card scam and a real friend asking for $20 look almost the same, so some messages are genuinely ambiguous. Practical hurdles: a Windows dev-server reload bug, PowerShell script restrictions, and keeping redaction aggressive without removing signals the detector needs.

Accomplishments that we’re proud of

A working end-to-end app: a practice game with adaptive difficulty and a real-time checker that gives a verdict, confidence, highlighted flags, and next steps. Calibration: on a fresh hard set, ECE improved from 0.166 to 0.116 and precision was 1.00 vs. 0.80 for the baseline. Across all test sets we never confidently called a scam “safe.” Honesty built into the product: the app says “unsure” when it should, and we report where we did not beat the baseline (raw F1: 0.857 vs. 0.889 on the hard set). Privacy enforced in code rather than promised: local redaction, a visible redacted view, and no storage of pasted messages. What we learned Calibration and the willingness to say “unsure” matter more to trust than a single accuracy number. Privacy is a design constraint: enforcing it in code is more convincing than promising it. For non-technical people, the next step matters more than the label. Evaluation on synthetic data can flatter a model, so hold-out sets in new wording are essential.

What’s next for SCAMFIRE

Evaluate on T-Mobile’s real data and taxonomy, and on real scam messages (everything here is synthetic). Finish live Nemotron testing and measure how much it improves “unsure” cases. Trusted Circle: with consent, alert a chosen family member on a high-risk message. “Turn this into a drill”: convert a real scam into a training card, with personal details removed. Put SCAMFIRE where people already talk, such as iMessage and SMS, so they can forward a suspicious message and get the verdict back in the same chat.

Built With

Share this project:

Updates

Submission history