Inspiration
An AI assistant that can call tools does what the loudest text in its context tells it. A user can talk it into leaking a secret or running a harmful tool call, and a better prompt doesn't fix that. We wanted a layer the model can't talk its way past. We also noticed that students rarely learn why an attack worked unless they have to explain it afterward, so we built both halves: a firewall, and a classroom that learns from what the firewall saw.
What it does
Injection Arena has two subsystems in one app.
1. Arena + Agent Firewall Framework (AFF). Students attack a company chatbot at three levels: Off, Basic and Strict. The AFF checks every input, every tool call and every reply, then a Gatekeeper picks one of four actions:
- Allow: the mock tool runs or the reply is released.
- Deny: a neutral refusal. Nothing runs.
- Review: the item is held for a human firewall admin. The student sees "pending human review."
- Divert: the firewall pretends the action worked, returns a believable decoy and logs trace-back details. The real tool never runs.
When actions disagree, the strictest wins: Divert > Deny > Review > Allow. In Off mode the firewall still computes what it would have done and logs it, so the Off/Basic/Strict comparison is real.
2. Classroom. The framework logs every score, rule and decision. Our classroom consumes that log: teachers see sessions and class insights, and each student gets a quiz generated from their own session (multiple choice plus explain-it-to-a-classmate questions with AI follow-ups), graded on a locked rubric. Any other app could consume the same log, such as a security dashboard or an audit trail.
How we built it
Each checked item gets a Sentinel score from 0 to 100, built from rules, keywords, session history, a risky-history lookup (Strict only) and a Gemini judge (Strict only). The judge can only raise the score:
$$S_{final} = \max\left(S_{rules+history},\ S_{judge}\right)$$
So deterministic rules are the boundary, and models recommend but never decide. Hard rules override the score, and any error or timeout fails closed to Deny. In Strict, a cumulative scan joins everything the student has seen in the session, which catches secrets leaked a few characters at a time.
After a Deny or Divert, an async worker uses Gemini to write a forensic report and draft a new rule. The draft is validated against benign fixtures, and nothing is applied until an admin approves it.
All tools are mocks on session state in Postgres, so nothing a student writes runs on a real host. The demo uses synthetic secrets only.
Challenges we ran into
(Edit this to match what really happened. These are the challenges our design had to solve.)
- Getting the bot to be attackable at Off without making the scenario silly.
- The split-secret attack: no single reply contains the secret, so per-message scanning misses it.
- Keeping decoys convincing without ever leaking the real secret. Every decoy is scanned, with a template fallback.
- Bot and security models are the same vendor, so the "different vendors" protection is gone. That's why rules decide and the judge can only raise the score.
- Three people building three parts against one contract package, with fakes until the real code landed.
Accomplishments we're proud of
- The same attack succeeds at Off, is partly caught at Basic, and is stopped at Strict, including a live human review and a Divert.
- A drafted rule that blocks an attack in a new session after admin approval.
- A quiz that turns an attack session into an explanation exercise.
What we learned
Layering matters more than any single check, and the gaps between layers are the best teaching material. We also learned how much cleaner a design gets when logging is a first-class output, because the classroom came almost for free from the event store.
What's next
Auth0 push approvals for reviewers, a local classifier score, honeytokens that report when a stolen secret is used, and a report of results per mode on a held-out attack set.
