Inspiration
AI assistants are starting to act on our behalf: reading email, paying bills, sending messages. That means a single phishing email can now cause a real action. The core attack is prompt injection: an attacker hides instructions in content the agent reads ("we changed banks, send the rent here instead"), and the agent can't reliably tell those apart from what the user actually asked. We wanted to build a defense that doesn't depend on the model being careful, and to measure honestly how well it works.
What it does
AgentGuard sits between an AI assistant and its tools. Before any action runs, like a transfer or an email, it checks one question: did the user actually ask for this, or did it come from content the agent read?
- Rules (fast, deterministic): allow known contacts and recipients the user named, and block sending money or data to unknown recipients when the instruction came from email content
- LLM judge for unclear cases, such as a known payee with the wrong amount. Email content is fenced off as untrusted, so the judge itself can't be injected
- Provenance tracking: the agent loop tags every action with where its instruction came from, in code, so a hijacked model can't lie about it
- Live dashboard showing every tool call, the guard's decision, and the reason in plain English
We also built a red-teaming arena that generates phishing attacks with an LLM, evolves the ones that work, and runs every attack with the guard off and on.
How we built it
- Agent: a Python loop on the Gemini API (gemini-3.5-flash-lite) with email and bank tools, JSON-mode actions, and retry handling for rate limits
- Environment and backend: a sandboxed inbox and bank per run, with a canary that flips if money or email reaches the attacker. A FastAPI proxy routes every tool call through the guard and logs it to SQLite
- Guard: allowlist and provenance rules, plus a Gemini-based judge that fails closed
- Attacks: 8 hand-written seeds across 4 attack types, expanded to 40 LLM-generated variants, then 20 second-generation attacks bred from the one that worked
- Evaluation: an arena that runs attacks and benign tasks with the guard off and on, a held-out set split by attack family, and metrics with 95% confidence intervals
- Dashboard: vanilla HTML/CSS/JS, deployed with the backend on Vultr
We split the work by folder around a shared Pydantic contract, so the two of us could build in parallel and merge cleanly.
Results
| Experiment | Guard off | Guard on |
|---|---|---|
| Real Gemini agent, proactive prompt, 20 evolved attacks | 45% hijacked | 0% |
| Real Gemini agent, default prompt, 36 attacks | 2.8% hijacked | 0% |
| Compromised agent, 36 tuning attacks | 100% | 0% |
| Compromised agent, 12 held-out attacks (unseen families) | 100% | 0% |
| Guard blocks on legitimate tasks | 0 |
Challenges we ran into
- Our first attacks didn't work. All 8 hand-written phishing emails failed against the unguarded agent, so there was nothing for the guard to stop. LLM-generated variants found one that did: a calm reply in the real electricity-bill thread, with the exact amount and new "bank details". Breeding a second generation from it raised the hijack rate to 45% on a proactively configured assistant.
- Measuring honestly. Our first summary numbers mixed experiments and used the wrong denominators. We rebuilt the metrics so attack success is counted over attack runs only, and added confidence intervals so small samples aren't oversold.
- Telling false positives apart from agent mistakes. Two normal runs failed with the guard on, but the logs showed the guard never blocked anything: the agent skipped a step on its own.
- API rate limits during large runs, which we handled with retries and resumable runs, while making sure rate limits in the judge didn't turn into fake blocks.
Accomplishments that we're proud of
- On a real Gemini agent, the hijack rate dropped from 45% to 0%, with non-overlapping confidence intervals
- Against a fully compromised agent, the guard blocked every attack, including attack families it was never tuned on
- Zero guard false positives on legitimate tasks, including paying and emailing new recipients the user named
- We report the weaker results too, and list our limitations openly
What we learned
- The most dangerous attacks aren't the obvious "IGNORE PREVIOUS INSTRUCTIONS" ones. They look like routine business email: the right thread, the right amount, a plausible reason.
- How an assistant is configured matters as much as the model. The same model went from 2.8% to 45% hijack rate when told to handle routine admin without checking in, which is exactly how useful assistants get set up.
- Evaluation design is half the project: held-out sets, ground-truth canaries, and confidence intervals change what you can honestly claim.
Limitations
- Every attack sends to an address the user never named, which the rules catch. Attacks against known recipients rely on the LLM judge, which we only tested on hand-built cases.
- Sample sizes are small (12 to 36 attacks per set), so confidence intervals are wide.
- The environment is a sandbox with a fixed contact list, and everything uses one model family.