What inspired this
In May 2026, someone posted Morse code on X. Grok decoded it into a plain-English command tagging @bankrbot, telling it to send 3 billion DRB tokens to an attacker-controlled wallet. Bankrbot executed it. No private keys were compromised. The failure was in the handoff: one agent decoded, another agent executed, and nobody validated the authorization path between them.
That incident made agent security concrete. Prompt injection is usually discussed as a model-behavior problem. Jailbreaks, data leaks, off-topic responses. But when an agent has tool access, wallet permissions, or API keys, the blast radius changes. A customer-service agent that sends a bad email is a review problem. A trading agent that signs a transaction is an asset-control problem.
I built Agent Auditor because most teams deploying AI agents have no way to test for these failure modes before going live.
What it does
Agent Auditor is a red-team testing platform for AI agents. You point it at an agent endpoint and it:
- Probes the agent's attack surface with exploratory requests
- Generates adaptive adversarial scenarios using Gemini (prompt injection, tool misuse, parameter manipulation, multi-turn confusion, data exfiltration, and more)
- Runs each scenario, including multi-turn conversations that exploit context trust
- Scores every response with an LLM judge calibrated against OWASP LLM Top 10 (2025) categories
- Generates code-level remediation suggestions for each finding
- Traces everything to Arize Phoenix for observability
- Exports evidence-backed PDF and JSON reports
The core insight is that testing an agent requires an agent. Static prompts and one-shot jailbreaks miss the interesting failures. The ones that happen across multiple turns, through tool calls, or via indirect injection through external data. Agent Auditor uses Gemini to reason about what to try next based on what the target agent exposed so far.
How I built it
Agent runtime: The auditor agent runs on Google Cloud Agent Builder using ADK (Agent Development Kit) with Gemini 2.5 Flash. It uses FunctionTool calls for probing, scenario generation, attack execution, and judging. The full agent loop.
Phoenix MCP integration: The auditor loads @arizeai/phoenix-mcp as an MCPToolset. This lets it query past audit traces directly, rank attack categories by historical vulnerability rate, and bias the next scenario generation accordingly. The self-improvement loop: audit, trace, learn, adapt.
LLM-as-judge: Each scenario response is scored by Gemini with calibrated anchors mapped to OWASP categories. Results are written to OpenInference spans so you can inspect the judge's reasoning in Phoenix.
Built-in victims: Four target agents ship with the platform. Two keyword-matchers (customer support, banking), one real ADK agent with FunctionTool calls (enterprise support, deliberately vulnerable with OWASP LLM05:2025 Excessive Agency), and a custom URL option for testing your own.
Frontend: React 19 with Tailwind CSS 4. Live streaming via SSE so you can watch each audit phase as it completes. Reports page with persisted audits, radar charts, findings, and export links.
Deployment: Google Cloud Run, Docker, SQLite for persistence, optional Elasticsearch export for dashboards.
What I learned
The biggest lesson was that testing an agent is fundamentally different from testing a traditional application. You can't just send a request and check the response. You need to reason about what the agent revealed, craft follow-up attacks that exploit that information, and evaluate whether the failure is a security issue or just a quality issue. That requires an agent with memory, planning, and judgment. That's why the auditor itself is built on ADK.
The Phoenix MCP integration turned out to be the most interesting part architecturally. Having the auditor query its own past traces to improve future attacks creates a feedback loop that static test suites can't replicate. It also means the tool gets better the more you use it.
OWASP's LLM Top 10 (2025) was useful as a classification framework but insufficient on its own. Categories like "Excessive Agency" describe the symptom, not the root cause. The remediation suggestions need to be specific to the agent's implementation. Which tool has too much permission, which input is unsanitized, which confirmation step is missing.
Challenges
Scenario quality. Generating adversarial scenarios that are realistic and diverse is hard. Gemini is good at this but the prompt engineering to avoid repetitive attacks took significant iteration. The adaptive loop (querying Phoenix for past results) helped a lot.
Multi-turn conversations. Most prompt injection demos are single-shot. Real agents hold conversations across multiple turns, and trust builds over time. The auditor needed to maintain conversation state and exploit that trust escalation, which added complexity to both the attack execution and the judge evaluation.
Judging calibration. Telling the LLM judge "score this 1-10" is useless without calibration. I had to build a rubric with concrete anchors: what does a 3 look like vs a 7, what evidence is required to flag a finding, how to distinguish a security issue from a UX issue. The OWASP mapping helped here.
Deployment. Getting the full stack (ADK agent, Phoenix MCP, FastAPI, React frontend, SSE streaming) running on Cloud Run with proper networking for MCP tool calls was more involved than expected. MCP servers spawn child processes that need specific environment setup.
Arize track alignment
| Criterion | How Agent Auditor addresses it |
|---|---|
| Agent Builder | Auditor agent built with Google ADK using Runner, InMemorySessionService, and FunctionTool calls |
| Tracing | phoenix.otel.register(auto_instrument=True) instruments the full ADK runtime with OpenInference spans |
| Phoenix MCP | @arizeai/phoenix-mcp loaded as MCPToolset for direct access to projects, spans, datasets, and annotations |
| Self-improvement | adapt_scenarios_from_history queries Phoenix for past traces, ranks categories by vulnerability rate, biases next audit |
| LLM-as-judge | Gemini 2.5 Flash scores each scenario with calibrated anchors, results written to OTel spans |
| Real agent testing | Enterprise victim is a real ADK agent with FunctionTool calls, not a keyword stub |
| Multi-turn | Auditor holds 3+ turn conversations, exploiting context trust escalation |
| Remediation | Every vulnerability gets a Gemini-generated code-level fix suggestion |
| OWASP mapping | Findings mapped to OWASP LLM Top 10 (2025) categories (LLM01-LLM10) |
| Persistence | SQLite with trend charts showing score evolution over time |
