Mutiny: Break Agent Rules, Prove It

Inspiration

AI agents are no longer limited to generating text. They can call tools, modify data, issue refunds, send emails, delete accounts, and take actions on behalf of users.

That creates a different kind of security problem: an agent can sound safe while still taking an unsafe action.

We wanted something closer to fuzzing than prompt review: define what an agent must never do, search for conversations that cause it to break those rules, prove the failure using real tool calls, and turn the failure into a permanent regression test.

That became Mutiny — behavioral fuzz testing for AI agents.

What it does

Mutiny lets developers define explicit, machine-checkable policies for their agent's tool use.

For example:

Refunds over $100 must not be issued without approval.

The workflow is:

Define Policy
      ↓
Search
      ↓
Execute Against Agent
      ↓
Observe Tool Calls
      ↓
Prove Violation
      ↓
Minimize
      ↓
Save Regression
      ↓
Fix Agent
      ↓
Run Tests
      ↓
PASS

Mutiny evolves adversarial conversations against the agent through an adapter. It then evaluates the resulting execution traces with a deterministic policy oracle.

We deliberately don't use an LLM judge as the final source of truth.

If the agent actually produces something like:

issue_refund(amount=250, approved=false)

while the policy forbids refunds above $100 without approval, Mutiny has a concrete, reproducible violation.

It then minimizes the conversation, saves the failure as a regression, and allows the developer to replay it after fixing the agent.

AI proposes. Code proves.

How we built it

Mutiny is built around a small, framework-independent core.

The Core contains the campaign engine, policy evaluator, fitness system, mutation engine, minimizer, execution traces, and regression system. Framework-specific integrations implement a common adapter interface, keeping framework-specific logic outside the trusted decision-making layer.

For the first adapter, we integrated the OpenAI Agents SDK.

The project also includes:

Python Core engine for search, policy evaluation, minimization, and regression replay
OpenAI Agents SDK adapter for testing real agent projects
CLI with mutiny init, mutiny run, and mutiny test
Project-scoped policy.yaml files
Hosted API for campaigns, persistence, and SSE
Hosted dashboard for projects, policies, campaigns, evidence, regressions, and tests
Customer-shaped sample project demonstrating the complete workflow
Regression replay shared between the CLI and Hosted platform

The public demo is hosted using Next.js/Vercel for the UI and Railway for the API.

For real customer projects, Mutiny can run locally through the CLI against the developer's own agent project.

What inspired us

The idea came from thinking about how we test ordinary software.

We don't trust a function simply because it worked once. We write unit tests, fuzz inputs, reproduce failures, minimize them, and turn important failures into permanent regression tests.

AI agents deserve the same treatment.

Traditional prompt testing often focuses on what the model says. But for an agent, the more important question can be:

What did the agent actually do?

That led to Mutiny's central idea: treat agent tool-use behavior like software behavior that can be fuzz-tested against explicit invariants.

We also wanted to be honest about existing work. Mutiny is not claiming to invent agent red teaming, evolutionary prompt attacks, custom policies, or regression evaluation. Existing projects demonstrate important parts of this space.

Our focus is the composed developer workflow:

search → prove → minimize → regress.

Challenges we ran into

Proof vs. vibes

Using an LLM as the final judge is tempting, but it introduces uncertainty into something that should behave like a test.

We therefore made the acceptance oracle deterministic for the tool-use policies Mutiny supports. The final decision is based on structured execution traces and actual tool-call arguments.

Hosted vs. local testing

A public Hosted service cannot directly reach an agent running on a developer's laptop.

We therefore separated the experiences:

Hosted: an easy-to-try public environment using a sandboxed sample project
Local CLI: the path for testing a developer's own agent project

Keeping that boundary clear was important. We wanted the demo to be easy without pretending that Hosted can magically access arbitrary private agents.

Making evolutionary search useful

Generating random adversarial prompts isn't enough.

Mutiny uses policy-aware fitness signals to guide the search toward relevant tool and argument boundaries while keeping the final acceptance decision deterministic.

Trustworthy minimization

A shortened exploit isn't useful if it no longer reproduces the failure.

Mutiny therefore re-executes candidates during minimization and only saves a minimized regression when the violation is reproduced again.

Building an actual developer workflow

Finding a vulnerability is only half the job.

We wanted the workflow to continue:

Find → prove → minimize → save → fix → test → PASS.

That meant building the CLI, project configuration, Hosted campaign history, regression artifacts, and replay system around the core engine.

What we learned

The biggest lesson was that agent safety needs tests, not more prompt text.

We learned that:

Deterministic oracles are much stronger than subjective graders when policies can be expressed over tool calls.
Real execution traces are more valuable than claims that an attack "looks successful."
A security finding becomes much more useful when it can be minimized and reproduced.
Every important exploit should become a permanent regression test.
The installation and onboarding experience is part of the product, especially for an open-source developer tool.
Developers and judges need to understand the core story quickly:

The policy said the agent must not refund more than $100. The agent issued a $250 refund. Here is the real tool-call evidence. Now let's make sure it never happens again.