RogueChaos

RogueChaos cover image
Product homepage
Contained Chaos Lab
Redacted Blast Trace
Twelve repaired findings
Evidence-linked hardening
Verified 100/100 Resilience Card

Inspiration

What it does

Breaking AI agents before attackers do

AI agents can read private data, call tools, make decisions, and act on a user’s behalf. That power creates a new problem: traditional tests confirm that an agent completes the happy path, but they rarely reveal what happens when the agent encounters prompt injection, poisoned tool output, approval bypasses, or requests for another user’s data.

That inspired RogueChaos—an open-source, Codex-native chaos-engineering toolkit for AI agents. It safely attacks an agent inside a contained environment, records what happened, turns failures into actionable hardening work, and verifies that the fixes improve security without breaking legitimate behavior.

What RogueChaos does

RogueChaos provides a complete attack-to-verification workflow:

Scan the target’s capabilities and safety boundaries.
Attack it with versioned, declarative security recipes.
Capture a redacted Blast Trace showing what the agent observed, decided, and attempted.
Grade each result with deterministic security rules.
Harden the target using finding-linked fixes and regression tests.
Verify the repaired agent against the same attacks and benign controls.
Publish a transparent RogueScore and shareable Resilience Card.

The bundled demo includes a deliberately vulnerable customer-service agent with synthetic users, orders, tools, approvals, and secrets. It exercises twelve contained attack scenarios covering risks such as prompt injection, secret leakage, cross-user access, excessive agency, approval bypass, and poisoned tool output.

In the validated demo, the vulnerable baseline scores 0/100. After hardening, the comparable rerun scores 100/100, fixes all twelve seeded findings, and preserves every benign utility control. These results describe the synthetic target and evaluated recipe suite—not universal agent security.

How it was built

The security engine is an offline-first Python CLI. Attack recipes are defined in YAML and validated against a versioned JSON Schema before execution. Recipes cannot run arbitrary code or expand beyond the operator’s declared safety boundary.

Framework adapters translate native agent behavior into a shared event model containing observations, messages, policy decisions, tool requests, tool results, containment actions, and errors. This allows the same graders and reporting pipeline to work across the local demo and integrations such as LangGraph, Google ADK, and OpenAI Responses.

Potentially sensitive native events exist only in memory. RogueChaos normalizes and recursively redacts them before persistence, producing typed evidence markers that prove a leak occurred without storing the leaked value.

RogueChaos also includes:

Immutable, replayable run artifacts
Terminal, JSON, Markdown, HTML, and JUnit reports
A Codex skill for the diagnosis-to-repair workflow
Finding-linked hardening briefs and regression fixtures
Comparable baseline-versus-verification reports
Privacy-preserving Resilience Cards
An optional Vertex AI Gemini remediation advisory
An interactive browser-based Chaos Lab

The optional AI advisory receives only privacy-minimized finding and score metadata. It never receives raw Blast Trace content and never changes the deterministic RogueScore.

Challenges faced

The hardest challenge was producing useful evidence without creating another security risk. Saving raw agent traces would make debugging easier, but those traces can contain credentials, personal data, or sensitive tool payloads. Redacting data before it reaches disk—while preserving enough typed evidence for objective grading—required a strict trust boundary throughout the architecture.

Another challenge was distinguishing a secure result from an incomplete one. A crashed or blocked attack must not be counted as a successful defense. RogueChaos therefore models pass, fail, blocked, error, and inconclusive outcomes separately and penalizes incomplete evidence.

Hardening introduced a different risk: an agent can become “secure” by refusing everything. To prevent that, verification reruns both the attacks and benign controls. A security improvement is not marked verified if legitimate behavior regresses.

Portability was also difficult because agent frameworks expose very different event formats. A framework-neutral event contract kept security logic out of individual adapters and made integration support measurable through shared contract tests.

Finally, combining a deterministic offline security gate with AI-generated remediation required a careful separation of responsibilities. Rule-based graders remain authoritative and reproducible, while Vertex AI acts only as an optional second opinion.

What was learned

The biggest lesson was that agent security needs more than a vulnerability score. Developers need a replayable chain connecting an attack recipe, agent behavior, policy decision, objective grader, remediation, regression test, and comparable rerun.

It also became clear that security and utility must remain separate measurements. Combining them into one number can hide an agent that is safe but useless—or useful but dangerously permissive.

Another important lesson was that containment and privacy cannot be added after execution. They must shape recipe validation, adapters, event handling, artifact storage, reporting, and model integrations from the beginning.

Most importantly, chaos engineering is a strong fit for agent development. Instead of waiting for unpredictable failures in production, teams can deliberately create contained failures, study the blast radius, repair the weakness, and prove that the agent is more resilient.

Try the RogueChaos Chaos Lab or explore the product site.