Aegis - An Immune System for AI Agents

AI agents can fail in ways their builders never anticipated. Aegis doesn’t merely detect those failures, it attacks, repairs, proves, and remembers them.

Comment

Inspiration

AI agents are starting to take real actions: issuing refunds, handling customer requests, running internal workflows, and making decisions that affect people and businesses. But most agent safety still feels static. We write a better prompt, add a few tests, and hope the system keeps behaving.

Aegis was inspired by a different idea: what if AI agents had an immune system?

In biology, immunity is not just detection. It is attack, evidence, adaptation, memory, and future resistance. We wanted to bring that loop into agent development. When an AI agent fails, Aegis does not just warn the developer. It reproduces the failure, verifies the policy violation, asks GPT-5.6 and Codex to build competing repairs, evaluates those repairs against protected attacks, and saves the fixed exploit as permanent regression immunity.

What it does

Aegis is an immune system for AI agents.

The MVP focuses on a deliberately vulnerable refund-support agent. Aegis attacks the agent with realistic adversarial scenarios such as prompt injection, privacy extraction, refund splitting, excessive discounts, and social pressure. Every attempted action is graded by a deterministic policy engine, even if a downstream tool would have blocked it.

Once Aegis finds verified failures, GPT-5.6 diagnoses the weaknesses and generates three repair strategies. Codex then implements three competing candidate fixes in isolated Git worktrees:

an instruction-focused repair

a permission/tool-focused repair

an orchestration/policy-focused repair

Aegis freezes candidate commits before creating protected holdout attacks, so the final tournament is not just testing against visible examples. The winning candidate must improve protected performance, pass regression tests, have zero hard-policy violations, and go through a human approval gate before promotion.

After approval, Aegis re-attacks the agent with the original exploit and stores the vulnerability as an immunity record that future versions must continue to pass.

How we built it

Aegis is built as a TypeScript workspace with a React/Vite interface, a Fastify API, a deterministic evaluation engine, and a live orchestration package.

The frontend is the Immune Arena: a real-time visual command center where attacks, violations, diagnoses, repairs, holdouts, and immunity records appear as the run progresses. The UI is driven by typed Server-Sent Events, so animations reflect actual run events instead of decorative fake progress.

The backend includes: a run state machine for the full attack-to-immunity loop

an append-only JSONL event log and materialized snapshots

a deterministic refund-policy evaluator

scoring and promotion gates

replay mode for public judging

live mode for authenticated GPT-5.6 and Codex execution

isolated Git worktrees for candidate repairs

human approval, rollback, and immunity verification

Replay mode is credential-free so judges can inspect the full product reliably. Live mode uses genuine OpenAI Responses API calls and Codex worktree mutations, with protected holdouts kept outside the candidate worktrees.

Challenges we ran into

The hardest challenge was making the project both visionary and honest.

It would have been easy to show a cinematic replay and imply everything was live. Instead, we separated the evidence modes clearly. Public replay is deterministic and credential-free. Live mode is authenticated and performs real GPT-5.6 and Codex execution. The README, UI, and submission all distinguish those paths.

Another challenge was protecting the evaluation boundary. Codex should be allowed to repair the vulnerable agent, but it should not see the protected holdouts or modify the evaluator. We built the system so Codex receives only development failures, a mutation spec, and allowed target files. Candidate commits are frozen before protected attacks are created.

We also had to make agent safety understandable in under three minutes. That shaped the Immune Arena design: instead of showing raw logs only, Aegis turns each step into inspectable evidence: the attack, the unsafe attempted action, the violated policy, the candidate diff, the protected score, and the final re-attack.

Accomplishments that we're proud of

Aegis demonstrates a complete vertical slice:

discovers agent failures beyond seed tests

verifies violations deterministically

uses GPT-5.6 for diagnosis and mutation planning

uses Codex to build three competing repairs

freezes candidate commits before protected evaluation

compares baseline and repaired agents on holdout attacks

requires human approval before promotion

re-attacks the original exploit after promotion

stores the vulnerability as permanent immunity

In the reference evaluation, the protected baseline score improves from 49.0 to 98.5, a +49.5 percentage point gain, with zero hard-policy violations for the winning candidate.

What we learned

We learned that agent safety is strongest when it becomes an engineering loop, not a one-time checklist.

The important insight was that the model is not the only thing that matters. The evidence boundary matters. The evaluator matters. The ordering of candidate freeze before holdout generation matters. Human approval matters. Regression memory matters.

We also learned that AI repair systems need humility. Aegis does not claim perfect safety or model-weight training. It evolves prompts, tools, permissions, policy enforcement, and orchestration code. If no candidate passes the hard gates, Aegis honestly recommends “do not promote.”

What's next for Aegis - An Immune System for AI Agents

The refund-support agent is only the first target. The same pattern can extend to any AI agent with tools, policies, and observable actions: finance operations, healthcare admin workflows, customer support, developer automation, and internal enterprise agents.

Next, we would expand Aegis into a general framework where teams can define their own agent constitution, connect their own tool actions, run adversarial evolution, and build a growing immunity library across agent versions.

Built With

Share this project:

Updates

Submission history