Gauntlet — Attack. Patch. Prove.

Gauntlet attacks AI-agent security boundaries, generates bounded code repairs, and independently verifies fixes through real execution, adversarial retesting, and security contracts.

Comment

Inspiration

As AI agents become more capable, they gain access to personal memories, external tools, and the ability to take actions on behalf of users. Each new capability introduces a new security boundary.

In September 2026, Google confirmed that a Gemini model had accessed three real companies' systems during a cybersecurity evaluation in May. The model mistakenly believed those systems were within its authorized testing scope. The incidents highlight how capable agents can cross operational boundaries even during controlled evaluations.

Source: Reuters, September 18, 2026

While exploring AI-agent security, I kept coming back to one question:

What happens after an AI-agent security vulnerability is discovered?

AI models can identify vulnerabilities and generate code fixes, but a proposed fix is not proof that the vulnerability has been eliminated.

A patch might block the original attack while leaving other attack variations exposed. It might prevent a security violation by disabling legitimate functionality. Or it might introduce another vulnerability.

I built Gauntlet — Attack. Patch. Prove. to explore how these problems can be addressed through bounded AI-generated repairs and independent execution-based verification.

My guiding principle is:

AI proposes. Execution proves. Boundaries contain.

What it does

Gauntlet is an AI-powered security testing and verified-remediation system for AI agents.

It follows a three-stage workflow:

1. Attack — Detect security boundary violations

Gauntlet executes adversarial scenarios against deliberately vulnerable AI agents and captures evidence showing how security boundaries are crossed.

I implemented three security scenarios:

P100 — Untrusted Data

Tests whether malicious instructions embedded in untrusted external data can redirect an AI agent or cause information leakage.

P300 — Effect Authority

Tests whether an AI agent performs actions beyond its authorized permissions, including unauthorized refund operations.

P400 — Personalization Provenance

Tests whether an AI agent improperly uses personal memory from earlier conversations, introduces unrelated personal context into a new request, or crosses user-context boundaries.

Rather than simply labeling outputs as unsafe, Gauntlet captures execution traces and evaluates observable behavior against defined security contracts.

2. Patch — Generate bounded code repairs

After identifying a violation, Gauntlet uses Kimi K2.7 Code through Nebius Token Factory to generate candidate source-code repairs.

However, I designed the repair process with explicit restrictions.

The model cannot freely rewrite the application. Candidate modifications must satisfy authorized source boundaries, structural validation, and syntax checks before they can proceed.

Accepted patches are applied in a disposable test workspace without modifying the original repository.

Invalid or unsafe repair candidates are rejected.

3. Prove — Independently verify the repair

This is the central capability of Gauntlet.

I don't assume an AI-generated patch is secure just because the model claims it fixed the problem.

Instead, Gauntlet executes the patched agent and independently evaluates whether it satisfies the defined security contract.

Verification includes:

  • Re-running adversarial interactions against the patched agent.
  • Testing additional attack variations and security boundary conditions.
  • Checking that the original vulnerability is blocked.
  • Confirming legitimate agent functionality is preserved.
  • Generating structured evidence and execution traces supporting the outcome.

Gauntlet reports distinct outcomes:

  • VERIFIED: The repair satisfies the required proof checks.
  • REJECTED: The proposed repair fails admission or validation requirements.
  • NOT_VERIFIED: The repair has not satisfied the complete verification requirements.

I also implemented recorded evidence replay so previous experiments can be inspected without requiring fresh model inference. Replayed evidence is explicitly labeled rather than presented as live execution.

Flagship demonstration: P400 personalization provenance

My primary demonstration focuses on an AI assistant that remembers personal information from earlier conversations.

A user asks for productivity-app names, but the vulnerable agent improperly includes unrelated personal memories in the model request.

Gauntlet identifies this as a personalization provenance violation.

I use NVIDIA Nemotron Super 120B through Nebius Token Factory for live attack and proof interactions, and Kimi K2.7 Code to generate a bounded repair proposal.

Gauntlet validates the candidate, applies an accepted patch in a disposable workspace, and runs the patched agent against the security contract.

The successful verified experiment demonstrates that the repaired agent blocks unauthorized personalization, wrong-person memory access, and poisoned persistent memory while preserving legitimate personalization.

P400 — Before and After Verification Results

The following table summarizes the demonstrated security and functionality checks from the successful P400 workflow.

Security check Before repair After repair
Unauthorized personalization VIOLATED BLOCKED
Wrong-person memory Not separately established in baseline BLOCKED
Poisoned persistent memory Not separately established in baseline BLOCKED
Legitimate personalization Not separately established in baseline PRESERVED

Final result: VERIFIED

The accepted repair passed Gauntlet's required security and functionality checks in the successful recorded experiment.

The original repository remained unchanged because the patch was applied and executed in a disposable test workspace.

These results represent the tested security contracts, not a claim that every possible adversarial scenario or vulnerability has been eliminated.

A repair that makes an agent secure by breaking everything is not a successful repair.

How I built it

I developed Gauntlet as a Python-based security verification system with an interactive browser console, deliberately vulnerable test agents, model integrations, and independent proof execution.

Technology stack

  • Python 3.12+ and FastAPI: Backend application and security workflow orchestration.
  • HTML, CSS, and JavaScript: Browser-based demonstration interface.
  • Nebius Token Factory: Hosted inference for attack, proof, and repair-generation interactions.
  • NVIDIA Nemotron Super 120B: Live attack execution and proof interactions.
  • Kimi K2.7 Code: AI-generated bounded source-code repairs.
  • Pydantic: Structured request and response models.
  • HTTPX: Asynchronous communication with hosted model APIs.
  • Pytest: Automated security-contract and application tests.
  • Docker: Containerized application deployment.
  • Render: Hosting the publicly accessible demonstration.
  • JSON evidence artifacts: Recorded attack traces, repair proposals, validation outcomes, and verification results.

System architecture

The frontend communicates with a FastAPI backend that orchestrates three major stages: Attack, Patch, and Prove.

During an attack, Gauntlet executes an adversarial scenario and evaluates the resulting agent behavior.

When a violation is detected, the repair stage requests a bounded source-code modification from Kimi K2.7 Code.

I implemented validation controls to constrain what the generated patch can modify.

Accepted repairs are applied to a temporary copy of the relevant code, where the patched agent can be executed and tested.

During proof, Gauntlet re-executes the relevant scenarios and evaluates deterministic security and functionality checks.

The results are presented through the web console, along with execution evidence.

The critical architectural decision is that AI models propose attacks and repairs, but Gauntlet's independent validation and execution logic determines the final security verdict.

Challenges I ran into

1. Reproducing genuine security violations

One of my biggest challenges was distinguishing an actual security violation from an AI-generated explanation of a possible vulnerability.

A model saying that an attack succeeded is not sufficient evidence.

I built deliberately vulnerable agents, security scenarios, and observable execution traces so Gauntlet could identify concrete contract violations.

2. Constraining AI-generated source-code changes

Allowing a language model to modify arbitrary application code would create significant security risks.

I had to design bounded repair rules that restrict edits to authorized locations and reject candidates that do not satisfy validation requirements.

This introduced an important distinction between generating a plausible patch and admitting a patch for execution.

3. Preserving legitimate functionality

Another major challenge was ensuring that a security repair did not simply disable the affected feature.

For example, a personalization vulnerability could be prevented by disabling all memory retrieval.

But that would also eliminate useful, authorized personalization.

I implemented proof checks that test both unauthorized personalization and legitimate personalization.

4. Managing nondeterministic model behavior

AI-generated patches are not always valid or effective.

Some candidates fail structural validation. Others may pass initial validation but fail broader security verification.

Instead of hiding these outcomes, I designed Gauntlet to report rejected and unverified repairs explicitly.

This was essential to making the results trustworthy.

5. Combining live AI execution with reproducible evidence

Live model calls demonstrate that the system is actually interacting with AI models.

However, live inference can introduce variability, latency, and inconsistent repair candidates.

I addressed this by separating live execution from recorded evidence replay and implementing deterministic verification checks.

This allows Gauntlet to demonstrate real model behavior while preserving inspectable evidence from previous executions.

Accomplishments that I'm proud of

As a solo developer, I'm particularly proud of building an end-to-end system that goes beyond identifying security vulnerabilities.

My key accomplishments include:

  • Built the complete Attack → Patch → Prove workflow.
  • Integrated NVIDIA Nemotron Super 120B and Kimi K2.7 Code through Nebius Token Factory.
  • Implemented three AI-agent security scenarios: P100, P300, and P400.
  • Demonstrated a successful P400 repair with independent security and functionality verification.
  • Developed bounded source-code repair validation to restrict AI-generated changes.
  • Executed accepted repairs in disposable test workspaces without modifying the original repository.
  • Implemented adversarial retesting, security-contract checks, and utility-preservation tests.
  • Preserved structured execution evidence and recorded replay capabilities.
  • Built and deployed an interactive browser-based demonstration.
  • Created an automated test suite covering the core security workflows.

The accomplishment I'm most proud of is showing that an AI-generated security patch can be judged by independently executed tests rather than by the confidence of the model that generated it.

What I learned

Building Gauntlet reinforced an important engineering principle:

Finding a vulnerability, generating a repair, and proving that the repair works are fundamentally different problems.

I learned that language models are useful for exploring adversarial scenarios and generating candidate solutions, but their outputs should not be treated as proof of correctness.

I also learned how important it is to separate probabilistic AI behavior from deterministic security enforcement.

Another major lesson was that security and functionality must be evaluated together.

A successful repair must block unauthorized behavior without unnecessarily breaking legitimate use cases.

Finally, I learned that failed repair attempts are valuable evidence.

If Gauntlet rejects a candidate or cannot verify a repair, that is not a failure of the verification system. It demonstrates that the system is enforcing its requirements rather than blindly trusting model-generated code.

What's next for Gauntlet

This hackathon project establishes the core workflow of attacking vulnerable AI agents, generating bounded repairs, and independently verifying those repairs.

My next goal is to evolve Gauntlet from a controlled security demonstration into a developer-focused tool that works with existing AI-agent applications.

I plan to explore:

  • Integration with existing agent frameworks: Supporting real-world agent applications through normalized execution traces and framework-independent interfaces.
  • Automated security boundary detection: Expanding detection of unauthorized memory access, untrusted instructions, and unauthorized tool effects.
  • Developer tooling: Integrating attack reproduction, bounded repair generation, and verification evidence into local development and CI workflows.
  • Stronger execution isolation: Moving beyond disposable test workspaces toward hardened sandbox environments.
  • Expanded security contracts: Supporting additional classes of AI-agent security vulnerabilities and more complex workflows.

My long-term vision is to make independently verified AI-agent security remediation a standard part of developing and deploying agentic systems.

Gauntlet — Attack. Patch. Prove.

AI proposes. Execution proves. Boundaries contain.

Built With

Share this project:

Updates

Submission history