Inspiration
I have a background in Infrastructure engineering, having designed chaos testing and resiliency benchmarks for regular cloud applications. Also while using AI I keep hitting the “you’ve reached your limit” wall when things go astray. So that made me realise that it happens a lot with agents too, where it burns tokens on things which are outside our control - such as real world infra problems and failures.
What it does
ResilienceGuard runs fire drills on AI agents, then has Nemotron fix them and proves the fix with the same drills.
Break: paste a GitHub link to a Python agent that calls Token Factory, or pick a cookbook agent. ResilienceGuard runs six drills inside a Token Factory Sandboxes microVM: a normal run, a brief outage, a rate-limit burst, a garbled tool call, a crash right after the agent acts followed by a rerun, and a sustained outage. A proxy injects the faults into real Token Factory traffic, so the agent’s code is never changed.
Report: you get a grade, a pass or fail per drill with the reason, and the exact duplicate the agent produced. It detects duplicates by what the agent actually executed, not by comparing wording.
Fix: Nemotron 3 Super reads the code, its input files and the audit evidence, then writes two candidate fixes in parallel.
Prove: each candidate is re-audited in its own microVM, branched from a single checkpoint of the agent. Fixes that break the agent go back to Nemotron with their error output, for up to three repair rounds. The best fix wins only on measured results.
How we built it
- Created a Fault-injection proxy - an OpenAI-compatible proxy in front of Token Factory. It injects 500s, 429s, truncated tool-call arguments, dropped streams and crashes, including conditional faults like “fail right after the agent calls this tool”.
Challenges we ran into
- “Exit code 0” isn’t success. Some agents swallowed errors and exited cleanly with no output, so I count those as silent failures.
- Sandboxes access arrived mid-hackathon, so I first built the full loop locally, then moved every audit into microVMs once Nebius enabled it.
Accomplishments that we're proud of
- Real findings on real, unmodified agents from Nebius’s own cookbook, not toy examples.
- A hosted demo where anyone can audit their own agent from a GitHub link, with fetched code isolated in microVMs.
What we learned
*Most agents are tested only on the happy path. Provider failures are normal, and a few seconds of trouble is enough to crash a production agent. *Retries alone aren’t enough. Retrying an agent that already acted creates exactly the duplicate you were trying to avoid, so you need retries plus idempotency.
What's next for ResilienceGuard
- A GitHub Action that runs the drills on every pull request and fails the build when an agent regresses.
- Opening the fix as a pull request instead of showing a diff.
Log in or sign up for Devpost to join the conversation.