Before vs after on the demo repo: Agent-Ready score 62 → 93, tasks solved 70% → 100%, with 95% intervals.
Every agent attempt, one square per step: with the repo's own docs vs with the AGENTS.md AgentReady wrote.
Replay of a real attempt: the agent follows the README, runs pytest (not installed), tries pip install and is blocked by the sandbox.
Root causes found by Nemotron 3 Ultra and Nano's label for every attempt.
The AGENTS.md written by Nemotron 3 Ultra, grounded in the test command AgentReady actually ran.
Verifier: an AI-written change with an invented package, eval and a bare except. Score 56, do not merge.
The same app on a phone: installable, paired with the PC by QR code.
Start page: a replay of a real run and the problems agents ran into.

Inspiration

AI coding agents now write a lot of our code, but in a real repository they waste most of their steps figuring out how the project works: which test command to run, which README still matches the code, where a module went. Teams write AGENTS.md or CLAUDE.md files to help, but nobody checks whether those files change anything. Benchmarks like SWE-bench compare models across many repositories. We wanted something that looks at your repository and shows, with numbers, what an instruction file changes there.

What it does

AgentReady plants small, real bugs in a copy of your code and keeps only the ones your own tests catch. NVIDIA Nemotron 3 Super writes a bug report for each one, like a user would, without saying where the bug is. Then a team of Nemotron Super agents tries to fix every bug in parallel, each in its own sandbox, and your tests decide whether they did.

Nemotron 3 Nano labels why every attempt struggled (lost in the code, couldn't run the tests, trusted an outdated README...). Tavily finds the official docs of the repo's tools, and Nemotron 3 Ultra turns all of it into an AGENTS.md, grounded in the test command AgentReady actually ran. Then the exact same bugs run again with the new file, and you get the score before and after, the success rate with a 95% interval, steps and tokens per solved task, and a replay of every agent step.

On the demo repo, the README told agents to run pytest (the project uses unittest), never mentioned that the code lives under src/, and pointed to files that no longer exist. With the AGENTS.md AgentReady wrote: tasks solved 7/10 → 10/10, Agent-Ready score 62 → 93, steps per attempt 19.1 → 13.2, tokens per solved task 82.6k → 47.6k.

There is also a Verifier for code an AI assistant wrote: paste code or a diff, with or without the repository. It checks for packages that don't exist on PyPI (assistants invent them), eval and shell=True, bare excepts and hard-coded keys, and with the repository it applies the change in a sandbox and runs the real tests. Nemotron reviews the diff on top.

It works on your PC and your phone (installable app): scan a QR code and the phone is signed in, and each device gets a live notice when the other starts or finishes a check.

How we built it

Defects are injected with Python's ast module, and each is checked against the real test suite, so every task can be graded without a human.
The agent under test is a small tool-using agent (list, read, search, edit, run, finish) driven by Nemotron 3 Super through Nebius Token Factory's OpenAI-compatible API.
Each Nemotron model does the job it suits: Nano labels every attempt (fast and cheap), Super does the multi-step tool use, Ultra makes the one or two calls that need the deepest reasoning.
Original test files are restored before grading, so editing a test never counts as a fix.
Attempts run in parallel locally, in network-less Docker containers, or as Nebius Serverless Jobs. The app is one container that can run as a Nebius Serverless Endpoint.
Only Python's standard library: it runs anywhere Python 3.9+ runs.

Challenges we ran into

Every task had to be gradeable without a human, so we only keep defects the tests catch. Agents sometimes "fix" a bug by editing the test, so tests are restored before grading. Agents are random, so each task runs several times and the report shows Wilson confidence intervals, and says so when they overlap. Nemotron sometimes writes code with raw newlines inside its JSON actions, so we built a lenient parser to not count that against the repository.

Accomplishments that we're proud of

A full loop that is honest with its own numbers: real bugs, real tests, a measured before/after, and a replay of every step.

What we learned

Most agent failures in a repo are not about intelligence but about orientation: wrong test command, missing PYTHONPATH, stale docs. A short, verified AGENTS.md fixes a lot of it.

What's next

Defect injectors for JavaScript, TypeScript and Go, a GitHub App that runs AgentReady on pull requests that touch docs, and a public table of results for popular open-source projects.