Ratchet - regression CI for self-improving agents

A recorded BLOCK verdict on the landing page, with the cascade sieve
github-triage blocked: 7 of 14 replays would close issues
FX converter passed: the world moved, not the agent
The drift card: old value, new value, Tavily source, ruling

Inspiration

Self-improving agents (Hermes, OpenClaw) rewrite their own skills while you sleep. Nothing checks whether the "improved" skill still does yesterday's job. We wanted the equivalent of CI for that loop: not a test suite someone has to write, but the user's own past successes, replayed.

What it does

Record. A capture proxy in front of Token Factory records every model and tool call. You bless a run you were happy with; it becomes a golden trace, and Nano extracts the world facts it relied on and the success criteria it must keep meeting.
Detect. The skills folder is a git repo. Agent writes land on candidate; main only changes through Ratchet.
Replay. The goldens that use the edited skill run again in fresh sandboxes, on the same model tier, with side effects served from cassettes.
Judge. Level 0 is a model-free diff of side effects. Then Nano screens every pair, Super checks the contested ones with cited steps, Ultra adjudicates what is left. Tavily tells "the world moved" apart from "the agent regressed".
Decide. PASS merges, WARN holds, BLOCK reverts. Every action is a git commit you can undo, and Ultra writes the one-paragraph verdict.

The dashboard shows where each task was decided, what each tier cost against an all-Ultra counterfactual, and the evidence behind every verdict.

Results

Three seeded scenarios on live Nemotron and live Tavily: 6 of 6 correct, 41–72 s from skill edit to verdict, uncached.
SkillRegress-Bench pilot (35 labeled edits, real models): detection 0.80, zero false blocks on cosmetic and world-drift edits, drift recognised 2/2, $0.038 per verdict, about half the all-Ultra cost.
Serverless Jobs verified end to end: a 14-replay scenario across five jobs, BLOCK 7/14.

How we built it

Python control plane (FastAPI, SQLite, an OpenAI-compatible proxy), a React dashboard, a locked-down Docker sandbox for replays, Nebius Serverless Jobs for fan-out, a Nebius VM behind a Cloudflare Tunnel for the public demo. 1,300 unit and integration tests, Playwright for the UI, a live verification log (docs/VERIFIED.md) for every claim about the APIs.

Challenges

Nondeterminism. Token Factory does not honor seed, so an unchanged skill still produces different replays. We judge meaning, not bytes, and replay contested pairs once more with the unchanged skill as a control. The benchmark measures that noise floor (0.99 pass rate on 122 unchanged pairs).
Drift is not regression. An FX converter whose outputs changed because EUR/USD moved must pass. The replay's own tool call supplies the candidate new value and Tavily has to corroborate it.
Cost discipline. Every judge call is tagged by tier and priced. The cascade exists so Ultra only sees what Nano and Super could not settle.
Serverless Jobs plumbing. File injection is capped at 64 KiB, the bucket mount ignores sourcePath, refuses chmod and turns renames into empty objects, and cold start is 3–4 minutes. Each one cost a run to discover and a small fix to the worker.

What we learned

Agreement rules beat any single judge: Nano alone blocks every cosmetic edit, Ultra alone is too expensive to run on every pair, and the cascade with "a BLOCK needs Level 0 or Super and Ultra agreeing" is what gets false blocks to zero. The open problem is the other side: 7 of 8 cosmetic edits are still held for a human because Ultra will not clear a differing replay against a diff that changed nothing.