Docket : Evidence-First AI Repair Verification

Docket verifies AI-agent code repairs with independent evidence, not trust. It reproduces the bug, reviews the fix adversarially, and lets recorded commands decide the final status.

Comment

Inspiration

Coding agents are good at editing files and reporting success. They are not good at proving it. Ask an agent to fix a bug and it will confidently tell you the bug is fixed. Often after running one narrow test, sometimes after running nothing at all. That gap between "the agent says it's fixed" and "the fix is actually verified" is where real incidents come from: duplicate charges that slip through, security checks that get quietly weakened, edge cases that were never re-tested after the "fix."

We wanted a tool that refuses to take the agent's word for it. Not a smarter agent, a skeptical judge that sits between the agent and "done," and that only ever says VERIFIED when it has recorded, independently-checkable proof: a reproduced failure, a reviewed diff, a second model arguing against the first, and command output that actually ran.

What it does

Docket is a local developer tool for verified software repair. You give it a bug report against a trusted repository. It:

  1. Inspects the repository and reproduces the bug for real : checking that the exit code, failing test name, and output pattern all match an expected failure signature, not just "something failed."
  2. Drafts an acceptance contract : objective, editable pass/fail assertions and stops for human approval before any repair agent touches code.
  3. Runs the repair agent (Codex) inside an isolated Git worktree, network disabled, commands restricted to an allowlist.
  4. Sends the patch to a separate adversarial reviewer model whose only job is to find reasons to block it.
  5. If blocked, allows exactly one controlled revision in the same worktree, never an open-ended retry loop. Then re-reviews and re-verifies from scratch.
  6. Runs the required verification commands for real and lets their pass/fail; not the agent, not the reviewer decide the final status: VERIFIED, REVISION_REQUIRED, or FAILED.
  7. Exports a repair receipt (JSON/Markdown/HTML) backed by a content-addressed evidence ledger: every artifact in the run; bug report, contract, each attempt, each finding, each verification result, the receipt itself is SHA-256 hashed and stored content-addressed, committing to a single "trace root" hash. If anyone tampers with a stored artifact afterward, ledger:verify catches exactly which object broke and why.

The interface treats a VERIFIED badge and a broken evidence trace as two separate facts shown side by side and never collapsed into one green checkmark.

How we built it

Next.js 16 (App Router) + React 19 + TypeScript, in an npm-workspaces monorepo split into small, single-responsibility packages: domain (the only place allowed to decide final status), event-log (append-only SQLite event stream via libsql/Drizzle), repository-inspector / reproduction, git-worktrees, sandbox (locked-down command execution), codex-adapter (the repair agent, real Codex SDK or a deterministic fixture agent for demos), adversarial-review, verifier, repair-receipt, orchestration (the state machine gluing it together), and ledger (canonical serialization, SHA-256 object store, event hash-chaining, trace verification) plus resources (an explicit provisioning lifecycle for worktrees/sandboxes).

Zod schemas validate everything crossing a boundary. Vitest covers units and integration paths; Playwright drives the full browser workflow end to end. Three execution modes let it run anywhere on the honesty spectrum: fixture (fully deterministic, no API keys, what CI and judges use), openrouter (real model calls on free-tier models), and real (paid GPT-5.6 + Codex).

Challenges we ran into

  • A silent connection-orphaning bug in the SQLite client. Wrapping an event-append in db.transaction() looked correct but quietly broke every :memory:-backed test, because the client handed the connection to the transaction object and never reclaimed it. Later calls opened a new, empty in-memory database. Traced it by reading the client's own source. Fixed with a single atomic correlated-subquery insert instead of a transaction wrapper.
  • A CSS grid trap that caused real horizontal-scroll bugs, not just an untested layout: a bare 1fr grid track (and, separately, an unconstrained flex item) defaults to min-width: auto, so a track's minimum size becomes the min-content size of its widest child, a 700px-wide table, or an unbroken file path in a dropdown, silently stretching the entire page. Confirmed with actual scrollWidth measurements (716px in a 390px viewport) before tracking down and fixing every bare 1fr in the stylesheet.
  • A stray local .env.local silently forced live OpenRouter mode for npm run dev, which looked like a flaky Playwright suite until we traced it to environment-variable precedence, not a code defect.
  • An event-ordering bug in the ledger integration: the terminal receipt.generated event was being appended after the trace manifest already captured the event head, so every later ledger:verify reported a spurious mismatch, a one-line fix once an end-to-end integration test caught it.
  • Running the real, paid model workflow honestly. A guarded live-evidence run against real GPT-5.6 and Codex hit a genuine Codex usage-quota wall mid-session. Rather than hide that, the failure manifest is preserved alongside two earlier real runs that completed but landed on REVISION_REQUIRED, not VERIFIED, the honest result of a lighter model combination against a deliberately tricky cross-user fixture.

Accomplishments that we're proud of

  • A two-attempt repair-revision cycle that mirrors how real repairs actually go: first attempt gets legitimately blocked by an independent reviewer for a real cross-user security gap, second attempt fixes it, fresh review clears it, fresh verification passes, all recorded, none of it staged.
  • A tamper-evident evidence ledger we can actually demonstrate, not just claim: corrupt one stored object's bytes on disk, re-run ledger:verify, and watch it correctly report INVALID with the exact broken hash.
  • A genuine fresh-clone proof : cloned the public repo into a clean directory and ran install → setup → full quality gate → Playwright → fixture demo, all green, with the full transcript preserved.
  • A forensic, deliberately unglamorous interface: no gradients, no celebratory language for a passing status, no hiding failures behind an accordion and it holds up down to a 320px mobile viewport.

What we learned

That "evidence-first" has to apply to the tool's own development, not just its output. The same discipline that makes Docket refuse to trust an agent's self-report is what caught our own bugs (the transaction leak, the CSS overflow, the event-ordering race) before they shipped. A receipt is only as trustworthy as the discipline behind writing it, and honestly labeling a REVISION_REQUIRED result instead of quietly discarding it turned out to be more convincing evidence than a cherry-picked VERIFIED run would have been.

What's next for Docket : Evidence-First AI Repair Verification

  • Wire @docket/resources' request/lease lifecycle into the orchestrator's actual worktree and sandbox creation, so resource leases get hashed into the evidence ledger like everything else.
  • Add ledger:backfill so repairs completed before the ledger existed can gain trace evidence retroactively.
  • Expand beyond the two trusted JavaScript/TypeScript fixtures toward broader language and repository support, with the same reproduction-first discipline.
  • A hosted, HTTP-native evidence export endpoint alongside the existing CLI, and a "Resources" panel in the UI once real lease data exists to show.

Built With

Share this project:

Updates