Roveproof

Inspiration

Roveproof was built in Indonesia to test software for the realities of the world. Most test matrices run on fast devices, reliable networks, and English-centric identity and address formats — so the failures that hit low-end phones, constrained mobile networks, mononyms, +62 phone numbers, Indonesian addresses, IDR, and Jakarta date/time semantics stay invisible right up until real users hit them in production. We wanted CI that actually represents those users, and we wanted the AI-generated fix to be provably safe rather than merely plausible.

What it does

Roveproof is an evidence-to-decision pipeline where each stage is bound to the previous by explicit IDs and hashes — no "latest", fixture, or implicit fallback in the real path:

1. Constrained journey CI. Runs a pinned Chromium indonesia-mobile-v1 profile (360×800 viewport, id-ID locale, Asia/Jakarta timezone, 4× CPU slowdown, constrained-3G network schedule) against a synthetic checkout target and writes an immutable, hashed evidence bundle (screenshots, trace, HAR, console, requests, assertions, timing, encoded bytes) under an external trust anchor. The baseline is a measured single-run observation — the anchored checkout failed · 8.2 MB · 19 s, reproducing three seeded defects: a required-last-name mononym rejection, a +62 phone normalization bug, and an oversized mobile checkout bundle.
2. Subscription-backed Codex diagnosis. One ephemeral, read-only, shell-disabled codex exec call returns schema-validated hypotheses, each cited to specific evidence artifacts, plus falsifiers and stated uncertainty. Captured pages and repository text are untrusted data, never instructions.
3. Failing-test-first bounded repair. A separate read-only Codex call authors a narrow regression test that must fail on the untouched baseline for the intended reason (proven in a disposable container). Only then does a second read-only call author one bounded source patch. Both are typed unified diffs, applied and run only inside an isolated sandbox.
4. Independent hash-bound verified approval. A fresh verifier workspace — distinct from the author sandbox, making no model call — reapplies the exact combined diff, re-verifies its hash, and reruns the original journey/profile/oracle. It requires one durable order, no seed failures, and measured budgets (≤2.0 MB, ≤8.0 s) before a candidate becomes review-ready: checkout succeeded · 1.4 MB · 6 s. A human then approves or rejects the exact candidateId + combinedDiffHash.

How I used Codex and GPT-5.6

Roveproof drives OpenAI Codex through the local Codex CLI, authenticated with Sign in with ChatGPT — subscription-backed model access, not Platform API billing. The model behind those Codex sessions is GPT-5.6. [CONFIRM the exact model your session ran — see the note at the bottom of this sheet; the repo deliberately records model: null and does not pin --model.]

Across one repair cycle, Codex/GPT-5.6 is invoked three separate times, each as an ephemeral, read-only, shell-disabled codex exec call:

• Diagnosis — returns schema-validated hypotheses, each cited to specific evidence artifacts (screenshots, trace, HAR, console, requests), plus falsifiers and stated uncertainty.
• Test-first authoring — authors a narrow regression test that must fail on the untouched baseline for the intended reason, proven inside a disposable Docker container before any source change.
• Bounded source authoring — only after the failing test is proven does a second call author one bounded source patch.

Codex credentials are never read, copied, logged, persisted, or mounted into evidence, candidate workspaces, commands, or sandboxes. (This project was itself built with Claude Code + Codex; the running product only calls Codex/GPT-5.6.)

How I built it

• Monorepo: TypeScript ^5.9 across npm workspaces (apps/*, packages/*).
• Control dashboard: Next.js 16 / React 19 (apps/control) — a loopback-only proof ledger.
• Constrained runner: Playwright 1.61 + pinned Chromium with the Indonesia Mobile device/network/locale profile (packages/journey).
• Typed contracts: Zod ^4 schema-validated envelopes for evidence, analysis, candidates, verification (packages/contracts).
• Sandbox isolation: Docker with a pinned node@sha256 image — --network none, read-only root, dropped capabilities, no-new-privileges, bounded PID/memory/CPU/time/output (packages/sandbox).
• Model access: local Codex CLI via Sign in with ChatGPT (packages/model-adapter).
• Orchestration + write-once, hash-bound provenance: packages/orchestrator, packages/store.

Challenges I ran into

The hard part is not calling a model — it is trusting its output. Generated code is a candidate, not a fix, so the whole system is built to distrust it: everything runs in a --network none container, one bounded candidate is capped at ≤5 files / ≤250 changed lines, an independent verifier re-runs the journey without any model call, and human approval binds to an exact diff hash. Keeping the pipeline fail-closed — uncertain or tampered evidence stays INCONCLUSIVE/REJECTED, never PASS — while still producing a clean before/after was the central engineering tension.

What I learned

Enforcement, not prompts, has to be the safety boundary. The moment a fix is proven by re-running the real journey under the real constraints — instead of asserted by the model — the claim becomes something a human can actually approve.

What's next

The MVP intentionally covers one seeded checkout, one Indonesia Mobile profile, and three deterministic defects. Next: more constrained profiles (devices, network classes, locales), more journeys beyond the single checkout, more seed-defect categories, and a gated path from approval (currently diff + rollback export only) toward real merge/deploy — without weakening the exact-hash-bound approval guarantee.

Built With

• chromium
• codex
• codex-cli
• docker
• gpt-5.6
• next.js
• node.js
• npm-workspaces
• openai
• playwright
• react
• typescript
• vitest
• zod