Inspiration
Teams can compare model prices and benchmark scores quickly, but migration risk lives in their own structured outputs, tool calls, errors, latency, and ambiguous behavior. A handful of hand-tested prompts does not reveal whether an application contract changed.
What it does
Testrig is an open-source migration review for AI model changes. It replays one representative corpus against a baseline and candidate, pairs executions by stable case identity, applies explicit deterministic checks, keeps free-form differences available for human review, and exports fingerprint-bound evidence.
The interface is a paired trace explorer: filter outcomes, expand a case, compare baseline and candidate duration lanes anchored at zero, inspect outputs/checks/inputs/reviews/evidence, export the verification artifact, or use the deliberate two-step deletion flow. Verdicts are server-authored and scoped to the exact corpus and configuration; Testrig does not claim semantic equivalence, migration safety, or production approval.
The public demo uses immutable synthetic fixtures and makes no provider calls. The separately configured self-hosted path adds zero-call capability preflight, explicit bounded billable confirmation, environment-only provider credentials, mounted corpora, strict request conformance, retained SQLite evidence, and verification-v2 export. One separately authorized three-case observation compared Claude Opus 4.8 with GPT-5.6 Sol; its evidence is presented as one retained observation, not a universal model ranking.
How we built it
Rust, Axum, Tokio, Serde, SQLx, and SQLite own request compilation, execution lifecycle, deterministic checks, reviews, verdicts, canonicalization, fingerprints, retention, and exports. Next.js, React, and TypeScript consume typed DTOs without reimplementing policy. Playwright covers the rendered desktop/mobile workflow, and Docker Compose provides the one-command synthetic demo.
Security and evidence boundaries are deliberate: missing usage or price remains null rather than zero; provider errors and model output are untrusted; secrets stay out of the browser, database, export, logs, fixtures, screenshots, and Git history; provider dispatch has no automatic retry; and unkeyed hashes are labeled integrity evidence rather than signatures.
How Codex and GPT-5.6 were used
A single long-running Codex Goal executed a binding phase ledger from contract lock through integration and release verification. One bounded frontend subagent was limited to apps/web; later correctness, privacy/security, and UX auditors were read-only. Codex Desktop Browser drove rendered QA, Playwright made those paths reproducible, and two official code-review rounds produced concrete findings that were verified, fixed, and regression-tested rather than turned into invented submission claims.
Codex implemented and checked the Rust/TypeScript system, adversarial fixtures, provider-specific request compilation, strict loopback conformance mocks, responsive trace explorer, evidence documentation, clean-checkout verifier, and recording workflow. GPT-5.6 Sol was also exercised as the candidate target in the one explicitly authorized retained provider comparison shown in the video. The repository preserves the execution ledger, build log, review dispositions, test evidence, and exact limitations.
Challenges and lessons
The hardest work was preserving authored verification semantics while compiling different provider request dialects. Anthropic structured output required a supported JSON Schema subset while local verification retained the complete authored contract; provider-specific tool history had to remain structurally valid; and request IDs, timing, usage, and unavailable cost needed distinct evidence semantics.
The other major lesson was that a migration result must stay inspectable. A single score hides failure modes. Stable case pairing, explicit adaptation labels, untimed checks/reviews, visible missing evidence, immutable configuration snapshots, and export fingerprints make the review auditable without overstating what the run proves.
What's next
Post-challenge work includes additional provider dialects, richer workload import ergonomics, multi-user access controls, encrypted retained payloads, and statistically powered performance courses. The current release intentionally avoids LLM judges, automatic routing recommendations, and broad compatibility claims.
Built With
- anthropic-api
- axum
- codex
- docker-compose
- gpt-5.6
- next.js
- openai-api
- playwright
- react
- rust
- sqlite
- sqlx
- tokio
- typescript