trace2test

Turn AI agent traces into deterministic regression tests, diagnose failures with GPT-5.6, and verify fixes in a sandbox before CI lets them ship.

Comment

Inspiration

AI agents do not fail like ordinary software. A prompt edit, model upgrade, or tool-schema change can leave every unit test green while silently changing which tools the agent calls, in what order, and under which safety conditions. Teams already have execution traces, but those traces usually remain passive observability data.

I built trace2test to turn every known-good agent trace into an executable behavioral contract.

What it does

trace2test provides a complete developer workflow:

  1. t2t import normalizes OpenAI Agents SDK or generic JSONL traces.
  2. t2t generate uses GPT-5.6 to propose an editable, schema-validated YAML regression suite from a known-good trace.
  3. t2t run replays that suite deterministically and fully offline, checking tool order, arguments, outputs, and safety policies. It emits terminal, JSON, JUnit, and Markdown reports and fails CI on regressions.
  4. t2t diagnose computes a deterministic structural diff, then asks GPT-5.6 to explain the root cause and propose an exact patch.
  5. t2t fix --verify applies the patch only inside a sandbox copy, reruns the agent, and replays the failing tests. It prints VERIFIED only when the fresh trace passes.
  6. t2t ui provides run history, assertion detail, trace diffs, failure clusters, and diagnosis/patch views.
  7. A composite GitHub Action brings the same regression gate to pull requests.

A real GPT-5.6 agent, really regressing

The featured support-agent example is not a scripted mock. GPT-5.6 chose every tool call at runtime through the Codex CLI provider while local functions implemented the tools.

With the safe prompt, the real run called:

lookup_subscription → check_refund_policy → cancel_subscription → escalate_to_human

I then weakened one instruction to permit fast cancellation. The next genuine GPT-5.6 run changed to:

lookup_subscription → cancel_subscription → check_refund_policy → escalate_to_human

trace2test caught both the exact sequence regression and the policy violation: cancel_subscription had no prior successful check_refund_policy. The committed traces let judges replay this evidence entirely offline.

How I built it with Codex and GPT-5.6

The core implementation was produced in one primary OpenAI Codex CLI thread running GPT-5.6 (gpt-5.6-sol). I defined the product thesis and architecture, split the work into verified milestones, reviewed each commit, ran the quality gates, and fed real smoke-test failures back into the same Codex thread.

Codex accelerated the monorepo scaffold, trace schemas and normalizers, assertion engine, CLI workflows, dashboard, GitHub Action, tests, and documentation. The product itself also uses GPT-5.6 through either the OpenAI API or an authenticated Codex CLI fallback, so builders can use its LLM features without managing a separate API key.

Challenges

The hardest problem was making probabilistic generation safe for a deterministic testing tool. The first real Codex-fallback smoke tests uncovered three strict structured-output bugs that unit mocks missed:

  • the model produced the wrong test-spec shape;
  • OpenAI rejected non-top-level JSON Schema references;
  • free-form Zod values produced typeless nodes forbidden by strict mode.

I used those failures as regression cases. Codex then added exact validation-error retries, normalized all references into top-level definitions, encoded free-form values safely, added exhaustive strict-schema tests, and kept a graceful schemaless fallback. Only after the live loop passed did I move on.

The second challenge was verifying fixes safely. trace2test never patches the working tree directly: it copies the target to an ignored sandbox, applies the proposed exact replacement there, regenerates a trace, and reruns the failing suite.

Accomplishments

  • 86 automated tests with typecheck, lint, build, demo, and CI all green.
  • Genuine safe and regressed GPT-5.6 episodes committed as offline fixtures.
  • Complete product loop, not just a proof of concept.
  • Deterministic replay needs no LLM, network, or paid service.
  • Interactive judge dashboard deploys as a static GitHub Pages snapshot.
  • Public MIT-licensed repository with setup instructions, architecture diagram, sample data, build log, and GitHub Action.

What I learned

The strongest agent tooling is hybrid. GPT-5.6 is excellent at extracting intent from traces and explaining divergences, while deterministic code should own validation, replay, CI exit codes, and proof that a fix actually works. The result is more useful than either an LLM-only evaluator or a conventional unit-test harness alone.

What's next

Next I would add more first-party adapters, richer semantic assertions, npm publishing, authenticated team dashboards, flaky-regression clustering across many runs, and pull-request annotations that link directly to the first divergent agent step.

Run it

Built With

Share this project:

Updates