TraceForge

Modernize undocumented workflows without guessing.

Comment

Inspiration

Critical workflows often survive because nobody is confident enough to replace them. Their real rules are scattered across UI behavior, API responses, database side effects, and operator memory. Generating a cleaner interface is easy; proving that refunds, approvals, inventory, and failure atomicity still behave correctly is the hard part.

TraceForge is built around that missing proof. Instead of asking an AI to guess a specification from a few screenshots, it treats migration as an experiment: observe behavior, expose uncertainty, choose the next counterexample, rebuild inside a narrow evidence boundary, and let an independent verifier decide what matches.

What it does

TraceForge runs one bounded legacy workflow through five server-owned stages:

  1. Observe records legacy decisions and SQLite before/after state.
  2. Infer asks a read-only GPT-5.6 Sol Trace Archaeologist for narrow, evidence-linked hypotheses, invariants, and explicit unknowns.
  3. Challenge asks Counterexample Hunter turns for discriminating inputs. The host—not the model—executes those inputs and exact boundary probes, then a Contract Critic resolves the behavior contract.
  4. Build gives Codex the contract, four failed proofs, and only disclosed scenarios. Codex can edit one workflow file in a detached worktree; it cannot change the verifier, commit, push, or deploy.
  5. Verify creates one verification-only input after the writing turn, resets both systems, and compares five deterministic fields across seven scenarios.

The demo begins with a $45 standard damaged return and a $120 VIP damaged return. Crossed and boundary inputs reveal an undocumented priority rule: damaged returns at or above $500 enter manual review before tier-specific behavior. A zero-stock probe exposes another hidden requirement: a replacement must fail atomically with no return record or inventory side effects. Codex repairs rule priority, inventory disposition, and stockout atomicity. The host finishes at 7/7 scenarios, 35/35 assertions, and zero mismatches.

The contract is intentionally narrow. It covers the observed DAMAGED branch. Non-damaged inputs are rejected with a typed OUTSIDE_EVIDENCE_BOUNDARY result before persistence or side effects, so the generated candidate cannot silently invent unsupported behavior.

How we built it

  • gpt-5.6-sol performs four read-only, schema-constrained archaeology turns: Trace Archaeologist, two Counterexample Hunter turns, and Contract Critic.
  • The host validates every cited evidence ID and executes every proposed scenario itself.
  • OpenAI Codex SDK performs the single code-writing role in an isolated detached Git worktree.
  • The repair input is immutable and contains the exact contract, four failed proofs, and disclosed scenario pack.
  • A one-file allowlist prevents the writer from editing the verifier, tests, repository history, or deployment.
  • TypeScript, Node.js, Express, node:sqlite, React, Vite, pnpm, native Server-Sent Events, Git worktrees, and JSON Schema form the product runtime.
  • Deterministic assertions compare decision, return status, refund amount, sellable inventory, quarantine inventory, and failure atomicity.
  • SHA-256 digests bind model inputs and outputs, repair inputs, candidate source and diff, commands, artifacts, evidence, scenario sets, and proof bodies.

Why GPT-5.6 and Codex are essential

GPT-5.6 is not a chat layer. In the authenticated source migration it completed four evidence-producing turns totaling 121,673 tokens. Each turn preserves its bounded application prompt, structured output, thread ID, timing, usage, digests, and cited evidence IDs. Its most important job is not summarization; it is choosing the next observation that separates plausible hidden rules.

Codex receives the resolved contract and the real failed proofs. It changed one permitted TypeScript workflow module. The system then ended the writing turn and handed control to a host-owned verifier. The writer never sees the verification-only scenario while editing and cannot issue its own passing proof.

Challenges we ran into

The hardest problem was separation of powers. A model that proposes a rule or writes code cannot also execute the legacy workflow or certify its own result. The host therefore owns scenario execution, evidence-ID validation, state resets, policy enforcement, tests, and proof issuance.

We also learned that provenance has two different meanings. The original successful source proof must remain byte-for-byte reproducible, while the current application needs to prove which scenario set and recorded verifier artifact it is replaying today. TraceForge now preserves the historical object and wraps it in a separately verifiable v2 evidence envelope that binds the current scenario set and generated verifier artifact.

The Local Runner exposed useful failures too: revoked authentication, usage limits, unsafe candidate patterns, and an older verifier fixture that did not reject out-of-contract inputs. We kept the policy fail-closed and added a host-owned SELLABLE boundary probe. A local proof can be issued only when the candidate rejects that input before any result or side effect.

Accomplishments we are proud of

  • Four real GPT-5.6 turns produce inspectable evidence and explicit uncertainty rather than a predetermined contract behind a chat UI.
  • The unknown lifecycle is enforced: Build cannot begin while an in-scope blocking unknown remains.
  • Codex receives the real contract and failed proofs but cannot edit or invoke the verifier.
  • The source candidate passed 56/56 candidate-safe tests; four replay-integrity guards stay separate and visible.
  • The final differential matrix passed 7/7 scenarios and 35/35 assertions with zero mismatches.
  • Historical proof integrity and current replay binding can be checked independently.
  • The public replay is honest: recorded GPT/Codex provenance plus a fresh host proof, not a disguised live model call.
  • Judges can optionally run a fresh build with their own Codex through a pinned, loopback-only Local Runner. No credentials, generated source, session handles, or local files leave their machine.

What we learned

The trustworthy output of AI-assisted modernization is not code alone. It is code plus evidence provenance, preserved failures, an explicit evidence boundary, a verifier outside the writer's control, and a precise statement of what was covered. The model's most valuable behavior was choosing the next experiment; the system's most valuable behavior was refusing to let the writer grade itself.

What's next

Next we will add browser-extension capture for real web applications, redacted evidence envelopes, separate legacy and candidate services, signed append-only proof bundles, more database adapters, and maintainer-approved pull-request publication. Deterministic assertions and human-controlled promotion will remain the authority for proof and deployment.

Scope and limitations

This submission demonstrates one controlled Web returns workflow in a TypeScript process with REST and SQLite. It does not claim universal behavioral equivalence or arbitrary-application reconstruction. The proof claims conformance only for the seven executed scenarios and asserted fields. The public deployment does not expose an anonymous write-capable model trigger; the optional Local Runner is the explicit path for a fresh Codex build.

Try it

Built With

Share this project:

Updates