Evidence-to-test

Inspiration

Browser recorders preserve clicks, but they rarely prove what those clicks meant. On brittle legacy UIs, they lose popup context, keep volatile selectors, and may confuse a successful HTTP response with a successful business outcome.

Passing the same noisy recording to a coding agent only moves the guessing downstream. We built Evidence-to-Test to create a verifiable boundary between observing a browser workflow and turning it into reliable automation.

What it does

Evidence-to-Test captures a fresh Chromium walkthrough and compiles it into a promoted evidence contract.

Every action carries its page or popup context, before-and-after state, a proved locator, required waits, correlated outcomes, integrity hashes, and explicit unknowns. Incomplete, contradictory, unsafe, or modified evidence is rejected or marked Needs human review.

The demo runs entirely against a local Synthetic Legacy Forms Portal. Its canonical workflow spans 13 actions across six views and deliberately includes repeated controls, changing DOM IDs, delayed state transitions, validation and recovery, and GET/POST responses that share both a pathname and HTTP 200 status.

After promotion, Codex receives only the bounded contract and produces the Playwright scenario. A deterministic verifier then checks contract compatibility, ordered action IDs, the scenario SHA-256, and an actual fresh replay.

A separate tamper demo modifies an isolated artifact and produces a real OUTPUT_MISMATCH rejection while leaving the original evidence unchanged.

The P0 intentionally proves this workflow on one synthetic profile rather than claiming support for arbitrary websites.

How we built it

Evidence-to-Test is written in TypeScript and Node.js using Playwright, Chromium, and Zod.

The pipeline is explicit: fresh capture → immutable snapshot → evidence analysis → safety and promotion gates → bounded contract → Playwright scenario → fresh replay

Browser events, snapshots, and trace data are journaled before analysis. Source artifacts are copied into an immutable SHA-256 inventory. Deterministic analyzers establish action boundaries, locator uniqueness, state changes, request correlation, and business outcomes.

Only a complete evidence set can produce the self-contained HTML report and bounded Codex handoff. The resulting scenario is independently checked and replayed without requiring credentials, an external portal, a runtime API call, or an AI account.

An inherited recorder foundation existed before Build Week. During the event, we hardened its isolation and integrity boundaries and built the standalone product around it: the synthetic portal, fresh-capture workflow, evidence report, Codex handoff, scenario provenance, replay verification, tamper demonstration, and judge documentation.

Challenges we ran into

• Combining physical and semantic browser events without duplicating or hiding actions.
• Preserving causality across popup transitions and delayed UI updates.
• Distinguishing a read preview from an accepted mutation when both used the same pathname and HTTP status.
• Proving stable locators while raw DOM IDs changed between runs.
• Keeping run-specific evidence hashes strict while deriving a stable semantic contract for replay.
• Giving Codex enough verified information to build a useful test without allowing it to invent selectors, waits, or outcomes.
• Demonstrating tamper rejection without modifying the original evidence.

Accomplishments that we're proud of

• recording:demo starts a new loopback portal and Chromium session on every run, captures a deterministic synthetic walkthrough, and does not replay a committed recording fixture.
• A fresh run promoted all 13 observed actions as replay-ready, including an application-owned validation rejection, operator correction, retry, and accepted mutation.
• The pipeline withholds incomplete or contradictory evidence from the authoring handoff, while the promotion gate rejects incomplete manifests and unsafe, undeclared, modified, or concurrently changed artifacts.
• The generated report shows per-action evidence chains, source and semantic hashes, promotion and replay status, and the number of actions requiring human review. The evidence contract preserves the unresolved reasons.
• The committed Playwright scenario is bound to the semantic contract hash, exact ordered action IDs, and scenario SHA-256.
• recording:verify performs another fresh capture, starts a new local portal, verifies scenario provenance, and executes the actual Playwright test. Provenance or replay failure returns a nonzero result.
• recording:tamper-demo changes one meaningful field in an isolated copy, receives a real OUTPUT_MISMATCH rejection, and verifies that the original evidence fingerprint remains unchanged.
• After dependencies and Playwright Chromium are installed, the complete judge path runs locally with no credentials, database, external target, or runtime AI service.

What we learned

Recording browser gestures is the easy part. The difficult part is proving context, intent, and outcome strongly enough that the recording can safely become automation.

Deterministic software should establish the facts, a coding agent should synthesize only within those facts, and deterministic replay should have the final word.

A browser recording becomes useful only when its integrity and meaning can be verified.

What's next for Evidence-to-Test

• Add more synthetic application profiles one at a time while preserving the same fail-closed guarantees.
• Package the bounded authoring handoff as an optional Codex skill.
• Improve action-level navigation and evidence comparison in the HTML report.
• Verify additional operating systems before claiming broader platform support.
• Explore additional export formats without turning the project into a generic recorder or automatic JSON-to-code generator.

Built With

• ai-agents
• browser-automation
• chromium
• developer-tools
• end-to-end-testing
• gpt-5.6-sol
• node.js
• openai-codex
• playwright
• pnpm
• security
• typescript
• vitest
• zod