ProofMode

An autonomous AI QA agent that verifies whether software actually delivers what it promises, using real browser tests, screenshots, and evidence-backed fixes.

Comment

Inspiration

Software teams often describe what their products are supposed to do, but verifying those promises usually requires someone to manually translate each claim into test cases, run the workflow, collect screenshots, and write a bug report.

That gap is especially common for early-stage products. A feature may technically exist while still failing in practice because of broken validation, missing feedback, inaccessible controls, or unexpected browser errors.

I built ProofMode to answer a more useful question than “Is this website broken?”

Does this product actually deliver the experience it promises?

ProofMode turns plain-language product claims into browser tests, executes them, and produces evidence-backed verdicts.

What it does

A user enters a website URL and one or more product claims, such as:

“Users can create an account with a valid email and receive a confirmation message.”

ProofMode then:

  1. Uses GPT-5.6 to interpret the claim.
  2. Generates explicit success criteria and a structured test plan.
  3. Uses Playwright to perform the workflow in a real browser.
  4. Captures screenshots, page observations, URLs, timing information, and console errors.
  5. Uses GPT-5.6 to evaluate the collected evidence.
  6. Classifies the claim as Verified, Failed, Partial, or Inconclusive.
  7. Produces reproduction steps, a likely root cause, and a suggested fix.

Every verdict is tied to captured evidence rather than relying only on the model’s judgment.

The project also includes an intentionally flawed demo application. It claims to validate email addresses, show a successful signup confirmation, and support keyboard navigation. ProofMode tests those claims and exposes the differences between the product’s promises and its actual behavior.

How I built it

ProofMode is built with:

  • Next.js and TypeScript
  • Tailwind CSS
  • Playwright for browser automation
  • The OpenAI Responses API
  • GPT-5.6 for test planning and evidence evaluation
  • Zod for validating structured model responses
  • Local artifact storage for screenshots, plans, evidence, and reports

The system separates reasoning from execution.

GPT-5.6 does not generate and run arbitrary browser code. Instead, it creates a constrained test plan using a limited set of approved actions such as clicking, filling fields, pressing keys, checking visible text, and capturing screenshots. Playwright then executes those actions.

After execution, the evidence is passed back to GPT-5.6 for evaluation. The evaluator is instructed not to mark a claim as verified unless the browser evidence supports that conclusion.

This structure makes the system safer, easier to debug, and more reliable during a live demonstration.

How I used Codex

Codex was central to the development workflow rather than being used only for isolated code suggestions.

I used Codex to:

  • Scope the MVP around a reliable claim-to-evidence workflow
  • Scaffold the Next.js application
  • Define shared TypeScript and Zod schemas
  • Implement the Playwright action executor
  • Connect GPT-5.6 to the planning and evaluation stages
  • Build the report interface and screenshot timeline
  • Create the intentionally flawed demo target
  • Add validation, timeouts, and error handling
  • Debug integration issues between the browser runner and application server
  • Write tests and improve the repository documentation

One of the most useful parts of working with Codex was using it across the full development loop: implementation, testing, debugging, and documentation. This helped keep the schemas, API routes, browser executor, and interface consistent as the product evolved.

Challenges I faced

Creating reliable browser plans

Natural-language product claims can be vague. A statement like “signup works” does not specify what counts as success, which fields should be completed, or what confirmation should appear.

To address this, ProofMode first converts each claim into explicit assumptions and success criteria before producing browser actions.

Keeping the agent constrained

Allowing a model to generate arbitrary automation code would make the system difficult to validate and unsafe to run.

I instead designed a small approved action set and validated every generated plan with Zod before execution. Unsupported or malformed actions are rejected.

Grounding verdicts in evidence

A model can make a confident conclusion even when the available evidence is incomplete. ProofMode therefore records the outcome of every browser step and requires the final evaluation to reference supporting and contradicting evidence.

When the evidence is insufficient, the system returns Inconclusive instead of fabricating certainty.

Making the demo dependable

Browser automation can fail because of timing, selectors, network behavior, or page changes. I added timeouts, screenshots on failure, console-error collection, readable error messages, and a controlled demo target so the full experience can be tested consistently.

What I learned

This project taught me that the most important part of building an AI agent is often not giving it more freedom, but designing the right boundaries.

GPT-5.6 is strongest in ProofMode when it handles interpretation, planning, and evidence synthesis. Playwright is strongest when it handles deterministic browser execution. Combining the two through validated structured outputs produced a more dependable system than asking one model to control the entire process without constraints.

I also learned that testing a product claim is different from testing a single interface element. A claim represents an expected user experience, so verifying it requires connecting technical behavior with the product’s stated intent.

What I’m proud of

I am most proud that ProofMode produces more than a pass-or-fail result. It creates an understandable chain of evidence showing:

  • What the product promised
  • How that promise was interpreted
  • What actions were performed
  • What the browser actually displayed
  • Why the final verdict was reached
  • How the failure could potentially be fixed

This makes the result useful not only to developers, but also to product managers, founders, designers, and QA teams.

What’s next

Future versions of ProofMode could include:

  • Testing authenticated applications
  • GitHub issue creation
  • Automatic pull-request generation
  • Visual regression testing
  • Mobile viewport testing
  • Scheduled verification runs
  • Comparison between marketing copy and production behavior
  • Team dashboards for tracking product claims over time

The long-term goal is to make product promises continuously testable, so teams can verify not only that their software runs, but that it delivers the experience users were promised.

Built With

Share this project:

Updates