PromiseProof

GPT-5.6 investigates, Codex repairs, a human approves and an unchanged deterministic verifier decides PASS.

Comment

A setting is a promise. When a product tells you that personalization is off, you would expect that nothing identifying you reaches its recommendation/personalization system right?

But what if every setting, service and backend says it is off while one identifying event still crosses that boundary? With a coding agent these issues can be detected and repaired but how can you trust it to be fixed?

As a developer, I have learned firsthand that it claiming "it's fixed" is indeed a claim, not a fact. This is the catch PromiseProof tackles: the agent that changed the code should not be the one that decides whether its own repair worked.

PromiseProof separates those roles. GPT-5.6 investigates, Codex repairs, a human approves the exact patch and an unchanged deterministic verifier decides PASS.

Instructions for Judges

No account, API key or install is required to see the core idea. The verdict path uses no backend API, evidence upload or model call.

  1. Watch the five-stage walkthrough (a recorded run through all 5 steps: Observe, Investigate, Replay, Repair, Prove): https://promiseproof.alex0paiva0.workers.dev/walkthrough/
  2. Challenge the proof yourself: https://promiseproof.alex0paiva0.workers.dev/verify/?judge=1
    • On load: PASS and BOUND_AND_REPRODUCED.
    • Press Tamper OFF evidence: BROKEN_PROMISE, violation PP_IDENTIFIABLE_EVENT_LEAK, and the report you sealed a moment ago becomes STALE_OR_MISMATCH.
    • Seal the failing result: it binds honestly as BROKEN_PROMISE. A report can bind a failure; it cannot manufacture a PASS.
    • Download report.json and report.md, or load your own OFF and ON bundles under "Bring your own evidence."
    • Reset to return to the original PASS.
  3. Inspect the bundled GitHub Action: https://github.com/AlexPaiva/PromiseProof/tree/submission-rc-03/.github/actions/verify

What I built during Build Week

PromiseProof started as an empty repository during the submission period. In one week I built the Signal Shelf reference application, two distinguishable cross-system defects, the deterministic evidence and evaluation layer, a bounded GPT-5.6 investigation, an authentic Codex source-repair workflow, exact human patch approval, isolated verification, the five-stage walkthrough, an external-evidence CLI, a hosted semantic challenge and a standalone GitHub Action tested on Windows, Ubuntu and macOS. The public commit history, the Build Week log, the pull requests and the immutable submission-final tag make that work inspectable.

This is not a pre-existing platform with one new feature. The complete working implementation shown here was built, tested, productized and released during the Build Week window. See BUILD_WEEK.md and the immutable snapshot at tag submission-final.

Inspiration

A setting is a promise. When a product or service presents a control as off the users expect the system to honor it everywhere, not just on the screen. In the example of a distributed product the interface, browser storage, the network and the backend can each look correct on their own while the combined behavior quietly breaks the promise: one hidden path still behaves as if the setting were on.

There is a second problem arriving fast: while coding agents can now investigate and repair failures like this the model that changed the code is not a trustworthy judge of whether the change actually kept the promise. I wanted to let AI do the open-ended work of diagnosis and repair, while making it structurally impossible for a model to declare its own success.

What it does

PromiseProof takes one concrete user-facing promise, "when personalization is off, nothing that identifies the user should reach recommendations" and then turns it into a deterministic check, and carries a real failure of that promise through a complete repair-acceptance loop:

  1. Observe: A Playwright journey and independent network capture catch one identifiable request crossing the service boundary while personalization is off. Broken promise.
  2. Investigate: GPT-5.6 proposes ranked competing causes and selects one allowlisted diagnostic replay, within strict limits.
  3. Replay: Deterministic code runs the chosen replay and records the facts.
  4. Repair & approve: Codex prepares a constrained two-file patch containing the source repair and a focused regression test in a disposable worktree. A human reviews the exact diff and approves it by its digest.
  5. Prove: The approved patch is applied only in a fresh worktree and an unchanged Playwright journey and deterministic evaluator decide PASS.

And a judge can verify all of it directly: watch the five-stage lifecycle, tamper one meaningful observation (using the live challenge page), see the prior sealed report become stale, see the current evidence become BROKEN_PROMISE, seal an honest failing report, download JSON and Markdown proof and run the same authority through the CLI or a GitHub Action.

How I built it

GPT-5.6 investigates

  • Receives a sanitized, versioned dossier through the OpenAI Responses API.
  • Proposes two to four ranked competing hypotheses.
  • Selects one of two allowlisted factual replays through a strict function call.
  • Its result schema has no verdict field and no free-form cause field.
  • It never sees the seeded fixture, source paths, logs, or root-cause labels.

Codex repairs

  • Produces an authentic constrained two-file patch containing the source repair and a focused regression test.
  • Works in a disposable git worktree with a restricted writable surface through the pinned Codex SDK.
  • Codex also ran repeated red-team reviews that surfaced the original free-text-cause and verdict-language weakness, which led to the ID-only investigation schema and the strict report validation the verifier uses today.

A human approves

  • Reviews the exact patch and approves it by its digest in a real terminal.
  • Nothing merges automatically, no model approval.

The unchanged verifier decides

  • The same Playwright journey and deterministic evaluator that caught the break.
  • Checks OFF, Reload, ON, Browser and Control.
  • Models are structurally excluded from PASS.

Adoption surfaces: a hosted browser verifier, a repository-local CLI and a self-contained GitHub Action tested as a standalone consumer on Windows, Ubuntu and macOS. No model call runs in the verdict path on any surface.

Challenges I ran into

  • Two failures with the same visible symptom: An initialization race and a preference-propagation failure both break the same OFF promise, but they produce different evidence, different violation codes and require a different diagnostic replay. One flag cannot explain both and one fix cannot silence the other.
  • Preventing a false green: Disabling all personalization must fail and not pass. OFF must stay protected while ON functionality keeps working, and the contextual recommendation feed must stay useful. The check is designed so "turn everything off" cannot satisfy the promise.
  • Keeping AI useful without granting it authority: GPT-5.6 can reason about uncertainty but cannot declare success. Codex can prepare a patch but cannot merge or approve it. Only unchanged deterministic evidence determines PASS.
  • Making the proof portable: Evidence is canonicalized and bound with SHA-256 to a pinned evaluator fingerprint, and check reproduces the complete report rather than trusting hashes, so a stale or forged report is rejected.

Accomplishments I'm proud of

  • Built the complete product within the Build Week window as a solo developer.
  • Two distinguishable seeded failures with different evidence and violation codes.
  • Two allowlisted diagnostic replays.
  • One authentic, human-approved Codex source repair.
  • Five independent verification clauses.
  • Three developer surfaces: browser, CLI and GitHub Action.
  • The GitHub Action tested as a standalone consumer on Windows, Ubuntu and macOS.
  • Byte-identical browser and CLI gate reports for the same evidence.
  • A hosted semantic tamper challenge that runs entirely in the browser.
  • Complete report reproduction rather than hash-only checking.
  • Zero model-owned verdicts and zero model calls in the verifier verdict path.
  • An immutable submission-final snapshot of the complete submission.

What I learned

The most useful lesson to me was that integrity and correctness are two different things. A report can be perfectly attached to its evidence and still honestly say that the promise is broken. While binding proves that a report is faithful to its evidence it does not (and should not!) manufacture a passing verdict.

I also learned that bounded AI is not weaker AI. GPT-5.6 stays valuable precisely because it investigates uncertainty, while its inability to award PASS is what protects the result.

Finally, the hard part of AI-assisted repair is not producing a patch. It is building an acceptance process that proves the exact approved patch fixed the restricted state without breaking the permitted one.

What's next

  • Build deterministic evidence-producer adapters so more apps can emit compatible evidence.
  • Validate the architecture against a second real application.
  • Add another contract family only after independent validation.
  • Improve developer onboarding around normalized evidence production.
  • Explore reusable typed interfaces once more than one integration proves the abstraction.

No promise of universal support, automated evidence collection or arbitrary contracts.

Testing and supported platforms

  • Hosted, no rebuild: https://promiseproof.alex0paiva0.workers.dev/verify/?judge=1
  • Repository-local CLI: npm run promiseproof -- gate|verify|check
  • GitHub Action: uses: AlexPaiva/PromiseProof/.github/actions/verify@submission-rc-03
  • Install for local runs: npm ci then npx playwright install chromium (Chromium is only needed for the walkthrough tests, not for the verifier!).
  • Supported platforms: the CLI is directly verified on Windows and the bundled Action is exercised on Windows, Ubuntu and macOS.
  • Local no-key checks: npm run demo:rehearse, then npm run test:external and npm run test:action.

Current limitations

  • One synthetic reference application (Signal Shelf), synthetic by design so one broken promise is visible and repairable end to end. Not a legal or regulatory compliance statement.
  • Exactly one external contract family is supported: activity-personalization/v1.
  • External evidence is evaluated but not collection-attested. Binding proves a report matches its evidence and the pinned evaluator source but it does not prove the evidence was collected honestly.
  • Integrity is repository-level and content-addressed. It is not a signature, notarization, certification or third-party attestation.
  • The main branch stays intentionally seeded-broken so the red-to-green repair can be demonstrated, production is not automatically repaired.

Built With

  • ci-cd
  • cloudflare-workers
  • codex
  • codex-sdk
  • deterministic-testing
  • developer-tools
  • esbuild
  • express.js
  • git-worktrees
  • github
  • github-actions
  • gpt-5.6
  • node.js
  • openai
  • openai-responses-api
  • playwright
  • typescript
  • vite
  • zod
Share this project:

Updates