CircuitProof: AI Proposes. Evidence Decides.

GPT-5.6 proposes adversarial failure hypotheses. Deterministic evidence decides. Humans approve supported repairs.

Comment

Inspiration

Automation agents increasingly have permission to call APIs, update records, rotate credentials, and trigger operational processes. Conventional tests cover expected paths, but they often miss open-ended failures: approval arriving too late, retries duplicating side effects, sensitive data crossing a trust boundary, or rollback covering only part of a change.

Frontier models can explore that larger failure space, but a persuasive model assertion is not proof. CircuitProof was built around a deliberate authority boundary: AI proposes. Evidence decides. Humans approve repair.

What it does

CircuitProof is an automation assurance studio for builders, platform teams, security reviewers, and governance teams. Its Build Week extension adds a native Challenge workflow called Adversarial Review.

CircuitProof sends GPT-5.6 a bounded, redacted snapshot of an automation twin. The model proposes structured failure hypotheses, each naming affected components, preconditions, a consequence, and a test idea. CircuitProof then maps every proposal to deterministic checks and records one of three outcomes:

  • Proved: local evidence reproduces the weakness.
  • Unsupported: the proposal does not map to the project or lacks reproducible evidence.
  • Human review: the question depends on unresolved external behavior or a business assumption.

For proved findings, an evidence drawer separates the model proposal from CircuitProof observations, assumptions, hashes, affected components, and any supported repair. A repair appears only when the backend derives an allowlisted operation from proved evidence. The operator reviews the exact action, explicitly approves it, and CircuitProof applies it transactionally before re-running deterministic adjudication.

How we built it

The extension uses GPT-5.6 through the OpenAI Responses API for bounded adversarial hypothesis generation. Provider output is constrained by a strict schema, hypothesis limits, redaction, and at most one schema-repair request. It cannot execute code, call project tools, approve workflows, manufacture deterministic evidence, or apply repairs.

The assurance layer combines:

  • Python 3.12, FastAPI, Pydantic, SQLite, and Uvicorn
  • React, Vite, React Flow, and Lucide icons
  • OpenAI Responses API with allowlisted GPT-5.6 models
  • Deterministic three-outcome adjudication
  • Version-bound persistence, provenance records, and audit events
  • Human-gated, server-derived repair and re-run
  • Pytest and Node test suites

The judge fixture is a synthetic software release agent with eight components. It intentionally includes testable permission, sensitive-boundary, approval-order, retry-idempotency, and rollback-coverage weaknesses, plus an external dependency that cannot be resolved from the local twin.

Challenges

The hardest part was preserving the model's exploratory value without letting it become an authority. We had to make provider outputs structurally useful while ensuring that every verdict remained grounded in local evidence.

Other difficult engineering problems included preventing stale responses from attaching to a newer project version, binding assumptions correctly, avoiding false-positive proof predicates, persisting repairs atomically, handling cross-project UI races, and keeping incomplete provider runs bounded without erasing earlier evidence.

Accomplishments

  • A complete challenge-to-evidence-to-repair-to-re-run workflow
  • A clear separation between AI hypotheses and deterministic verdicts
  • Stable evidence, provenance, project-version binding, and audit history
  • A redacted provider trust boundary with strict contracts
  • Human approval for every supported repair
  • A zero-build offline replay sandbox for immediate judge evaluation
  • A live Windows judge path using the actual GPT-5.6 integration

CircuitProof existed before Build Week. The submitted work clearly discloses that baseline; judges should evaluate the post-July-13 Adversarial Review extension and its accompanying changelog.

What we learned

The most useful pattern was not asking the model to certify its own claims. GPT-5.6 is strongest here as a creative adversary that searches beyond a fixed test catalog. Deterministic checks are strongest as the decision boundary. Human review remains essential whenever the answer depends on an external system or business assumption.

Codex served as the primary engineering collaborator: auditing the existing product, converting the direction into a specification and implementation plan, implementing backend and frontend changes through focused tests, generating adversarial edge cases, and coordinating independent reviews. Sulabh retained product direction, governance policy, architecture choices, approval boundaries, and final experience decisions.

Judge path

The repository includes two supported evaluation paths:

  1. Open release/build-week/sandbox/index.html for a zero-build functional replay with no API key, account, network request, dependency installation, or server.
  2. On Windows 10/11, follow docs/build-week/JUDGE_GUIDE.md and run release/build-week/run-judge-demo.ps1 for the live GPT-5.6 Challenge flow.

The sample is synthetic. The OpenAI API key remains backend-only, and external behavior that cannot be reproduced stays explicitly marked Human review. CircuitProof provides assurance evidence, not certification.

What's next

Next steps include expanding deterministic adapters for more automation platforms, adding team review workflows, strengthening evidence export and policy packs, and testing the authority-boundary pattern across broader agentic systems.

Built With

Share this project:

Updates