Graft AI

Inspiration

Coding agents have become excellent at reading repositories, writing patches, and running tests. But many important software bugs do not exist in the code alone—they appear only when someone interacts with the product users actually see.

A release console can display a green success message while applying the wrong configuration. An administrative dashboard can report that a setting changed even though the underlying system ignored it. A browser test can also fail after a harmless redesign even when the product still behaves correctly.

This creates a major gap for companies adopting Codex. Much of the software they depend on still lives behind internal dashboards, vendor portals, and legacy web interfaces that were designed for humans rather than agents. Codex may understand the repository, but the final mile of debugging and operating the real product often remains manual.

We built Graft around one idea:

Give coding agents eyes on the software they ship—not just the code.

GPT‑5.6 Sol can understand and interact with a live interface through Codex Browser. Graft turns that temporary visual understanding into something developers can trust and keep: verified evidence, permanent regression tests, reviewable Codex repairs, CI coverage, and typed MCP capabilities.

What it does

Graft is a local-first developer tool and Codex plugin that turns GUI-only bugs and browser workflows into permanent developer infrastructure.

Its primary workflow is Graft Repro:

1. A developer provides a natural-language bug report.
2. GPT‑5.6 Sol turns it into a safe semantic reproduction plan, including expected states, inputs, invariants, forbidden actions, approval boundaries, and evidence requirements.
3. Codex Browser operates the real web application while Graft captures meaningful semantic checkpoints.
4. Graft checks the actual result through a separate read-only verifier instead of trusting a success banner or the model’s confidence.
5. Once the bug is confirmed, Codex generates a deterministic Playwright regression test.
6. Codex receives a bounded evidence package and repairs the application inside an isolated worktree.
7. Graft runs the exact same regression test again and certifies the corrected behavior for CI.

Our hero example uses Launchpad, a representative internal release console used by a platform team.

A developer requests a 10% staging canary rollout. Launchpad displays a successful result, but the underlying system has actually applied 100%.

Graft proves:

• Requested canary: 10%
• Applied canary: 100%
• Visible result: success
• Oracle result: failed
• Classification: product_regression

The green screen is not accepted as proof. The same generated regression test fails against the broken application and passes unchanged after Codex repairs it.

Graft also distinguishes between two failures that often look identical to traditional browser automation:

• Product regression: the interface completes, but the actual result is wrong. Codex repairs the application.
• Interface drift: labels, navigation, layout, or controls changed, but the underlying behavior remains correct. Codex repairs only the generated adapter.

The public capability contract remains unchanged in both cases.

Through Graft Connect, a certified workflow can also become a typed MCP tool. A fresh Codex session can invoke the operation in natural language, pause for explicit human approval, execute it deterministically, verify the real effect, and safely deduplicate retries without visually rediscovering the interface every time.

How we built it

We designed GPT‑5.6 Sol, Codex, and Graft to have separate responsibilities.

GPT‑5.6 Sol handles the semantic judgment:

• Understanding a natural-language bug report
• Planning a safe reproduction
• Identifying meaningful application states
• Separating variable inputs from fixed constraints
• Recognizing side-effect and approval boundaries
• Extracting stable behavior from a changing interface
• Distinguishing interface drift from product regression

Codex Browser handles live interaction with unfamiliar browser software. Graft does not implement another screenshot, mouse, keyboard, or visual-control agent.

Codex performs the engineering work:

• Generating deterministic Playwright regression tests
• Compiling certified workflows into typed MCP adapters
• Repairing adapter code after interface drift
• Repairing application code after a demonstrated product regression
• Producing reviewable diffs in isolated Git worktrees

Graft owns the trust and lifecycle layer:

• A semantic capability schema with typed inputs, states, transitions, invariants, and postconditions
• A dedicated Chromium capture and deterministic Playwright runtime
• A separate read-only verifier that generated code cannot access
• Durable SQLite state for jobs, approvals, effects, receipts, and certificates
• Explicit approval immediately before consequential writes
• Idempotency and interrupted-effect reconciliation
• Sandboxed generated adapters with restricted authority
• Dynamic MCP tool registration
• Content-addressed evidence with SHA-256 integrity checks
• JSON, JUnit, HTML, and Playwright trace reports
• A local Console, CLI, Codex plugin, and GitHub Actions workflow

Stable certified workflows intentionally use deterministic Playwright rather than repeatedly calling a visual agent. GPT‑5.6 Sol is used when an interface is unfamiliar or has changed; normal testable software handles the repeatable path.

Challenges we ran into

The hardest problem was not making a browser click. It was deciding when a GUI result deserved to be trusted.

A success message is useful evidence, but it is not ground truth. We had to keep verification separate from the browser, the generated adapter, and the model context. Only the read-only oracle can establish whether the durable effect matches what the developer requested.

A second challenge was deciding what Codex is allowed to repair. A browser workflow can fail because a button moved, or it can complete successfully while the product performs the wrong operation. Allowing the agent to repair the wrong layer could hide a real defect.

Graft therefore enforces a strict repair boundary:

• interface_drift permits an adapter-only repair.
• product_regression requires an application repair.
• Neither repair may weaken the public contract or expected behavior.

Consequential actions introduced another challenge. A confirmation dialog alone is not sufficient. We implemented explicit prepare and commit phases, trusted-terminal approval, durable idempotency, and effect reconciliation so a timeout or retry cannot silently create the same operation twice.

We also treated provenance as part of the product. Live browser interaction, deterministic Playwright execution, retained Codex repairs, fixture-backed judge paths, and model outputs are labeled separately. We never infer visual-agent calls from browser actions or present replayed evidence as a new live run.

Finally, generated code had to remain less powerful than the system executing it. Graft restricts generated adapters from obtaining unrestricted network, filesystem, environment, verifier, or commit access.

Accomplishments that we're proud of

• Graft caught a convincing false-green result: Launchpad reported a successful 10% rollout while the verifier proved that 100% had been applied.
• The generated regression test fails on the broken application and passes unchanged after the application is repaired.
• Graft correctly distinguishes a genuine product regression from a harmless interface redesign.
• Codex repairs the application for behavioral defects and only the adapter for interface drift.
• The public capability contract remains unchanged across both repair paths.
• Stable execution accepts zero false successes and creates zero duplicate effects.
• A fresh Codex session discovers the generated launchpad.deploy_canary MCP capability and invokes it from natural language.
• The tool pauses before the write for explicit approval, returns a receipt only after the oracle agrees, and safely deduplicates an identical retry.
• Repairs remain inspectable Codex worktree diffs rather than hidden model actions.
• The project includes a complete local product experience:

Report → Reproduce → Prove → Generate Test → Codex Fix → Certify

• We also dogfooded Graft on a real identity and resume issue found in Graft’s own Console.

What we learned

The durable layer between probabilistic visual understanding and deterministic software is not a screenshot, selector, or recorded coordinate. It is semantic intent.

Buttons move. Labels change. Menus are reorganized. But the underlying operation—its inputs, invariants, approval boundary, and expected result—can remain stable.

We also learned that models are most useful when they propose meaning and generate repairs, while deterministic systems retain authority and decide truth.

GPT‑5.6 Sol can understand the running product. Codex can generate and repair the code. But Graft’s policy, approval, idempotency, certification, and verifier layers decide whether an action may run and whether it actually succeeded.

Finally, honest provenance increases trust. Clearly separating live interaction, deterministic execution, retained Codex evidence, and fixture-backed judge paths made the system more credible rather than less impressive.

What's next for Graft AI

The Build Week release proves Graft on a resettable, Chromium-accessible developer workflow with deterministic postconditions and a configurable read-only verifier.

Next, we want to:

• Make target and verifier setup easier for developers
• Support more verifier providers and reusable capability templates
• Expand into feature-flag consoles, security dashboards, device labs, internal administration tools, vendor portals, and legacy web systems
• Improve native Windows and macOS packaging
• Strengthen semantic drift analysis across application versions
• Let teams share certified capabilities, evidence, and compatibility histories
• Make it easier to promote unfamiliar GUI workflows into durable MCP integrations

Our broader goal is to help companies adopt Codex across the software they already use.

Codex Browser gives agents access beyond the repository. Graft turns what they see into verified tests and tools developers can trust.

The next trillion users on the internet will be AI agents.

Agents are already doing a lot, like browsing the web, doing research, making purchases, and managing legacy CRMs, but they're doing it on top of software that was designed for humans clicking buttons in a browser, which is slow, inconsistent, and brittle.

Agents need a completely different foundation. Instead of visual interfaces like forms, buttons, and dashboards, they need machine-readable interfaces like APIs, MCPs, and CLIs. Agents also need thorough documentation, to enable them to discover, sign up for, and instantly start using new tools programmatically, without needing a human in the loop.

We're enabling this at Graft. Graft will work to build agent infrastructure to bridge the gap between software that was designed for humans to use but agents can use it much faster, cheaper, better and more accurately. We're enabling the biggest agent blocker in big companies.

Built With

• browserautomation
• bun
• chromium
• cicd
• codex
• codexbrowser
• codexsdk
• devops
• githubactions
• gpt-5.6
• jsonrpc
• jsonschema
• mcp
• node.js
• openai
• playwright
• plugins
• regressiontesting
• rust
• security
• sqlite
• sse
• tauri
• testing
• typescript