Gatehouse

The submit tool doesn't exist until your repro runs. Gatehouse turns WebMCP tool availability into the acceptance gate.

Comment

Why this use case is a strong fit for WebMCP

Open-source maintainers need useful bug reports, while agents can cheaply produce reports that have never been executed. Gatehouse puts the acceptance workflow where the reporter's agent already works: in the web page. The page initially exposes tools for inspecting the target, writing a reproduction, running it, and requesting a person's attention. Only after the current reproduction fails 5/5 on the pinned reported build and passes 5/5 on the pinned reference build does the page register submit_report. Editing the reproduction or producing a later non-green result revokes that tool. WebMCP is not a transport wrapper here; its live tool surface is the workflow state, and the reporter's browser records the local evidence.

How it creates a better user experience

Reporters get immediate, specific feedback instead of waiting for a maintainer to answer “cannot reproduce.” A failed attempt can distinguish a script that fails on both versions from one that passes on both, helping a good-faith reporter isolate the change. The locally approved artifact contains the exact reproduction, pinned versions and bundle hashes, all ten samples, bounded logs, and a timeline. Maintainers can replay it against the same target, open a compact receipt, and export a regression-test starting point. Gatehouse does not replace the project's issue tracker; the receipt is designed to travel through the workflow maintainers already use.

What people and agents can do together that was difficult or impossible before

The agent iterates on executable evidence before the report enters a review queue: it reads the target, writes a minimal reproduction, executes it against both versions, interprets the differential, and revises the code. The page enforces the project's acceptance bar through tool availability. Once the evidence reaches the stable local differential, submit_report can stage it but cannot itself record local approval. A separate visible page control records that browser-local approval. The control is unauthenticated, does not verify identity, is not a cryptographic signature, and can be activated by automation. Reporter, agent, and maintainer therefore meet around the same replayable evidence before triage begins.

How we implemented WebMCP

The top-level document registers four tools with document.modelContext.registerTool({ name, description, inputSchema, execute }) (src/surface/surface.js). run_repro loads locally vendored qs 6.12.0 and 6.12.1 bundles, checks their SHA-256 comparison identifiers, and runs the same reproduction five times against each build inside fresh Workers nested in an opaque-origin sandboxed iframe. Worker termination enforces the timeout. The differential opens the gate only for reported-build fail 5/5 plus reference-build pass 5/5; mixed samples stay UNSTABLE. That stable result registers submit_report with an AbortSignal and binds it to the reproduction hash; editing or a later non-green result closes the registration. The tool stages an artifact, the page can record browser-local approval through a separate visible control, and the browser can replay or encode the result as a receipt. Native-Chrome and logic browser evals cover the tool lifecycle and failure cases.

Inspiration

On January 31, 2026, the curl project shut down the bug bounty it had run since 2019 — eighty-seven confirmed vulnerabilities, over $100,000 paid out. Not for lack of funding. Because a seven-person volunteer team could no longer afford to read the inbox.

For years, over 15% of curl's security submissions were real. Through 2025 that fell below 5%. Daniel Stenberg named it "death by a thousand slops": fluent, confident reports citing functions that don't exist.

The reflex is to build a wall. We think that's the wrong read, and Stenberg's own data says so: months earlier, AI-driven analysis had surfaced real curl bugs in volume — good enough that maintainers merged around fifty fixes from the first batch alone.

Same technology, opposite outcome. The variable was never AI. It was whether anything had been verified before it cost a human their afternoon. So we stopped asking how to keep agents out, and asked instead: what if the submit tool didn't exist until the claim was proven?

What it does

Gatehouse inverts the order of bug reporting. The agent proves the bug first, and only then is it allowed to file.

When an agent lands on the page, document.modelContext exposes four tools: read the target, write reproduction code, run it, and request human review. There is no tool for submitting a report. That capability does not exist.

The reproduction runs in an isolated browser sandbox against two real published versions of the qs library — the affected version (6.12.0) and the fixed version that followed (6.12.1). Gatehouse verifies the SHA-256 fingerprint of each version's file before running, so the agent cannot quietly test something else.

The verdict accepts exactly one outcome: fails on the bad version and passes on the good one — five runs per side, all agreeing. Both-fail means the repro is broken. Both-pass means nothing was isolated. Only a true version regression clears the gate.

When it clears, submit_report appears at runtime — bound to that exact reproduction. Edit one character and the tool unregisters itself. The capability vanishes.

For the reporter, feedback is immediate and specific: not "we couldn't reproduce this" three days later, but "both versions passed — you haven't isolated anything" in seconds. For the maintainer, every report that clears the gate arrives with the reproduction code, pinned versions, file fingerprints, both run results, logs, and a timeline — replayable in one click, exportable as a regression test.

Gatehouse doesn't replace GitHub. It attaches executed, replayable evidence to an ordinary issue.

How we built it

The page registers its base tools through document.modelContext.registerTool() in a single top-level module (src/surface/surface.js). Everything is registered at the top level — some clients discover nothing registered inside an iframe, so we don't use one.

Reproduction code executes in an isolated browser sandbox, five times per version. Before either run, the SHA-256 fingerprint of the version's file is checked against a known value. The judge then evaluates the pair of results and accepts a single combination: fail on bad, pass on good.

On a passing verdict, the page calls registerTool() again to expose submit_report, bound to the exact reproduction that just passed. That registration is held by an AbortSignal; any change to the reproduction fires it and the tool disappears. Tool availability is therefore not a UI affordance — it is the acceptance state, readable by the agent at any moment by listing tools.

submit_report stages material locally. The user can then record a browser-local approval, and the maintainer can review, replay, generate a receipt, or export the reproduction as a regression test.

Before trusting any of this, we built a separate throwaway probe page to establish what a given browser or agent client actually does with WebMCP, rather than what the docs say it does. That probe is public: https://github.com/Caleb0796/webmcp-probe

Challenges we ran into

The hardest part was resisting the temptation to overclaim. It would be easy to present Gatehouse as an authentication or trust system. It is not, and we say so on the page itself:

  • Local browser approval is not identity verification. It is not a cryptographic signature, and it can be triggered by automation.
  • submit_report stages material. It does not send anything outbound on its own.
  • Browser-side evidence does not defend against a malicious client. It raises the cost of low-effort noise; it is not a security boundary.

Being precise about that boundary is part of the design, not a caveat bolted on.

Technically, most of the time went into the fingerprint check and sandbox isolation — making sure the two runs are genuinely the same code against genuinely different versions, and that a modified reproduction can never inherit a passing verdict from an earlier one.

Environmentally, WebMCP fails silently in ways that cost whole afternoons. A secure context is required: localhost counts, a LAN IP does not — document.modelContext is simply undefined there, feature flag or no. And the Origin-Agent-Cluster: ?0 response header kills WebMCP outright, which a host can add on your behalf without telling you.

Accomplishments that we're proud of

  • The gate is real, not a demo. It runs against a real qs regression and two real published versions — not an invented toy case.
  • A complete loop, end to end: write → run → clear the gate → approve → inbox → replay → receipt → export as test. 223/223 code tests passing, and 13/13 browser evaluations passing in native Chrome against the deployed URL — six of them through document.modelContext itself (the run is recorded in evals/RESULTS.md).
  • We measured the platform instead of assuming it. The probe page exists because we didn't want a single claim in this submission resting on documentation we hadn't verified ourselves.
  • We documented what Gatehouse can't do, on the product itself. That was a deliberate choice and we'd make it again.

What we learned

  • Grade capability presence off typeof document.modelContext, nothing else. On Chrome 152, WebMCP.enable over CDP returns OK even in a launch where the page API is entirely absent. A probe that reads the CDP domain reports "supported" when it isn't.
  • Secure context is non-negotiable, and its absence is silent rather than loud.
  • Origin-Agent-Cluster: ?0 is a silent killer — dump your headers and confirm.
  • Iframes are unreliable ground for tool discovery across clients.
  • Dynamic registration is the interesting primitive. Most WebMCP demos expose a fixed tool list. Registering and revoking tools as a function of verified state turned out to be far more expressive than we expected going in — it lets a page communicate workflow state to an agent without any prose at all.

What's next for Gatehouse

  • More than one target. The gate logic is library-agnostic; the next step is a configuration surface so a project can define its own known-good/known-bad pair and its own passing condition.
  • Maintainer-side integration. Today Gatehouse produces a replayable receipt a human attaches to an issue. The natural next move is a first-class path from cleared gate to draft issue — without ever claiming a verification Gatehouse hasn't actually performed.
  • Closing the trust gap honestly. Browser-local approval is the current boundary. Server-side attestation would be the real answer, and we'd rather ship it properly than describe it as done.
  • A gate you can author. The long version of this idea isn't a bug tracker at all — it's a pattern: any workflow where a capability should exist only after a condition has been demonstrated by actually running code. WebMCP makes that expressible for the first time.

Built With

Share this project:

Updates

Submission history