patchbay

Inspiration

Every “AI writes code” demo stops when the code shows up on screen. Nobody shows it actually running. For integration code — the webhook glue, auth headers, retry logic, half the fields you forgot about — that’s exactly the part where it falls apart. The model’s draft looks right and is quietly wrong about one URL path, and you only find out when the webhook silently drops in prod at 2am.

So I built patchbay to close that gap. The agent writes the connector AND runs it, in a sandbox, against the real documented example from the vendor’s docs, before you ever see a file. If it’s red, the model gets the test output back and tries again. Up to four attempts, two parallel repair branches each.

What it does

You type one sentence. “When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag.” Pick TypeScript or Python. Hit build.

Five steps run:

Plan. A small Nemotron (3.5 Lightning) extracts source, target, trigger, action, field mapping. Fast and cheap, usually under a second.

Research. Tavily hits the actual vendor docs — biased toward each vendor’s own domain, so Stripe questions get Stripe pages, not Stack Overflow guesses. A mid-tier Nemotron (3 Super) distills the endpoints into a spec with example payloads taken straight from the docs.

Generate. Nemotron 3 Ultra writes the connector, the tests, a README. TS by default, Python when you flip the toggle.

Verify. A Nebius sandbox runs it — typecheck first, then vitest or pytest. Network switched off. The test harness runs the connector against the documented example payload (the “contract” test), so a run only goes green when the code actually works with the shape the API returns.

Repair. If anything is red, two branches of the strong model get the failing output at different temperatures. First one green wins. And it’s monotonic — the agent will never hand back an attempt that’s strictly worse than one it already had, because I learned the hard way that Ultra will absolutely do that if you let it.

Also: patchbay scan . walks any repo for integration opportunities (empty webhook handlers, vendor SDKs imported but barely used, .env.example entries pointing at Stripe/Slack/etc with no code behind them) and hands you ready-to-run build prompts. --build-top=3 just does the top three.

How we built it

Three Nemotron tiers on Nebius Token Factory — Lightning for the small structured steps, Super for research, Ultra for the code. Verify runs in Nebius Sandboxes, forked from a checkpoint with the test deps pre-installed, network off per fork. The whole thing is OpenAI-compatible under the hood, so pointing at a different provider is one env var.

Tavily does both the search and the page extraction. We tried training-data-only research first and the connectors kept making up endpoints that don’t exist.

Six ways to call it: a CLI, an MCP server (one-line install in Claude Code / Codex / Cursor), the web demo, a Slack slash command, a GitHub Action that reacts to /patchbay build in PRs, and patchbay push that opens a PR on your repo with the finished connector.

The self-improvement loop. Every patchbay run that goes red → green is a repair pair. We extracted 173 of those from committed benchmark runs and fine-tuned Qwen3-30B-A3B on Nebius Token Factory Post-training. Then pulled the checkpoint, deployed it on a Nebius AI Cloud H100 NVLink preemptible behind a vLLM endpoint, and pointed patchbay’s repair step at it via one env var. Three Nebius products in one closed loop: inference generates the data, post-training trains on it, AI Cloud serves it.

Honest ablation on 20 TS Core prompts:

Fine-tune on repair + Ultra generate    Pure Ultra (both steps)

Green 45% (9/20) 80% (16/20) Cost / build $0.088 $0.144

30B on 173 pairs doesn’t beat 550B Ultra on raw green rate — but 1.64× cheaper per build, and the failures are almost all near-green (7/8, 9/10, 12/13). More training pairs is the obvious next move; the pipeline that produces them is kind of the point. Dataset is open: patchbay-nebius/connector-repair-dataset.

Challenges we ran into

A contract test that silently never ran. The regex that detected “did the contract check pass?” was anchored at the start of vitest’s fullName, so a describe('contract', …) nested inside an outer describe('connector', …) was classified as “didn’t run.” Two prompts stayed red for 10% of benchmark attempts. Fix was ten characters. Benchmark went from 8/10 to 10/10 on the core set.

Parallel repair with three branches was worse than two. Temperatures 0.1 / 0.4 / 0.7 produced too much variance — “best of three” was often “best of three near-misses.” Rolled back to two branches.

Linear → Discord kept failing because the model was being too careful. The generated connector added a wall-clock check that rejected the frozen 2024 timestamp in the documented example. Fix: one prompt line about freezing with vi.useFakeTimers.

Vercel monorepo setup took three independent fixes. tsconfig extends chain broken by rootDirectory, esbuild bundled into webpack by mistake, workspace symlinks invisible to the build.

The fine-tune serving pipeline. Token Factory serves fine-tunes behind an invite-only beta we’re not in, so we pulled weights ourselves. First attempt: urllib → S3 403 after 6 shards (turns out urllib doesn’t handle the 307 → S3 signed URL redirect correctly). Second attempt: openai SDK → python not found because vllm-openai image only has python3. Third attempt: worked.

Every red benchmark case has a committed post-mortem in docs/build-log.md with the diff that fixed it.

Accomplishments that we’re proud of

Two benchmarks, both committed. TypeScript Core 20 at 90.2% green over 92 runs, extended Brutal set (OAuth2 refresh, S3 presigned uploads, GraphQL nested mutations, HMAC-signed events) at 69% green. Python Core 20 at 73.3%, Python Brutal 20 at 70%. Every per-run JSON is committed. npm run bench -- --passes=5 reproduces. Six surfaces for one agent (CLI, MCP, web, Slack, GitHub Action, push subcommand). Public uptime from day 13. The web app, Nebius Token Factory, and Tavily are watched by openstatus. Build-in-public from day 1. Every red case has a committed post-mortem. 27× cheaper than Claude Opus 4.7 on the same task (~$0.07/connector vs ~$1.90).

What we learned

Variance is the real benchmark. Single pass tells you nothing. The jump from 85% → 90% only showed up after three averaged passes — individual passes gave us 95%, 83%, 92%.

The cheapest fix is a better prompt. Four +5-point benchmark jumps came from prompt edits, not model changes or token budget cranks.

Preflight before sandbox is free quality. Running esbuild’s parser before every sandbox call catches ~50% of first-draft failures in milliseconds instead of 15-30s. The saved attempts turn into actually useful repair.

Monotonic repair is non-negotiable. The strong model will absolutely regress if you let it.

Nebius Sandboxes change the ergonomics of agent verification. Isolated, network-off, checkpointed — exactly what you want for “run model output without blast radius.”

What’s next for patchbay

More target languages (Go, Java, maybe Elixir). More repair pairs — the fine-tune is clearly data-starved, 1000+ pairs is the next milestone. A hosted version where you don’t need your own Nebius account. And something that connects the GitHub Action to Linear so a new ticket with the right label auto-opens a PR with a built connector.

If any of that sounds useful, the code is MIT and I’m around.

Built With

• docker
• esbuild
• model-context-protocol
• nebius
• nemotron
• next.js
• node.js
• openstatus
• pytest
• python
• slack
• tavily
• typescript
• vercel
• vitest