ProofRun

• 

• 

• 

• 

• 

• 

Inspiration

I sell website rebuilds to local trade businesses. The hardest part was never the building — it was the twenty minutes per prospect spent proving there was a problem worth paying to fix. Open the site on a phone, squint at it, write an email that says "your site looks dated," and watch it get ignored, because "looks dated" is an opinion and the owner has heard it from six other people trying to sell them something.

The thing that actually works is showing someone the rebuilt page. Nobody argues with a page that loads faster and lets them tap the phone number. But building a speculative page per prospect is hours of work for a cold email that probably gets deleted.

So the question was: can an agent do the whole loop overnight — find the business, measure what's actually broken, rebuild the page, test its own rebuild, and hand me a draft — while leaving every decision with me?

The part I cared most about was the testing. An agent that writes a page and tells you it's better is just a more expensive opinion. An agent that writes a page, runs the same check suite it ran on the original, fails, reads the failure, fixes it, and runs again is doing something I can actually trust. What it does ProofRun is an always-on sales operator. The assistant is the shell; ProofRun is its flagship skill. Give it a niche and a city. It then:

Discovers candidate businesses with Tavily search, and screens every hit through Nemotron Nano — which correctly skipped Roto-Rooter as "a national franchise, not an independent local business." Enriches with Tavily extract plus a Playwright text pass, pulling only facts the site actually states. Captures the live site at desktop (1280×800) and mobile (390×844): screenshots, DOM snapshot, and 13 automated checks.

Audits with Nemotron Ultra 550B, producing ranked findings as validated JSON. Every finding names its evidence source and a confidence level, and says what would raise that confidence. Writes a replacement single-page site as real HTML/CSS. Runs and tests its own output in a sandbox, headlessly, against the same check code. On failure, the exact failing measurement goes back to the model and it tries again, up to three repairs. Every attempt is logged.

Proves it: before/after screenshots, score delta, full test log, retry trace. Drafts a personalized email citing only verified findings, in the operator's own voice, with an unsubscribe line and a physical-address placeholder. Waits. A human approves, edits or rejects — and that decision is written to memory, which the next draft reads.

Six screens: Morning Brief, Queue, Live Run, Proof (the hero), Approve & Edit, Memory Viewer. Nothing is ever sent. There is no SMTP client, no email API client, no transport of any kind in the repo. agent/send-adapter.js throws if you call it. Approved drafts are exported to a file for a human. It fails open. If the rebuild still fails checks at the retry cap, the run is reported as failed and no draft is written. ProofRun does not pitch a rebuild that failed its own tests.

How we built it Everything runs on Nebius Token Factory, on NVIDIA Nemotron models, across three tiers. No model ID appears anywhere in the code — code asks for a role and config/routing.json maps it to an environment variable: Role Model Why audit, redesign_plan Nemotron-3-Ultra-550b-a55b Ranking real defects by business impact and attaching honest confidence is the hardest judgment in the loop. The 1M context fits the whole DOM snapshot. build, repair nemotron-3-super-120b-a12b Long HTML/CSS generation is throughput-bound, not judgment-bound, at a third of Ultra's cost per token. classify, extract, summarize_tests, draft_email, brief NVIDIA-Nemotron-3-Nano-30B-A3B Screening, extraction and style work. At $0.06/M, screening dozens of hits a night is effectively free.

The architecture is deliberately boring and config-driven, so it can be cloned and turned into a real sales agent by editing config and adding skill folders: tools/checks.js — one check suite, run against both the original and the rebuild. Same code, same viewports, same thresholds. That is the only honest basis for a delta. tools/sandbox.js — three layers before model-written code executes: a static gate (no scripts, no external references, no inline handlers), network deny-all at load, and a child process with a scrubbed environment holding no API keys. agent/core.js — reads skills/*/skill.json off disk. There is no list of skill names in it. Adding a capability means adding a folder. memory/ — plain markdown files as the source of truth. No database. jobs/nightly.js — the batch, designed as a Nebius Serverless Job, with a "Run now" button on the same code path.

Structured output is belt-and-braces: response_format: json_schema with strict: true, and zod validation, and on a validation failure the model is handed its own error once before the step fails.

Challenges we ran into

A 200 OK containing nothing. The hardest bug by far. Nemotron 3 models are reasoning models, and on Token Factory max_tokens covers the reasoning trace and the visible content. A long prompt would return finish_reason: "length" with content: "" — a successful response with nothing in it. One call spent 40,085 characters on reasoning before producing a single word of output. The client now detects empty-at-length, escalates the budget, retries, and records the escalation in the trace.

The retry loop didn't fire, and the reason was my fault. My first build prompt handed the model every threshold the check suite uses — 44px, 4.5:1, 390px. It passed first try every time. That's teaching to the test, and it made the repair loop decorative. I rewrote the brief at intent level ("mobile-first", "WCAG AA") — what a real operator would actually say — and left the exact measurements to the repair prompt. Now failures are genuine misses, and the loop earns its place.

Comparing two scores that weren't comparable. A live site has outbound links to probe; a self-contained generated page has none, so that check is skipped on the rebuild. Comparing percentages with different denominators would have flattered the result. The headline delta is now computed only over checks that produced a result on both runs, and every excluded check is named on screen with the reason it dropped out. The model thought the future was fake. Ultra flagged a real business's October 2026 reviews as "future-dated and probably fabricated" — because its training data predates today. Any prompt reasoning about recency now gets the real date injected.

Drafts that leaked their own scaffolding. Live runs produced a repeated Subject: line inside the body, an Applied: 1 label, an unfilled {{city}} placeholder, internal finding ids like F1 in prose, and one unbroken wall of text. Rather than trust the prompt, each is now a schema refinement that rejects the output and makes the model repair itself.

Region-pinned models. Each Nemotron model lives in exactly one region, and the docs show both a global and a regional host without saying whether the global one routes everywhere. The client tries the regional endpoint first, falls back to global, and caches whichever answered.

Accomplishments that we're proud of

The agent tests its own work and visibly fails. On the demo run it wrote a page with a 78-character title, the check suite caught it, the exact measurement went back to the model, and the second attempt passed: 30 → 100. On a live run against a real plumbing company it needed all three repair attempts to get there. That's a real engineering loop, not a scripted animation.

Memory provably changes the output, and there's a test that fails if it doesn't. scripts/try-memory.js prints draft v1, records an operator correction, redrafts with the audit, proof, model and prompt all held constant, and asserts the text changed. v1 opened "Hi Harbor & Vine Plumbing,". After the correction, v2 led with the measured defect and dropped the score entirely — and the model reported back exactly which rules it followed.

Honesty is enforced in code, not promised in a prompt. The schema rejects a finding that claims high confidence from inference. Checks declare what they did not look at. Text over a background image is reported as unmeasurable, never as a pass. Page weight is described as a floor, not a total. Every number on screen ships with its limit beside it. A clean one-command path. npm start, press Run now, and the whole loop runs in about two minutes with the live feed streaming every tool call, model call and retry.

What we learned

Reasoning models break the max_tokens mental model. If the budget covers thinking as well as output, every token estimate built for non-reasoning models is wrong, and the failure mode is silent — a valid response with an empty string in it. Handle finish_reason: "length" explicitly. Where you put the specification decides whether verification means anything. Put the thresholds in the build prompt and the model passes trivially. Put them only in the feedback loop and you find out what it actually gets wrong. Validate structure rather than asking nicely. Every "please don't do X" in a prompt was eventually violated. Every schema refinement was obeyed on the repair, because the model got its own error back and fixed it. Cheap models are good enough for most of an agent. Seven of nine roles run on Nano 30B. The expensive model is reserved for the two places judgment actually matters. A full demo run is roughly 32,000 tokens. An honest number is more persuasive than a big one. "33 → 100 over 10 comparable checks, 3 excluded" is a better claim than "100/100", because it tells you what was measured.

What's next for ProofRun

Finish the Serverless Job deployment so the nightly batch runs on Nebius on a schedule rather than on my laptop. A second real skill. skills/missed-call-offer/ is a stub with a manifest and a prompt but no runner, because placing test calls to real businesses is a consent and recording question before it's an engineering one. That gets answered properly before it ships. Nemotron Super VL on the screenshots. The audit currently reasons over a DOM snapshot and check output. A vision model reading the actual rendered before/after would catch what the DOM cannot: visual hierarchy, whether the hero image is a stock photo, whether the page simply looks untrustworthy. Widen the check suite honestly. Real Core Web Vitals, throttled mobile loads, and a 320px viewport — each added with its limits stated, the same way the current thirteen are. Learn from outcomes, not just corrections. Right now memory records what I edited. It should eventually record which drafts got replies.

Built With

• chromium
• css
• html
• javascript
• nebius-token-factory
• nemotron-3-nano-30b
• nemotron-3-ultra-550b
• node.js
• nvidia-nemotron
• playwright
• tavily