VerifierForge

• 

  Cut the cost of repetitive AI work with small models that are trained, checked, and deployed automatically.

Inspiration

Inside most companies' LLM bills hides the same pattern: a few workloads that are high-frequency, simple, and programmatically verifiable. Teams pay generalist prices for specialist work. In our demo dataset, one internal data assistant turns 95,000 questions a month into SQL and costs $5,500. Tasks like that don't need a frontier model. They need a small model you can trust — and trust requires evidence, not vibes. Our tagline is the whole thesis: evidence before inference.

What it does

VerifierForge runs a six-step loop: it connects to your LLM traffic, discovers costly workloads, forges a small specialist model, proves it, ships it, and guards it.

A GPT-5.6 agent audits every discovered workload with four analysis tools and decides whether forging pays off. It writes the full training config; a human approval gate is the only manual step. The system then rents the cheapest available GPU at live market price, trains with GRPO against an automatic verifier, and validates on held-out tasks the model never saw: 58.3% → 78.3% pass@1, while a random-reward control run stayed flat. Shipping is a canary toggle with a live Guardian that re-scores production answers using the same verifier (0.95 live pass rate). Serving scales to zero: one click wakes a GPU in about five minutes for about two cents, and idle machines terminate themselves. One full autonomous forge cost $0.18.

How we built it

Eight days, one person plus a teammate on frontend and video, and Codex running in a single long-lived session for roughly seven days. Every work wave followed the same loop: a written plan with acceptance gates, budget caps, and stop conditions; deep planning reviewed before execution; autonomous execution across four environments (laptop, SSH'd GPU pods, cloud consoles, and machines the system provisioned for itself); a version record per wave (v0.1 to v0.39.5) and a semantic tag per milestone; and no merge without green tests, secret scans, and the relevant eval gate. Development ran on Codex with GPT-5.6 (sol); the product's runtime brain is GPT-5.6 (luna) behind strict schemas. Stack: verl + GRPO + vLLM for training and serving, FastAPI + React, Supabase, S3, and the RunPod API for capacity-aware provisioning.

Challenges we ran into

Silent GPU incompatibilities (a Blackwell card that looked healthy and trained nothing). Dependency drift that broke serving twice until we froze the stack. A checkpoint whose recorded hash didn't match anything — which taught us that manifests need per-file hashes, an algorithm spec, and a physical location for every byte. Trainer and vLLM fighting over VRAM on one card, solved by a sequential train-then-verify gate instead of a bigger GPU. And keeping ourselves honest: our spurious-reward control exists because gains that come from formatting tricks look exactly like real gains until you falsify them.

Accomplishments that we're proud of

A closed evidence chain: +20.0 points on held-out tasks with a flat control. An agent that passed its own adversarial evaluation gate at 1.0 / 1.0 / 0 / 1.0 — we gave our agent a verifier, which is the product's philosophy applied to itself. A fully autonomous forge lifecycle: decide, rent, train, verify, collect, terminate, for $0.18. Scale-to-zero serving with a five-minute cold start. 485 passing tests, and a hosted product judges can actually touch.

What we learned

Verification-first works twice: the same discipline that makes small-model training trustworthy also makes agent behavior trustworthy. Rigor should be allocated by consequence, not by recent pain — our worst incidents came from the layers that had never hurt us before. Stateless workers plus S3 turn dead machines into a non-event. And small data is enough when the gates are strict: fifty seed examples, honestly verified, beat a bigger pile of unchecked ones.

What's next for VerifierForge

Two discovered workloads are still waiting for their forge button. Next: a second provider adapter (Nebius — the interface is ready), a read-only connector to customer data warehouses, auto-serving policies, and forging beyond SQL into every verifiable task class a company runs. The loop stays the same: evidence before inference.

Built With

• fastapi
• grpo
• javascript
• openai
• post-train
• python
• pytorch
• railway
• react
• runpod
• supabase
• verl
• vllm