Reforge

An AI agent that migrates legacy Python code and proves nothing broke.

Comment

Inspiration

Every Python team has an upgrade it keeps postponing. A major release like Pydantic v2 breaks validators, configs and serialization across a codebase, while the old version keeps collecting CVEs. The work is tedious, risky and easy to get subtly wrong, so it waits until a security audit forces it. I wanted an agent that doesn't just write the migration but proves it didn't break anything.

What it does

Reforge takes a GitHub repository or a code snippet and runs a five-stage agentic pipeline:

  1. Scan & baseline: detects the pinned dependency, checks known CVEs on OSV, and runs the existing tests under v1.
  2. Break & slice: installs v2 in an isolated sandbox, records exactly what breaks, and orders affected files with a static AST scan.
  3. Ground & plan: grounds every broken symbol in the official migration docs via Tavily; Nemotron 3 Ultra writes one validated refactor plan.
  4. Patch & self-heal: Nemotron 3.5 Lightning applies patches first; misses escalate to Super, then Ultra. Failing tests go through a heal ladder, and any fix that makes things worse is rolled back.
  5. Prove parity: compares the original and refactored code on test outcomes, return values, schemas and output, re-checks CVEs, and produces an audit PDF, a git-ready patch and side-by-side diffs.

How I built it

  • Backend: Python + FastAPI, with NDJSON streaming so the UI shows every stage live.
  • Models: three NVIDIA Nemotron tiers on Nebius Token Factory, behind one OpenAI-compatible endpoint.
  • Router: cheap-first escalation with a cost ledger. Every call is priced and compared with sending the same tokens to Ultra:

$$\text{Cost} = \sum_{i} \left( \frac{T^{in}_{i}}{10^6} \, R^{in}_{tier(i)} + \frac{T^{out}_{i}}{10^6} \, R^{out}_{tier(i)} \right)$$

$$\text{Savings} = \text{Cost}_{\text{Ultra-only}} - \text{Cost}_{\text{actual}}$$

where:

  • \(T^{in}_{i}\), \(T^{out}_{i}\): input and output tokens of model call \(i\)
  • \(tier(i)\): the Nemotron tier that handled call \(i\) (Lightning, Super or Ultra)
  • \(R^{in}_{tier}\), \(R^{out}_{tier}\): that tier's Token Factory price in USD per 1M input / output tokens (Lightning \$0.06 / \$0.24, Super \$0.30 / \$0.90, Ultra \$1.00 / \$3.00)
  • \(\text{Cost}_{\text{actual}}\): what the run really cost with routing

  • \(\text{Cost}_{\text{Ultra-only}}\): the same tokens priced at Ultra rates, the "no routing" baseline

  • Safety: model output is validated, never trusted. Patches must match exactly once, pass an AST syntax gate, and are written atomically with rollback.

  • Frontend: zero-build HTML/CSS/JS with a live execution trace, model-router panel and results workspace.

  • Deployment: Docker on Google Cloud Run, with daily usage limits and a judge key so the credits last through judging.

Reference run: full migration for $0.12 vs $0.32 on Ultra alone (62% cheaper), with 1/1 CVE resolved, 100% output parity and a 5.28× hot-path speedup. The patch stage alone saved 88.5%.

Challenges

  • Making the models reliable, not just smart. Lightning sometimes copied placeholder text from the prompt example; the patcher now rejects it. Thinking mode on the smaller tiers burned tokens without improving patches, so it's off for Lightning and Super.
  • Proving correctness without trusting the model. Tests alone weren't enough, so I added a parity verifier that compares outputs and schemas between both versions and flags any attempt to edit the tests.
  • Speed. The demo needed a full migration in about a minute, so patching runs per file in parallel, sandboxes are reused, and identical reruns hit a plan cache at $0.
  • Budget. With limited credits, every design choice had a cost: routing, caching, per-run budgets, and daily caps on the hosted demo.

What I learned

  • Cheap models do most of the work well when the task is narrow and the output is verified. The big model is best used once, for planning.
  • Grounding in official docs beats relying on model memory for API migrations.
  • In agentic systems, verification and rollback matter more than the generation itself.

What's next

The engine is library-agnostic: adding a migration means adding a migration pack (symbol rules, detection and docs), not new agent logic. Next up: SQLAlchemy 2.0 and NumPy 2, private repositories, and opening pull requests automatically.

Built With

Share this project:

Updates

Submission history