Inspiration
AI coding assistants are excellent at generating code, but they often struggle with newly released breaking changes and security updates. Their training data may describe an older API, causing them to confidently produce code that is already outdated.
We built PatchForge around a simple principle:
Trustworthy Patch = Current Evidence + Safe Changes + Passing Tests
Instead of relying only on a model's memory, PatchForge reads current migration documentation, updates the repository, tests every change in an isolated environment, and creates a pull request only after verification succeeds.
Our goal is to turn dependency migrations from a repetitive and risky maintenance task into a grounded, automated, and reviewable workflow.
What it does
A developer connects a GitHub repository and enters an instruction such as:
Upgrade Pydantic to v2
PatchForge then:
- Clones the repository and runs its existing tests to establish a baseline.
- Detects the package, current version, target version, runtime, and related dependencies.
- Searches current official documentation using Tavily.
- Uses NVIDIA Nemotron through Nebius Token Factory to transform that research into a structured Delta Spec of breaking changes.
- Updates dependency and runtime versions deterministically.
- Patches affected source files using parallel coding-model workers.
- Runs the updated project inside an isolated sandbox.
- Uses actual test failures to guide repair attempts.
- Rejects unsafe changes such as edited tests, skipped checks, weakened type safety, or secret-like content.
- Opens a GitHub pull request only when verification succeeds.
Every successful migration is saved as a reusable skill. When another repository needs the same upgrade, PatchForge can reuse the validated migration knowledge, reducing model calls, research time, latency, and cost.
How we built it
PatchForge is a Python and FastAPI application with a lightweight HTML, CSS, and JavaScript dashboard.
NVIDIA Nemotron, accessed through the OpenAI-compatible Nebius Token Factory API, acts as the migration architect. It interprets the developer's request and converts current documentation into a structured migration plan. Coding-model workers then apply file-level changes, while deterministic Python logic handles package versions and runtime requirements.
Tavily searches official documentation and migration guides. PatchForge prioritizes authoritative package domains, retrieves full page content, filters low-quality sources, and performs additional searches using errors discovered during failed test runs.
The verification system supports both Python and JavaScript/TypeScript repositories. Docker provides the primary isolated sandbox, and we also implemented a Nebius Serverless AI Jobs backend for asynchronous test execution.
A GitHub App handles authentication, repository access, branches, commits, and pull requests. Credentials remain in the PatchForge backend and are never exposed to the model or untrusted repository code.
The dashboard uses Server-Sent Events to display pipeline progress, research sources, model calls, token usage, estimated cost, latency, test results, saved skills, and the final pull-request URL.
Challenges we ran into
The biggest challenge was making an autonomous coding workflow safe enough to use on real repositories.
We needed to ensure that untrusted repository code could never access GitHub or model credentials. We also had to prevent the agent from making a broken migration appear successful by editing tests, skipping checks, disabling type safety, or introducing secret-like content.
Finding reliable migration information was another challenge. Search results frequently contain outdated tutorials, incomplete examples, or pages that only mention the library in passing. PatchForge addresses this by prioritizing official documentation, retrieving full pages, validating migration examples against the collected evidence, and searching again when real test failures provide better context.
Supporting both Python and Node.js introduced additional complexity because they use different package managers, runtime versions, test frameworks, build systems, and error formats.
Finally, we had to find the correct balance between AI reasoning and deterministic control. Models are valuable for interpreting documentation and adapting source code, but versions, credential boundaries, safety limits, and final pass-or-fail decisions should not depend on model confidence.
Accomplishments that we're proud of
We are proud that PatchForge is a complete GitHub-to-pull-request workflow rather than only a code-generation prototype.
Our main accomplishments include:
- Integrating NVIDIA Nemotron through Nebius Token Factory as a central reasoning component.
- Grounding migrations in current documentation retrieved with Tavily.
- Creating repair loops driven by actual compiler and test failures.
- Opening pull requests only after verification succeeds.
- Isolating GitHub credentials from models and untrusted repository code.
- Supporting Python and JavaScript/TypeScript projects.
- Managing dependency and runtime versions deterministically.
- Tracking model calls, tokens, latency, and estimated cost.
- Converting successful migrations into reusable skills.
- Evaluating migrations involving Pydantic, Flask, Django, pandas, Express, Node security updates, TypeScript, and React.
The reusable-skill system is our favorite feature. It means the value of a successful migration does not disappear after one pull request—the next repository can reuse that validated knowledge.
What we learned
We learned that a dependable coding agent requires more than a capable language model. It also needs current evidence, constrained tools, observable execution, deterministic safety checks, and an external source of truth.
For PatchForge, that source of truth is the repository's verification suite:
$$ \text{Model Confidence} \neq \text{Correctness} $$
Real test failures also turned out to be extremely valuable agent context. Giving the repair model the exact error, affected source file, and relevant read-only test produces better results than asking it to guess from a generic failure message.
We also learned that probabilistic and deterministic components work best when each is assigned the right responsibility:
PatchForge = AI Reasoning + Deterministic Controls
- AI Reasoning: Research and code adaptation
- Deterministic Controls: Versions, security, and verification
Finally, we learned that successful agent work can become durable knowledge. Saving verified migrations as reusable skills makes future upgrades faster, cheaper, and more consistent.
What's next for PatchForge
Our next steps are to:
- Scale verification workloads using Nebius Serverless Jobs.
- Add persistent queues and database-backed job history.
- Support monorepos and coordinated multi-package migrations.
- Introduce human approval checkpoints for high-risk changes.
- Expand support to more programming languages and package managers.
- Add enterprise policies, audit logs, and deployment controls.
- Improve evaluation coverage across additional frameworks and security advisories.
- Build a shared registry of versioned and validated migration skills.
Our long-term goal is to make dependency maintenance continuous, explainable, and safe—without forcing developers to choose between automation and verification.
