GitTree-CVE

Inspiration

Tools such as Dependabot can update a dependency version, but that does not prove the resulting build is secure. Coding agents have a similar limitation: they often work in one shared tree and produce a patch without demonstrating that the vulnerability can be reproduced before the fix or that it is actually blocked afterward.

There is also a problem when the same agent writes both the fix and the regression test. A test can pass for the wrong reasons—for example, by doing nothing, mocking the vulnerable function, or rejecting legitimate requests along with malicious ones.

GitTree-CVE tackles this by generating several candidate fixes and testing them in isolated sandboxes. It uses four NVIDIA Nemotron model tiers with Nebius Token Factory Sandboxes (ConTree), starting from a shared checkpoint with networking disabled. A patch is only proposed after it passes a sequence of exploit, regression, and audit checks.

What it does

GitTree-CVE takes a CVE or CWE identifier and a vulnerable Python repository. The repository can be a built-in preset or a public GitHub URL. It then runs the following pipeline.

Gather advisory information (Tavily + Nemotron 3.5 Lightning). Tavily Search & Extract retrieves current NVD, GitHub, and vendor advisories. The pipeline filters out library-internal CVEs that are not relevant when the target is an application-level weakness. Nemotron 3.5 Lightning turns the evidence into an execution plan: the vulnerability class informs patch strategies, severity determines the number of parallel branches (B ∈ {3, 4, 5}), and advisory requirements feed into both the prompts and the final audit rubric.
Create a base checkpoint and generate candidate patches (ConTree + Nemotron 3 Nano). GitTree-CVE builds a VM snapshot (result_image_uuid) containing the repository and its dependencies, then disables networking with networking.enabled = false. Nemotron 3 Nano generates several candidate patches and corresponding regression tests in parallel, using different strategies such as strict API migration, defensive input validation, parameterized queries, and cryptographic policy pinning.
Verify each candidate in two phases. Each candidate runs in a ConTree VM cloned from the same base checkpoint:
- Phase A — reproduce the exploit: Run the regression test against the unpatched code. It must fail with evidence of the vulnerability, such as a failed assertion or an exception raised in the target code. Syntax errors, import errors, and missing symbols do not count as exploit evidence.
- Phase B — verify the fix: Apply the patch and require both the new regression test and the repository's existing test suite to pass.
Cross-check with a deterministic attack catalog. Model-written tests can miss important cases. For example, checking whether '..' appears in a filename will not catch every symlink or hardlink traversal in CVE-2007-4559. GitTree-CVE therefore runs deterministic, non-LLM payloads against the unpatched and patched code for path traversal (CWE-22), SQL injection (CWE-89), and JWT signature verification (CWE-347). It also checks legitimate inputs, so a patch cannot pass simply by rejecting everything. If a candidate's own test provides no usable Phase A evidence, the system re-checks it against the validated attack catalog before starting repair rounds.
Audit candidates and escalate when needed (Nemotron 3 Super + Ultra). Nemotron 3 Super scores surviving patches from 0 to 100 against the advisory requirements, cross-validation results, and attack catalog. A score of at least 60 is required. If all Nano branches fail, the orchestrator sends their sandbox stdout and stderr traces to Nemotron 3 Ultra, or to Super in budget-saver mode. If no candidate passes every gate, the result is "No patch proposed."
Open draft pull requests and retain training examples. For imported GitHub repositories, a verified run can open a Draft PR with the minimal code diff, regression test, and evidence ledger. The ledger includes ConTree exit codes, attack catalog results, audit score, and Token Factory completion IDs. Verified patch-and-test pairs can also be formatted into conversational supervised fine-tuning (SFT) datasets and uploaded to Nebius Token Factory through /v1/files and /v1/fine_tuning/jobs.

A candidate patch pᵢ and regression test tᵢ are eligible for selection only when they meet all four conditions:

Verified(pᵢ, tᵢ)  ⟺  Fail_evidence(C₀, tᵢ)
                  ∧  Pass(C₀ ⊕ pᵢ, T_repo ∪ {tᵢ})
                  ∧  ∀ a ∈ A_repro : Blocked(C₀ ⊕ pᵢ, a) ∧ Control(C₀ ⊕ pᵢ)
                  ∧  S_Super(pᵢ) ≥ 60

Here, C₀ is the unpatched codebase checkpoint, A_repro is the set of deterministic attack payloads that reproduce on the original code, and S_Super(pᵢ) is the Nemotron 3 Super audit score.

How we built it

Nebius Token Factory inference. We built a multi-tier model router in app/nemotron_router.py using the OpenAI-compatible endpoint at https://api.tokenfactory.nebius.com/v1. It routes work across four NVIDIA open-weight models:
- nvidia/Nemotron-3_5-Lightning handles advisory extraction and strategy planning. Listed pricing: $0.06 per 1M input tokens and $0.24 per 1M output tokens.
- nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B generates candidate patches, repairs failed branches, and helps synthesise attack-catalog adapters. Listed pricing: $0.06 per 1M input tokens and $0.24 per 1M output tokens.
- nvidia/nemotron-3-super-120b-a12b performs rubric-based security audits and budget-saver escalation. Listed pricing: $0.30 per 1M input tokens and $0.90 per 1M output tokens.
- nvidia/Nemotron-3-Ultra-550b-a55b handles deeper, multi-trace investigations when all Nano branches fail. Listed pricing: $1.00 per 1M input tokens and $3.00 per 1M output tokens.
Nebius Token Factory Sandboxes (ConTree). We implemented an asynchronous REST client in app/contree_client.py against https://api.tokenfactory.nebius.com/sandboxes/v1. It supports OCI image import, checkpoint chaining through result_image_uuid, disposable and retained VM instances, and network isolation using networking.enabled = false. An in-guest NIC probe checks that the isolation is in place. We also built a local Docker fallback using docker commit, --network none, and dropped capabilities, with the same checkpoint workflow.
Tavily Search & Extract. The integration in app/tavily_intel.py retrieves CVE/CWE information at runtime. That information influences the number of branches, patch strategies, target version, and audit rubric.
Backend and frontend. The backend uses FastAPI, httpx, and sse-starlette to stream updates through Server-Sent Events (SSE). The dashboard is built with Next.js 16, React 19, and Tailwind CSS v4. It includes a replay mode that can show recorded ConTree runs without requiring an API key. The project has 1,139 automated tests: 1,098 backend pytest tests and 41 frontend Node tests.

Challenges we ran into

Preventing models from gaming the tests. When asked to make tests pass, an unconstrained coding model may modify conftest.py, remove constraints from requirements.txt, or shadow a standard-library or third-party module—for example, by adding a root-level jwt.py that shadows PyJWT. It may also write a test that fails in Phase A because its own patch introduced a missing symbol. We added patch_policy.py and a Phase A pytest report parser to reject structural and import failures, prevent module shadowing across different sys.path layouts, and use runtime profiling (sys.setprofile) to check that attack-catalog adapters execute the repository's code rather than reimplementing the logic inline.
Handling weak model-written tests. In our live CWE-347 JWT signature-verification runs, all five Nano branches produced patches, but their tests either crashed against the unpatched code or failed in the same way before and after the fix. Rather than discard every candidate, we added an Attack Catalog Re-Check. When a candidate's test gives no valid Phase A evidence, the orchestrator uses the validated deterministic attack catalog as the regression suite. This recovered all five branches in that run.
Catching incomplete fixes. In live runs for CVE-2007-4559, a path-traversal issue in Python's tarfile handling, several branches added checks such as '..' in member.name. Those checks passed their own tests, but the attack catalog still reproduced traversal through symlinks and hardlinks. The system rejected those patches in favour of using filter='data'.
Working around SDK limitations and token budgets. The ConTree Python SDK (0.3.6) did not expose a way to disable networking on Image.run(), so we used the REST API directly to set networking.enabled = false. We also found that Nemotron-3_5-Lightning, unlike Nemotron-3-Nano, enables chain-of-thought reasoning by default. With a 700-token output budget, the model initially used the budget on reasoning_content before returning JSON. Setting extra_body={"chat_template_kwargs": {"enable_thinking": false}} alongside response_format={"type": "json_object"} produced structured outputs more reliably across the model tiers.

Accomplishments that we're proud of

Low cost per run. In our recorded live ConTree runs, a CVE run cost between $0.0018 and $0.0026. We routed patch generation and repairs to Nemotron 3 Nano and reserved Super and Ultra for auditing and escalation; the UI records the corresponding completion ID.
Checkpoint-based sandbox testing. Preset replays and demo runs use network-isolated ConTree microVMs cloned from a shared dependency checkpoint. This lets us test whether an exploit works before the patch and whether it is blocked afterward.
Multiple layers of verification. The pipeline combines Phase A/B pytest checks, patch-policy validation, cross-checks across candidate branches, runtime-profiled attack payloads, and a Nemotron Super audit. The test suite includes 1,098 offline backend unit and integration tests and 41 frontend tests.
Repository-to-PR workflow. GitHub repositories can be imported in memory with zip-bomb and decompression safeguards. When a candidate passes verification, the system can open a Draft PR with the diff and its sandbox evidence.

What we learned

Several small-model candidates can be more useful than one large-model attempt. Generating three to five patches with NVIDIA-Nemotron-3-Nano-30B-A3B (3B active parameters) and checking them in deterministic ConTree gates was faster, cheaper, and more reliable in our tests than asking a frontier model for one patch and trusting the result.
Patch generation and exploit verification should be separate steps. Models can often produce a small security fix, but writing a robust exploit test is a different task. Lightweight model-generated adapters, checked through runtime profiling and paired with deterministic payloads, worked better for our pipeline.
Reasoning settings need to match the task. Setting enable_thinking by workflow stage in Nebius Token Factory helped keep planning and code generation efficient while leaving more reasoning capacity for audits and escalation.

What's next for GitTree-CVE

Expand the attack catalog. Add deterministic payloads and legitimate-use controls for SSRF (CWE-918), insecure deserialization (CWE-502, including pickle and yaml), XXE (CWE-611), and command injection (CWE-78).
Support more languages. Extend the checkpoint and Phase A/B runner beyond Python and pytest to Node.js/TypeScript (vitest or jest), Go (go test), and Rust (cargo test) environments in ConTree.
Complete the Nemotron LoRA loop. If Nemotron Nano and Super become supported targets for LoRA fine-tuning in Token Factory, move the automated /v1/fine_tuning/jobs workflow from Qwen3-Coder-30B-A3B to a fully Nemotron-based remediation loop.
Add GitHub App and CI integration. Package GitTree-CVE as a GitHub App that automatically starts ConTree sandbox runs when a Dependabot alert or SARIF code-scanning finding is opened.