fixproof: Evidence-driven autonomous software repair

FixProof home page: upload a Python project (zip, GitHub URL, local folder, or sample bugs) to start the verified-fix workflow.
Pick the bug: 4 of 6 tests failing in demo-project, with test_refresh_token_is_accepted selected for repair.
Fix Certificate: “Verified with conditions”, SAST clean, fixed on the first attempt (13 tests, 4,551 Nemotron tokens, 98s).
Evidence ledger: all 6 checks passed, including 5 model-written attack tests run by tools.
Who did what: Nemotron proposed and wrote tests; deterministic tools verified, with limitations and certificate hash shown.
AI explanation of the // → / fix, with the Create Pull Request form ready.
Auto-generated GitHub PR #1 showing root cause, fix, and 4 passed attack tests.

Inspiration

AI coding tools can produce a patch that passes the visible tests and is still wrong. The weak fix satisfies the failing test but breaks the behavior the code is documented to have, and in most tools the model that wrote the fix also judges it. We wanted a repair system where "the AI says it's fixed" is never the evidence.

AI proposes the fix. Evidence earns the merge.

What it does

FixProof takes a failing test and a description of the expected behavior, and returns a patch together with a Fix Certificate that lists exactly what evidence supports the verdict.

Reproduce the original failure. If the bug does not reproduce, the run stops.
Propose a minimal patch with NVIDIA Nemotron. Scope checks reject any edit to tests or config.
Re-run the reproduction test and the full regression suite after every patch.
Scan the patch with a differential security check that reports only the vulnerabilities the patch introduces.
Attack the patch: Nemotron writes adversarial tests, and each one must quote the issue, a docstring or the README. The quote is verified mechanically, so a hallucinated requirement cannot veto a correct fix. The tests then actually execute in the sandbox.
Iterate: a failing, spec-grounded test becomes a counterexample that is fed back to the fixer, with bounded retries and repeated-patch detection.
Verdict: a policy turns the executed evidence into a sealed, tamper-evident certificate. Skipped tests, unparseable output and zero executed tests never count as a pass.

Once a certificate is verified, one click opens a GitHub pull request whose description contains the root cause, the attack-test results and the evidence table, so reviewers see proof and not just a diff. A reviewer can also dispute an attack test and re-verify the same patch without any model call.

FixProof works with Python projects, detected automatically. You can run it on a bundled example, a zip upload, a GitHub URL or a local folder, and a live sync worker re-runs your tests every time you save a file in your editor.

How it works

   Failing test + expected behavior
                │
                ▼
   ┌────────────────────────────┐
   │ 1. Reproduce the failure   │  (deterministic)
   └─────────────┬──────────────┘
                 ▼
   ┌────────────────────────────┐
   │ 2. Nemotron proposes patch │  (MODEL)  ◄──────────────┐
   └─────────────┬──────────────┘                          │
                 ▼                                         │
   ┌────────────────────────────┐                          │
   │ 3. Scope check + apply +   │  (deterministic)         │
   │    regression run          │                          │
   └─────────────┬──────────────┘                          │
                 ▼                                         │
   ┌────────────────────────────┐                          │
   │ 4. Differential security   │  (deterministic)         │
   │    scan                    │                          │
   └─────────────┬──────────────┘                          │
                 ▼                                         │
   ┌────────────────────────────┐                          │
   │ 5. Attack tests: Nemotron  │  (MODEL writes,          │
   │    writes, quote verified, │   code verifies          │
   │    executed in sandbox     │   and executes)          │
   └──────┬───────────────┬─────┘                          │
          │ spec-grounded │ all survive                    │
          │ failure       ▼                                │
          │      ┌────────────────────────────┐            │
          │      │ 6. Verdict policy +        │ (determin- │
          │      │    sealed Fix Certificate  │  istic)    │
          │      └─────────────┬──────────────┘            │
          │                    ▼                           │
          │            GitHub Pull Request                 │
          │                                                │
          └────── counterexample fed back to fixer ────────┘

The model reasons and writes. Plain code runs, checks and decides. The model never decides the verdict.

How we built it

Model: NVIDIA Nemotron through Nebius Token Factory, using an OpenAI-compatible client with JSON validation and retries. The model only reasons, writes patches and writes attack tests.
Deterministic engine: snapshot hashing, reproduction, patch application, regression runs, executing the attack tests and the verdict are plain code, not model output.
Security and delivery: a pure-Python AST scanner for injection, path traversal, unsafe deserialization, hardcoded secrets and weak hashes, and a GitHub Git Data API integration that opens pull requests without a local clone.
Sandbox: a Docker adapter configured with no network, a read-only source mount, a non-root user and dropped capabilities, plus a local process sandbox for development.
Live sync: a watchdog-based CLI with debounce that posts file changes to the backend, which re-runs tests and streams results to the dashboard over WebSocket.
Backend: Python, FastAPI and Pydantic.
Frontend: React, TypeScript, Vite and Tailwind, showing the evidence ledger and the certificate.
Quality: over 350 automated offline tests, and a benchmark harness with held-out oracles and validated bug cases.

Challenges

Making "no evidence" count as no evidence. A run with zero executed tests, a crash or an unparseable report must never look like a pass.
Stopping the model from cheating or hallucinating. Patches cannot touch tests, and attack tests have to quote the spec.
Offline sandbox. Projects can only use the standard library plus a fixed package list, which also shaped how we handle missing imports.
Supporting three languages with one trust model. Each runner reports results in a different format, so we normalized them into one result schema that the verdict policy can judge identically.
Being honest about scope. The README has a status table that marks every capability as done, untested live, or not built.

What we learned

A verdict is only worth as much as the checks behind it. We stopped saying a patch is "correct" and started saying exactly which checks it survived and which it didn't run.

What's next

Property-based tests and mutation testing, Nebius Token Factory sandboxes and serverless jobs for long verification runs, a larger benchmark with published live-model results, and end-to-end repair of arbitrary repositories.