Inspiration

AI coding agents can generate patches and run tests, but the evidence created during one repair is rarely preserved in a form that another agent can independently execute.

Repository instructions, agent memory, and rules files can preserve written knowledge. However, written knowledge is different from executable proof.

For example, a note may say:

Timed signatures must reject values older than max_age.

A future agent can read that instruction, but it still does not know:

which regression originally proved the behavior;
whether the regression was modified;
which environment produced the result;
whether independent tests passed;
whether the new revision still preserves the behavior.

This inspired Issue2Proof, a regression-memory system that combines NVIDIA Nemotron models with isolated Nebius Token Factory Sandboxes.

The central idea is:

Regression memory for AI coding agents that survives the agent.

Issue2Proof turns a verified investigation into an executable, hash-backed Memory Capsule. A future coding agent can recall that capsule before editing and replay it afterward, regardless of which model or IDE performs the change.

What Issue2Proof does

A user provides:

a public GitHub repository or ZIP archive; and
a description of the behavior that should remain true.

Issue2Proof then performs a complete evidence workflow:

Inspect the repository.
Identify implementation files, tests, and dependency manifests.
Establish the original baseline in a Nebius Sandbox.
Ask NVIDIA Nemotron to generate a focused regression.
Confirm that the regression exposes the original problem.
Generate bounded repair candidates.
Execute the regression, existing tests, and withheld tests in fresh Sandboxes.
Accept a repair only when every required gate passes.
Store the result as a hash-backed Memory Capsule.
Recall and replay that capsule when a future agent modifies related code.

The result of a future replay is one of:

preserved
violated
uncertain

Issue2Proof does not replace an AI coding assistant. It provides an independent evidence and memory layer that can work with Codex, Claude Code, Cursor, CI systems, and other MCP-compatible tools.

How NVIDIA Nemotron is used

NVIDIA Nemotron is not used as a single generic chatbot. Issue2Proof assigns different Nemotron models specific responsibilities within the software-engineering workflow.

NVIDIA Nemotron Super

nvidia/nemotron-3-super-120b-a12b is the main engineering model.

It is used to:

translate a behavior description into a focused regression;
analyze repository context;
reason about the original failure;
produce structured engineering decisions;
explain conflicts between a current revision and historical memory;
identify likely causes and propose the next verification experiment.

The regression-generation stage is especially important. Issue2Proof does not ask Nemotron only to propose a patch. Nemotron first helps define what must be proven.

The generated regression is then executed against the original repository in a Nebius Sandbox. It cannot become evidence unless it demonstrates the expected failure.

NVIDIA Nemotron Nano

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B is used for economical memory indexing.

After a repair is verified, Nano generates:

a concise behavioral summary;
retrieval keywords;
affected concepts;
future-change triggers;
compact agent-facing memory metadata.

This allows future agents to recall relevant evidence without sending the complete investigation to a larger model.

NVIDIA Nemotron Ultra

nvidia/Nemotron-3-Ultra-550b-a55b is reserved for difficult, high-risk conflicts.

Ultra is not called for every investigation. It is used only when engineering analysis identifies ambiguity that justifies deeper reasoning.

Examples include:

a current change intentionally modifying a historical contract;
multiple Memory Capsules appearing to conflict;
incomplete evidence that cannot be resolved by ordinary replay;
a high-risk change affecting security or compatibility boundaries.

This conditional escalation protects the limited project budget.

Supporting Token Factory models

Additional models available through Nebius Token Factory provide specialized supporting roles:

Kimi K2.7 Code acts as a bounded coding challenger for repair generation.
Qwen3 Embedding performs semantic Memory Capsule retrieval.

Nemotron remains responsible for the core regression, engineering, indexing, and conflict-analysis workflow. The supporting models are used where their specialized capabilities add value.

How Nebius Token Factory is used

Issue2Proof accesses the models through Nebius Token Factory’s OpenAI-compatible inference API.

The application records:

model name;
assigned role;
prompt-token count;
completion-token count;
finish reason;
estimated cost;
structured response status.

Model responses are not trusted automatically.

Every response passes through:

JSON structure validation;
output-size limits;
Python syntax parsing;
source-path restrictions;
protected-test enforcement;
truncation detection;
model-response fingerprinting.

Invalid or incomplete responses are rejected before execution.

How Nebius Sandboxes are used

Nebius Sandboxes are the execution foundation of Issue2Proof.

Uploaded repositories, build hooks, tests, and model-generated code never execute directly on the application host.

The default environment starts from:

python:3.12-slim

Issue2Proof resolves the image through the Nebius Sandbox service and records the resulting environment information for the investigation.

Separate dependency preparation

Dependencies may require network access, while untrusted repository code should not receive network access.

Issue2Proof separates these activities.

The dependency-preparation Sandbox:

receives dependency manifests without repository source;
installs supported published binary wheels;
prepares a reusable dependency environment;
records resolved versions;
does not execute repository code.

The verification Sandboxes then:

receive the prepared dependency environment;
receive the bounded repository files;
disable networking;
install the project offline;
execute pytest as an unprivileged user;
enforce time, process, memory, file, and output limits.

Original baseline Sandbox

The first execution establishes whether the repository can run before any repair is attempted.

This distinguishes:

an actual behavioral failure;
an existing failing suite;
dependency setup failure;
pytest collection failure;
incomplete execution evidence.

Regression-reproduction Sandbox

The generated regression is executed against the original revision.

A regression is useful only when it demonstrates the reported problem. If it passes against the original defective behavior, Issue2Proof rejects it as weak evidence.

Candidate verification Sandboxes

Repair candidates are evaluated using fresh isolated operations.

The checks can include:

generated regression;
ordinary existing tests;
withheld independent tests;
protected source and test hashes;
skip and test-count validation.

A candidate is accepted only when all selected checks pass.

Memory-replay Sandbox

When a future agent modifies related code, Issue2Proof recalls the relevant Memory Capsule and runs its frozen regression in a fresh Sandbox.

This is what changes the capsule from passive documentation into executable memory.

Sandbox evidence

Each operation records information such as:

Sandbox operation UUID;
image identity;
command;
exit code;
timeout state;
stdout and stderr;
output truncation state;
execution duration;
network-disabled state;
disposable-operation state;
resource cost.

The user can inspect this execution timeline in the web interface.

Memory Capsules

A verified investigation can be converted into a Memory Capsule containing:

repository identity;
behavioral claim;
affected paths;
assumptions;
risk boundaries;
invalidation conditions;
frozen regression source;
SHA-256 regression hash;
Nemotron-generated recall index;
Sandbox operation identifiers;
original verification results;
environment provenance;
capsule version;
supersession history.

A simplified example is:

{
  "claim": "Timed signatures reject values older than max_age.",
  "repository_key": "pallets/itsdangerous",
  "scope": {
    "paths": ["src/itsdangerous/timed.py"]
  },
  "evidence": {
    "verdict": "checks_passed",
    "regression": {
      "path": "issue2proof_generated/test_regression.py",
      "sha256": "..."
    },
    "operation_ids": ["..."]
  },
  "version": 1
}

SHA-256 protects content integrity. Protected deployments can also authenticate portable bundles using HMAC-SHA256.

Model and Sandbox workflow

Repository + expected behavior
              ↓
        Repository inspection
              ↓
      Nebius baseline Sandbox
              ↓
 Nemotron Super regression generation
              ↓
  Nebius regression reproduction
              ↓
 Kimi/Nemotron repair candidates
              ↓
 Fresh network-disabled Sandboxes
              ↓
 Regression + existing + withheld tests
              ↓
       Verified Memory Capsule
              ↓
  Nemotron Nano memory indexing
              ↓
   Lexical or semantic future recall
              ↓
      Fresh Sandbox memory replay

How I built it

The application uses:

Python;
FastAPI;
Pydantic;
SQLite;
HTTPX;
pytest;
JavaScript;
HTML and CSS;
Docker;
Nebius Token Factory;
Nebius Sandboxes;
Contree SDK;
MCP;
GitHub Actions;
Render and Vercel.

FastAPI exposes the web and REST interfaces. A background worker manages long-running investigations, checkpoints, Sandbox operations, cancellations, and resumable jobs.

SQLite stores:

investigation reports;
job states;
workflow events;
cost reservations;
Memory Capsules;
embeddings and recall metadata.

The frontend shows:

repository import;
source and test selection;
live job progress;
model-routing evidence;
Sandbox operation timelines;
generated patches;
Memory Capsule creation;
semantic and lexical recall;
historical memory replay.

Integrations

Issue2Proof is available through more than the hosted website.

MCP

The MCP server exposes:

recall_repository_memory
audit_repository_revision

This allows a compatible AI coding agent to complete:

recall → edit → audit

The agent receives verified behavioral constraints before editing. After editing, Issue2Proof uploads the bounded revision and replays the selected regressions in Nebius Sandboxes.

REST API

A vendor-neutral context endpoint returns claims, scope, assumptions, boundaries, and regression hashes.

GitHub Action

The pull-request Memory Gate can recall capsules from changed paths and optionally request remote Sandbox replay.

Portable bundles

Memory Capsules can be exported as integrity-checked, authenticated bundles and imported into trusted Issue2Proof deployments.

Challenges I faced

Safely executing model-generated code

Model-generated regressions and patches cannot be trusted merely because they came from a capable model.

Nebius Sandboxes solved the main isolation problem, but the controller still required:

bounded file selection;
path validation;
hash verification;
output limits;
timeouts;
network control;
operation ownership;
disposable environments.

Separating dependency installation from untrusted execution

Many Python projects require packages from PyPI. Allowing repository code to execute in a network-enabled environment would increase risk.

The solution was to prepare binary-wheel dependencies in a separate Sandbox containing no repository source, then execute the project offline in another Sandbox.

Distinguishing setup failure from repair failure

A pytest command can fail because of:

an incorrect import path;
missing dependencies;
collection errors;
unsupported packaging;
an actual assertion failure.

Issue2Proof records these as different evidence states instead of labeling every nonzero exit code as a failed repair.

Maintaining independent verification

A repair model should not see every test used to evaluate its output.

Issue2Proof allows selected tests to remain outside model context. These withheld tests execute only after the candidate has been generated.

Handling incomplete model responses

Models can return invalid JSON, truncated output, invalid Python, or unrelated modifications.

I added strict structured-output validation, source parsing, context limits, finish-reason checks, protected tests, and fallback routing.

Staying within a \$25 budget

The limited credit allowance required deliberate model and Sandbox routing.

Before every paid operation, Issue2Proof records a conservative reservation.

If (B) is the total budget and (r_i) is the recorded cost of operation (i), a new operation is permitted only while:

$$ \sum_{i=1}^{n} r_i + r_{\text{next}} \leq B $$

For this deployment:

$$ B = \$25 $$

The application also uses:

Nemotron Nano for lower-cost indexing;
Nemotron Super for ordinary engineering work;
conditional Nemotron Ultra escalation;
batched embedding requests;
cached embeddings;
free lexical retrieval;
per-operation cost reconciliation.

Evaluation

I created a reproducible 30-trial known-incident benchmark across four verified bug histories.

Eight trials deliberately introduced regressions related to previously repaired behavior.

Evaluation method	Regressions detected
Plain existing CI	3 of 8
Memory Capsule replay in Nebius Sandboxes	8 of 8

The corresponding rates were:

$$ \text{Plain CI detection rate} = \frac{3}{8} \times 100 = 37.5\% $$

$$ \text{Memory Capsule detection rate} = \frac{8}{8} \times 100 = 100\% $$

Query-only and changed-path-aware retrieval selected the intended capsule in 12 of 12 authored trials.

These are controlled known-incident experiments rather than a universal production benchmark. The artifacts and methodology are included in the repository.

What I learned

The main lesson was that model intelligence and execution evidence solve different parts of the problem.

Nemotron can:

interpret the behavioral requirement;
generate a focused regression;
analyze failures;
index verified memory;
explain conflicts.

Nebius Sandboxes can then determine what actually happens when the generated artifacts execute.

Combining these capabilities is more useful than asking a model to both propose a solution and declare that its own solution is correct.

I also learned that:

Regression memory needs executable evidence, not only prose.
Environment provenance is part of the evidence.
Withheld tests improve independence.
Setup failures must be separated from behavioral failures.
Model routing is essential when credits are limited.
Every verdict should link to commands, hashes, operation IDs, and results.
Repository memory should remain useful across models and IDEs.

What's next

Future work includes:

JavaScript, TypeScript, Go, Java, and Rust support;
persistent managed database storage;
GitHub App installation;
automatic pull-request comments;
larger production-history evaluations;
repository-specific environment adapters;
provider-reported billing synchronization;
organization-level access controls;
direct IDE extensions.

Links

Live application: https://issue2proof-live.onrender.com
Interactive tour: https://issue2proof-demo.vercel.app
Source code: https://github.com/kushalar03/issue2proof

Built With

docker
embeddings
fastapi
github
html5
httpx
javascript
mcp
pydantic
pytest
python
rag
render
rest
sqlite
uvicorn
vercel

Updates

Kushala R started this project — Oct 29, 2026 03:49 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.