ReproFix AI

Architecture Diagram

ReproFix AI — Reproduce the Result. Prove the Fix.

Autonomous ML debugging and reproducibility powered by NVIDIA Nemotron 3 on Nebius Token Factory.

A language model proposes — deterministic code decides.

Inspiration

Machine learning research has a reproducibility problem.

A GitHub repository may work on one machine but fail on another because of missing dependencies, incompatible libraries, runtime errors, incorrect configurations, or numerical discrepancies. Even when code executes, reproducing the results claimed in a research paper can require hours of manual investigation.

Existing AI coding assistants can suggest changes, but a convincing explanation is not proof that a fix works.

We built ReproFix AI to close that gap: an autonomous engineering system that investigates ML repositories, proposes targeted repairs, executes experiments, and independently verifies outcomes using real evidence. Our goal is to make ML debugging more reproducible, transparent, and trustworthy.

What it does

ReproFix AI transforms debugging into an evidence-driven workflow:

Reproduce → Diagnose → Repair → Verify → Report

Users provide a public GitHub repository, an optional expected metric or research paper, and an investigation goal.

ReproFix then:

Inspects the repository: Identifies source files, dependencies, frameworks, and possible entry points.

Builds the environment: Installs dependencies and prepares an execution environment.

Runs the baseline: Executes the original project and captures logs, errors, exit codes, and measurable results.

Diagnoses failures: Uses specialized AI agents to investigate exceptions, dependency problems, and metric discrepancies.

Generates repairs: Produces targeted source-code patches rather than unnecessary rewrites.

Tests experiments: Applies candidate patches and executes the modified repository.

Verifies outcomes: Uses deterministic checks to decide whether a change actually improves the result.

Produces evidence: Generates execution reports, patch diffs, model usage statistics, and an interactive causal evidence graph.

Unsuccessful experiments are rejected and reverted. ReproFix does not claim success merely because an AI model says a patch should work.

How we built it

Multi-Agent Engineering Architecture

ReproFix uses a Python orchestration engine coordinating five specialized agents:

Analyzer: Inspects repository structure, source code, and execution context.

Planner: Organizes investigation steps and selects debugging strategies.

Diagnoser: Analyzes failures and generates root-cause hypotheses.

Repairer: Produces minimal candidate code patches.

Reviewer: Reviews proposed changes before execution.

The orchestrator manages experiment limits, execution budgets, model usage, state transitions, and evidence collection.

NVIDIA Nemotron 3 + Nebius Token Factory

The AI reasoning layer uses NVIDIA open-source Nemotron models served through Nebius Token Factory.

Instead of sending every task to the largest model, ReproFix implements task-aware routing:

Model

Primary responsibility

Nemotron 3 Nano

Repository summarization, log digestion, and classification

Nemotron 3 Super

Investigation planning, crash diagnosis, code repair, and review

Nemotron 3 Ultra

Complex reasoning, behavioral discrepancies, and escalation after rejected hypotheses

The model router supports fallback behavior and records model selections, token consumption, and inference costs.

This architecture enables experimentation with different model tiers while keeping the system observable.

Independent Deterministic Verification

This is ReproFix's central innovation.

AI agents propose hypotheses and patches. Deterministic execution decides whether they work.

The verification framework evaluates:

Whether the application executes successfully.

Whether measured numerical results satisfy the requested tolerance.

Whether results remain stable across clean re-runs.

Whether generated patches apply correctly.

Whether protected files remain unchanged.

Whether expected results have been hardcoded into patches.

Whether a candidate patch produces measurable progress.

For a requested metric, the comparison follows:

abs(measured - expected) <= tolerance

A patch is not accepted merely because the model provides a persuasive explanation. The system checks actual observations and reverts changes that do not improve the experiment.

Execution and Sandboxing

ReproFix provides a sandbox abstraction with Docker-based execution support and a separate local development backend.

The system can install dependencies, run baseline commands, execute patched code, capture logs, and collect execution evidence.

The Docker backend supports stronger isolation controls. The local development backend is explicitly identified as unsafe for untrusted repositories and is intended for development and controlled demonstrations.

Developer-Focused Web Application

The application uses FastAPI, HTML, CSS, and JavaScript to provide an interactive engineering workspace.

Its features include:

Repository investigation and configuration.

Live investigation progress.

Searchable history of previous runs.

Execution logs and terminal output.

Side-by-side and unified patch diffs.

Interactive causal evidence graph.

Numerical metric visualization.

Model usage and inference-cost reporting.

Downloadable evidence-backed reports.

The interface is designed to make AI engineering decisions inspectable rather than hiding them behind a single generated answer.

Supporting Technologies

ReproFix also integrates Tavily for optional external evidence search, GitHub-based repository workflows, SQLite-backed run state, and a benchmark framework called ReproBench.

Results and Demonstration

We evaluated ReproFix through automated tests, benchmark tasks, and real investigations using NVIDIA Nemotron on Nebius Token Factory.

Real AI Investigation

In one recorded pose-estimation investigation, ReproFix executed the target repository and attempted to reproduce a specified validation accuracy.

Recorded results:

Target metric: val_accuracy = 0.87 ± 0.02

NVIDIA Nemotron calls: 32

Model inference cost: Approximately $0.4062

Sandbox execution time: Approximately 6 minutes and 9 seconds

Experiment 1: Produced a metric value of 0.4106, outside the requested tolerance

Another experiment: Introduced a runtime crash

Final outcome: Not repaired; unsuccessful changes reverted

The system preserved the execution evidence and correctly refused to mark the investigation as successful.

This demonstrates an essential property of ReproFix: failed experiments remain failures, even when an AI model generated the proposed fix.

ReproBench — 30 Reproducibility Tasks

ReproFix includes ReproBench, a suite of 30 ML debugging tasks covering dependency failures, training issues, preprocessing, evaluation, configuration, checkpoint loading, architecture errors, and other realistic failure categories.

The scripted reference-fix evaluation validates the benchmark pipeline across all 30 tasks.

Importantly, this result is not presented as a 100% autonomous Nemotron repair rate. Scripted reference-fix verification and live AI-agent performance are separate measurements.

Additional Documented Repair

The repository also documents a pose-estimation debugging example involving an uninitialized landmarks variable, a minimal one-line patch, and successful re-execution.

These examples illustrate both the potential of autonomous repair and the importance of verifying the final result against the specific user-requested objective.

Challenges we ran into

Reproducing arbitrary ML environments

Different repositories depend on incompatible Python packages, framework versions, datasets, hardware configurations, and execution commands.

We addressed this through repository inspection, dependency handling, configurable execution, and structured failure reporting.

Preventing false-positive repairs

A language model can generate plausible code without proving that it fixes the problem.

We separated AI-generated proposals from deterministic verification, including patch guards, numerical checks, execution outcomes, and rollback rules.

Managing reasoning cost

Complex debugging can require repeated investigation and model calls.

We implemented tiered Nemotron routing, token accounting, cost reporting, experiment limits, and escalation strategies.

Handling unsuccessful investigations

Not every repository can be repaired within the available time, dependencies, or experimental budget.

ReproFix preserves rejected hypotheses, failed experiments, logs, and final stop reasons so users can understand what happened and continue investigating.

Accomplishments that we're proud of

Our biggest accomplishment is building ReproFix AI into a functional, end-to-end agentic engineering platform that goes beyond generating code suggestions. It investigates real machine learning repositories, executes experiments, proposes targeted repairs, and independently verifies outcomes using execution evidence.

Key accomplishments include:

Built a five-agent debugging system: Developed specialized Analyzer, Planner, Diagnoser, Repairer, and Reviewer agents coordinated through an autonomous orchestration engine.

Integrated NVIDIA Nemotron with Nebius Token Factory: Implemented intelligent routing across Nemotron 3 Nano, Super, and Ultra, selecting models based on task complexity while tracking token usage and inference costs.

Developed a seven-point deterministic verification framework: Created independent checks for execution success, numerical accuracy, reproducibility, patch integrity, protected files, hardcoded results, and measurable progress. AI-generated confidence never determines success.

Created ReproBench with 30 reproducibility tasks: Built a benchmark covering realistic ML failures, including dependency conflicts, training errors, preprocessing issues, and configuration problems. All 30 tasks passed scripted reference-fix verification.

Demonstrated real Nemotron-powered investigations: Executed investigations against real GitHub repositories, including a recorded run with 32 model calls costing approximately $0.41. The system rejected unsuccessful patches and produced an evidence-backed final report instead of falsely claiming success.

Built a complete developer interface: Implemented an interactive dashboard featuring live execution progress, terminal logs, code diffs, numerical metrics, past investigations, and causal evidence graphs.

Delivered a publicly accessible application: Deployed ReproFix AI on Render, published the source code under the MIT License, and documented installation, testing, and usage.

What we're most proud of is the system's commitment to trustworthy AI engineering: a language model proposes, but deterministic code decides. ReproFix demonstrates that autonomous coding agents can be designed to explain their decisions, verify their work, and acknowledge when an experiment fails.

What we learned

Building ReproFix reinforced several important lessons:

AI reasoning and execution verification should be separate systems. Smaller and larger language models can play complementary roles in an agentic workflow. Real execution evidence is more valuable than model-generated confidence. Failed experiments provide useful information when they are recorded and explained. A usable developer tool needs observability, reproducibility, and clear reporting—not just an LLM-powered interface.

What Makes ReproFix Different?

Traditional AI coding assistants primarily generate suggestions or patches.

ReproFix focuses on the full experimental lifecycle: Repository → Baseline → Hypothesis → Patch → Execution → Verification → Evidence

Its differentiators include:

Multi-agent debugging powered by NVIDIA Nemotron. Task-aware model routing through Nebius Token Factory. Independent deterministic verification. Automatic rejection and rollback of unsuccessful changes. Reproducibility-oriented numerical metric checks. Interactive evidence graphs and transparent cost tracking. Structured reporting of both successful and unsuccessful outcomes.

What's next for ReproFix AI

Future improvements include broader ML framework coverage, stronger automated environment recovery, larger-scale live Nemotron benchmark evaluation, GPU-backed execution, and deeper integration with continuous integration workflows.

We also plan to improve benchmark comparisons between different Nemotron routing strategies and expand automated repair coverage for complex ML research repositories.

Built for the Nebius x NVIDIA Global AI Hackathon

Track: Coding and Agentic Engineering

NVIDIA technologies: Nemotron 3 Nano, Super, and Ultra

Nebius infrastructure: Nebius Token Factory

Live Demo: https://agentic-reprofix.onrender.com

Source Code: https://github.com/Ritik7nehra/Agentic-Reprofix

Demo Video: https://youtu.be/aNwI9mMWnks

Core principle: A language model proposes — deterministic code decides.