Next Experiment

Inspiration

A fine-tuning job can finish with status succeeded and still produce a model that fails at its task. The next step is often another training run with a different learning rate or more data. Each attempt costs money, and an improvement can leave the original problem unexplained.

Next Experiment grew out of a question: could an agent choose a small experiment that tells us what to try next, then test the proposed fix before committing to a full retrain?

What it does

Next Experiment investigates failed LLM fine-tunes. It reads the training recipe and available evidence, ranks likely causes, and chooses checks by how much uncertainty they are expected to remove per dollar.

When it finds a candidate fix, it trains that fix alongside a control and evaluates both on a separate check set. The report shows the measured effect, uncertainty, and cost. If the evidence is inconclusive, the result stays inconclusive. A full retrain requires approval.

How we built it

The agent runs in Python behind a FastAPI service, with progress streamed to a browser interface. NVIDIA Nemotron 3 Super ranks hypotheses; a planner selects the next check. Free probes inspect the training configuration, data, and adapter weights when available. Tavily supplies reports of known issues.

For experiments that require training, the agent submits paired LoRA jobs to Nebius Token Factory, downloads the adapters, and scores them on separate examples. Code applies the statistical decision rules, and Nemotron 3 Ultra writes the verdict from those results. Paid steps have budget checks and approval gates.

We built fault-injected cases and compared the agent with a single model call, a ReAct agent using the same tools, and a fixed checklist. We published the cases and results in the FT-Fail dataset so the comparisons can be inspected.

Challenges we faced

An improvement does not always identify the cause. In our benchmark, increasing adapter capacity could partly compensate for a learning rate that was too low. The checklist sometimes confirmed that second explanation as well. Choosing a control that distinguishes competing explanations became a central part of the project.

We also found that changing the seed parameter on Token Factory produced identical adapters in our runs. We had to shuffle the training files to introduce variation between repeats. Otherwise, repeated jobs would give a misleading impression of independent evidence.

Adapter loading caused another problem: the downloaded weights lacked a prefix expected by PEFT. The default loading path could leave random weights in place with only a warning. We added explicit handling for those weight names before trusting the evaluation results.

Results

In the saved demo, a Qwen3-0.6B fine-tune answered 14% of 200 held-out queries correctly. The agent investigated a learning rate of 1e-5, tested 1e-4 against the unchanged recipe, and proposed a retrain. The retrained model reached 42.5% on those queries. Estimated model-call and training cost was about $0.075.

Across 13 Token Factory cases, Next Experiment ranked the planted cause first in all nine faulty cases and raised no false alarms on four healthy cases. Average estimated cost was $0.144 per case, compared with $0.205 for ReAct using the same tools. All nine faulty cases involved a low learning rate, so this benchmark measures one failure family.

What we learned

Diagnosis and recovery need separate measurements. Ranking the cause correctly did not guarantee that the tested fix restored performance: the agent met the benchmark’s fix-success criterion in six of the nine faulty cases.

We also learned how much an investigation depends on the evaluation setup. Adapter loading, variation between training runs, and the choice of control can change what a result means. The next work is to expand the end-to-end benchmark beyond learning-rate failures, using fresh held-out cases for each failure family.

Built With

• css
• docker
• fastapi
• html
• hugging-face-transformers
• javascript
• lora
• nebius-ai-cloud
• nebius-token-factory
• numpy
• nvidia-nemotron-3-super
• nvidia-nemotron-3-ultra
• peft
• python
• pytorch
• qwen3
• scikit-learn
• sqlalchemy
• sqlite
• tavily