ML Experiment Investigator

Turn ML code, logs and metrics into evidence-linked hypotheses and a clear next experiment with NVIDIA Nemotron on Nebius.

Comment

Inspiration

When a machine-learning experiment behaves unexpectedly, its logs rarely explain why. The clues are scattered across training code, configuration files and metrics, and connecting them takes time.

I wanted to build an assistant that helps organize this evidence and suggest what to test next—while making the difference between observations and hypotheses visible.

What it does

ML Experiment Investigator lets users upload a small project ZIP or load an included sample. It reads supported files as text without executing uploaded code.

The application detects metrics and errors, calculates train–validation gaps from paired JSON accuracy values, and retrieves relevant scikit-learn documentation using Tavily Search and Extract. NVIDIA Nemotron, served through Nebius Token Factory, then produces observations, possible explanations and a proposed controlled experiment.

Users can inspect the metric lines, computed findings, citations and documentation links. The app checks whether cited lines were actually sent to the model, flags blank or punctuation-only citations, and performs limited numeric checks. These checks assist review; they do not prove that an answer is correct.

How we built it

I built the backend in Python and the interface with HTML, CSS and JavaScript.

Nebius Token Factory provides the Nemotron inference used for investigations, letting me focus on the application rather than hosting the model. Tavily retrieves documentation before the model call, with results restricted to scikit-learn.org. Uploaded files and retrieved documentation are treated as untrusted data.

The included synthetic experiment compares an MLP classifier with and without feature scaling. Its recorded validation accuracy increases from approximately 0.507 to 0.840 after scaling. This demonstrates one controlled comparison, not a universal result.

MLflow records the sample results. A separate exporter prepares metadata from other MLflow runs for users to review before uploading, and a trusted local probe can repeat the synthetic comparison without executing uploaded projects.

The hosted demo runs on Render. API keys remain in the server environment, and a separate demo password controls access to investigations.

Challenges we ran into

The biggest challenge was that a plausible explanation is not necessarily supported by evidence.

Testing revealed incorrect calculations, citations pointing to irrelevant lines, and overly confident interpretations of proposed experiments. I added deterministic findings, citation checks and numeric warnings, while keeping their limitations explicit.

I also fixed a Windows-specific SQLite connection cleanup issue and added request limits and credential-pattern checks. These measures help protect the demonstration, but they do not constitute full production security or a guaranteed spending cap.

Accomplishments that we're proud of

I brought the project from a local prototype to a working hosted demonstration with live Nemotron and Tavily calls.

The project has 31 passing automated tests covering evidence handling, citation parsing, access controls, request limits and mocked provider interactions.

Most importantly, the interface makes the investigation inspectable: users can examine the reported results, calculated gaps, reasoning and sources instead of relying on an unexplained answer.

What we learned

I learned that citation existence and factual support are different problems. A model can cite a real line and still make an unsupported claim.

I also learned to separate software testing from model evaluation. Passing unit tests verifies specific application behaviours; it does not establish that every generated diagnosis is correct. The project includes nine synthetic review cases, but these are a starting point for human evaluation, not a completed accuracy benchmark.

What's next for ML Experiment Investigator

I would expand the evaluation with independently reviewed cases, strengthen claim-support checks, improve experiment-data import, and support more metric formats and model types.

The goal is to help users move from an unexpected result to a useful next experiment, while preserving human judgment over the conclusions.

Built With

Share this project:

Updates

Submission history