-
-
Hidden cohort regression
-
Cache correctness experiment
-
Project overview
Inspiration
A faster implementation can return the wrong answer. A model with a better overall score can perform worse for a particular group of users.
ML Analyser started with a practical question: how can developers check whether a proposed improvement deserves to be accepted?
The result is a developer tool that connects AI suggestions with measurable experiments, explicit correctness checks and evidence that can be replayed.
What it does
ML Analyser provides three connected workflows:
Counterexample Lab: NVIDIA Nemotron reads the included Python pricing-cache source and proposes a cache configuration. The developer reviews and approves the experiment. A deterministic verifier then compares the original and candidate configurations, checking both calculation counts and price correctness. Failing queries become concrete feedback for subsequent suggestions, within a bounded three-attempt campaign.
Prediction Comparison: Developers upload a CSV containing known answers and paired baseline/candidate predictions from their own classification or regression models. The app calculates metrics including F1, precision, recall, MAE and RMSE. Regression Radar checks individual cohorts and class recall, while an approximate paired bootstrap interval helps identify uncertain improvements. This workflow evaluates supplied predictions without training a model or calling an AI provider.
Configured repository experiments: The workbench imports public GitHub repositories for read-only inspection. Supported evaluation manifests define how to run ML training, HTTP or process benchmarks. Execution requires approval of the exact plan and uses separate baseline and candidate copies.
What the examples demonstrate
In a captured live Nemotron experiment, the accepted cache configuration reduced reference lookups from 120 to 30, with zero incorrect prices. A separately labeled product-only negative control reduced work further but returned incorrect prices and was rejected. These are calculation counts from a synthetic workload, not production latency or cost savings.
The prediction-comparison example demonstrates another failure mode: overall accuracy improves from 78% to 90%, while mobile accuracy falls from 90% to 60%. The cohort guardrail rejects the candidate despite its improved overall score.
Both workflows let users download evidence and independently recompute the results.
How it was built
The backend uses Python, FastAPI, Pydantic and SQLite. The interface uses JavaScript, HTML, CSS and a Three.js landing-page visualization. GitHub OAuth controls access to the repository workbench, while the public Lab and prediction comparison provide accessible demonstrations.
NVIDIA Nemotron 3 Super 120B-A12B runs through the Nebius Token Factory inference API. It supplies source-grounded engineering hypotheses and cache-configuration suggestions. We use prompting and structured-output validation rather than fine-tuning.
The model proposes; Python executes the configured measurements; deterministic rules decide acceptance. The web application is deployed on Render, while model inference is served by Nebius.
Challenges
The hardest challenge was keeping suggestions, measurements and decisions clearly separated. A convincing explanation must never appear as an executed result.
Other challenges included binding approval to the exact experiment, preserving original repositories, making failures understandable, and distinguishing new AI runs from scripted examples and saved evidence.
Accomplishments and lessons
The project includes verified live Nebius inference, replayable experiment reports, cohort-level regression checks and accepted, rejected and inconclusive outcomes. Validation includes 206 passing backend tests on Linux and 34 frontend tests.
The central lesson is that a useful engineering agent needs clear ways to fail. Counterexamples, correctness limits and reproducible measurements make its recommendations easier to assess.
What's next
Next steps include broader benchmark adapters, experiments with real user datasets, repeated ML training runs and production worker isolation. Arbitrary repository execution still requires a supported evaluation setup; importing code alone cannot establish its correctness or model quality.
Log in or sign up for Devpost to join the conversation.