Inspiration

While learning ML, we saw models score very high in testing and then fail on real data. The usual cause is data leakage, like scaling before the train/test split. The code runs fine, so normal reviewers never flag it and the results quietly lie. We built MirageML to catch these bugs, explain them, and show the damage.

What it does

MirageML reviews Python ML scripts and finds hidden pipeline bugs that quietly inflate accuracy. It catches fitting a scaler before the train/test split, fitting on test data, resampling before the split, target leakage, tuning on the test set, and weak evaluation. Each finding comes with a severity, the exact line, a plain-English explanation of the impact and a corrected code snippet. The tool also gives an overall quality score. Detection uses deterministic static analysis, and Gemini only explains and fixes what the rules found, so it can't invent bugs.

How we built it

We built MirageML in Python with Flask. The core is deterministic: Python's ast module parses the code, and a variable-flow tracker follows where each piece of data came from, so rules can tell whether a scaler or sampler touched data before the train/test split. Each rule flags a specific bug with the exact line and severity. The Gemini API then explains the impact and writes a fixed snippet, but only for bugs the rules already found, so it can't invent findings. If the API is unavailable, the app falls back to rule-based fix hints and still works. The frontend is plain HTML, CSS and JavaScript, and the app is deployed on Render. We built it during the hackathon using Google Antigravity as our AI coding assistant, and we tested the rules with pytest.

Challenges we ran into

Tracking data without a full dataflow engine. To know if a scaler touched data before the train/test split, we had to follow variables through assignments and tuple unpacking. A full analysis was out of scope for a hackathon, so we built a lightweight tracker that handles the common patterns and returns partial results instead of crashing on unusual code. Avoiding false positives. Correct pipelines, such as a scaler fit on the training set only, look similar to leaky ones. Our early rules flagged some of them, so we tuned the rules against clean examples until they stopped.

Accomplishments that we're proud of

Built and deployed a working tool during the hackathon. MirageML is live, and anyone can paste ML code and get a review in seconds. Deterministic detection. The same code always gives the same findings, so results are reproducible and the LLM can't hallucinate bugs. Measured results. On a small labelled benchmark we built ourselves, MirageML caught [X of Y] seeded bugs with [N] false positives on clean pipelines.

What we learned

Data leakage is subtle. It hides in code that runs perfectly, and even a small ordering mistake, like scaling before the split, can inflate results. Static analysis and LLMs work best together. Rules are reliable for detecting bugs, and the LLM is good at explaining them. Each does the job it's best at. Guardrails matter. Making Gemini explain only what the rules found removed hallucinated findings and made the tool trustworthy.

What's next for mirageML

More leakage patterns. Add checks for group leakage, duplicate rows across splits, and leakage through feature engineering and cross-validation. Full notebook support. Track variables across Jupyter cells, so bugs spread over multiple cells are caught. A bigger benchmark. Test on real public notebooks and report precision and recall on code we didn't write ourselves.

Built With

Share this project:

Updates

Submission history