LeakLens

What it does

LeakLens reviews machine-learning pull requests for the bugs that ordinary linters and type checkers never catch: data leakage, non-reproducible training, and broken evaluation. It reads a diff, runs a deterministic engine of eight rules over only the added lines, and flags problems like fitting a scaler before the train/test split, splitting with no random seed, reporting accuracy on the training set, or tuning a decision threshold on the test set. It returns a findings table, a 0 to 100 reproducibility risk score, and a merge verdict. GPT-5.6 then turns those structured findings into a written review: it confirms or dismisses each candidate, explains the concrete risk, and proposes a fix.

Inspiration

Code style and types have solid automated guardrails. The correctness of an ML experiment does not. A scaler fit before the split, a missing seed, an accuracy number read off the training set: each one quietly inflates results or makes them impossible to reproduce, and each one passes code review because nothing is looking for it. LeakLens puts that class of bug on the same footing as a failing lint check, caught on the pull request before it merges.

How I built it

I built the entire tool with Codex running gpt-5.6-terra. Codex scaffolded the unified-diff parser, the eight-rule engine in checks.py, the Markdown reporter and risk score, the MCP server, the CLI, and the test suite. The architecture is split on purpose: a fast deterministic layer finds candidates with exact line numbers and never invents a bug, and a reasoning layer where GPT-5.6 confirms the findings, weighs severity, and writes the fixes. LeakLens runs two ways: as an MCP server, so inside Codex GPT-5.6 calls the review_ml_diff tool directly, and as a standalone CLI with an optional --explain mode for use outside Codex.

What I learned

A deterministic scanner and an LLM cover each other's weak spots. Regex and AST checks are precise and never hallucinate, but they are blunt about severity and cannot explain themselves; GPT-5.6 is the reverse. Handing the model structured, pre-verified candidates instead of a raw diff makes its review noticeably more grounded, and the MCP tool interface turned out to be the clean seam between the two halves.

Challenges I ran into

The demo fixtures carry comments that name each planted bug, so the scanner could have "passed" by reading the comments rather than the code. Stripping comment lines and matching only real code on added diff lines was the difference between a review that looks right and one that is right. The other balancing act was keeping the rules high-signal: broad patterns catch more but raise false positives, so the deterministic rules stay narrow and leave the judgment calls to GPT-5.6.

What's next

More rules (temporal splits for time series, PyTorch determinism flags, dependency pinning) and a GitHub Action so the review runs on every pull request automatically.

Built With

• codex
• gpt-5.6
• mcp
• openai
• pytest
• python
• scikit-learn