Project Story

Inspiration

Machine learning engineering is a loop: read the problem, look at the data, build features, train and tune, evaluate, then decide what to change and go round again. Two of those stages are pure code, which is exactly why systems like AIDE and MLE-bench have made the loop a target for automation.

What drew me in wasn't getting an LLM to write model code — that part is solved. It was the harder question underneath: how do you make a search loop you can leave alone for an hour and still believe the number it hands you? An agent that computes its own score can hallucinate an improvement, and a tree search will happily climb a fabricated gradient. On this benchmark the run-to-run noise is $\sigma \approx 0.0008$ while the threshold for a "real" improvement is $\varepsilon = 0.002$ — the same order of magnitude. An agent that accepts single-seed results will spend its entire budget climbing randomness and produce a confident, meaningless answer.

So I built the harness first and let the agent do the modelling.

What it does

RankLoop runs all five stages of the loop autonomously on KuaiRand-Pure, a short-video recommendation dataset. The task is within-user ranking over logged impressions, scored as

$$\text{primary} = \tfrac{1}{2}\left(\text{GAUC} + \text{nDCG@5}\right)$$

The agent maintains a small tree of candidate solutions. Each node is a complete runnable Python file; each edge is one hypothesis-driven change. It can draft a new direction, improve the incumbent, or debug a candidate from its own traceback. It writes every line of model code itself.

Result: test primary 0.5976 against the official FM baseline's 0.5946 — Δ +0.0030. Because 27.1% of test users have no positive interactions at all, nDCG@5 caps at 0.729 and the oracle ceiling is 0.8645, so that gain is 1.1% of the achievable range rather than 0.7% of a notional 1.0.

Across five runs it evaluated 20 candidate solutions and, with a hyperparameter sweep and ensemble search, 36 measured experiments — with zero human interventions during any run and zero unrecovered errors.

How I built it

Four properties make unattended search reliable, and each is an architectural guarantee rather than a convention:

Independent evaluation. Candidates write prediction arrays to disk. The harness scores them by calling the organizers' untouched evaluate.py. Nothing self-reported ever enters the search.

Noise-aware acceptance. A promising candidate is re-run across three random seeds and judged on the mean, so a single lucky seed cannot be promoted.

Enforced train/test separation. The data API strips every outcome column from the test split and raises if a candidate asks for test labels. Test is scored exactly once, on the final submission.

Failures are data. Crashes and timeouts are captured in a subprocess, classified, and returned to the model as bounded repair prompts — three attempts, then the branch is abandoned. Everything lands in a JSONL run log.

One check underpins all of it. The pipeline asserts that its cached data reproduces the organizers' own data.load() row for row across all three splits, because submission row_id indexes that exact ordering. Building that cache also cut per-iteration cost from ~90s to ~20s by removing a repeated CSV parse — over a 50-iteration budget, the difference between comfortable and fighting the wall-clock ceiling.

The search is driven by Claude Opus 5 through the Claude Code CLI in headless mode, so a run consumes a subscription rather than pay-as-you-go API credits. Models are numpy and PyTorch on CPU; no GPU was used at any point.