RecLab

RecLab — an autonomous ML-research agent that earned every basis point Track 2 · Autonomous ML Research Agent for Recommender Systems · KuaiRand-Pure

FINAL RESULT Validation primary 0.6056 (GAUC 0.6731, nDCG@5 0.5381) vs the official FM baseline 0.6016 (0.6674 / 0.5357) — a gain of +0.0040, discovered fully autonomously in 6 iterations, 34 minutes, 319K tokens, 0 GPU-hours and 0 manual interventions, on a laptop, for about $2.55 of total API spend.

WHAT IT IS

Track 2 asks for an agent that runs the MLE loop on KuaiRand-Pure by itself: reproduce the official FM baseline, then propose → code → train → evaluate → reflect until the convergence rule (epsilon = 0.002, N = 3) fires, and hand in the validation-best checkpoint's hidden-test submission with per-iteration logs. RecLab does exactly that, unattended — and adds the part the brief only implies: it cannot cheat.

The organiser's harness is SHA-256-locked and re-verified around every iteration
The LLM may write exactly one file: workspace/pipeline.py
Every candidate pipeline runs in a sandboxed subprocess (import blocker, I/O fence, timeout, RSS cap), passes a static AST leakage scan, and then a shuffle-invariance test: predictions must be bit-identical when the target's feedback columns are randomly permuted — only then does the organiser's own evaluate.py score it
12 out of 12 red-team attacks blocked: feedback columns as features, target rows leaked into rate tables, labels read from disk via a hidden read_csv, monkeypatched evaluate, NaN / misaligned / shuffled scores, writes into the harness, exec-smuggling

Six LLM roles drive the loop: Analyst (deterministic EDA and error buckets), Proposer (hypothesis, expected gain, confidence, backlog), Skeptic, Auditor (rejects only for four concrete leak categories — and knows that context-column session features are NOT leakage), Engineer (writes the full file; RecLab computes the diff), Reflector (keep/revert plus run memory), plus a Scribe for the report. A stopping-rule-aware scheduler front-loads high-variance exploration at 0 strikes and forces safe exploit moves at 2 strikes.

Robustness is a first-class feature: tracebacks go back to the Engineer with error history and targeted hints for a dozen measured failure classes; a repair that reproduces the same error signature escalates to a full rewrite from the last working file; a method scoring far below its stated expectation triggers a diagnosis pass fed with feature importances and the cited recipe's exact values; no-op iterations are caught by prediction fingerprint; rate limits wait or stop the run resumably instead of costing a strike.

THE WINNING RUN (pure_openai_40), ITERATION BY ITERATION

Iter 0 — faithful FM baseline port through the contract — 0.6010 Iter 1 — LightGBM on prefixed side-table joins — 0.6016 Iter 2 — plus history-only temporal target encodings — 0.6048 Iter 3 — LambdaRank exploration (kept as evidence, reverted) — 0.6026 Iter 4 — plus session / list-position context features — 0.6056 ← validation-best Iter 5 — 3-seed ensemble (below best, reverted) — 0.6052

Converged by the epsilon 0.002 / N 3 rule. Hidden-test submission: 170,588 rows, history = train + validation, model not refit on validation labels, verified with the kit's checker.

THE 45-RUN STORY

We ran the agent 45 times end-to-end while hardening it. Every failure was diagnosed from the run's own logs, fixed with a commit, and verified by the next run. Humans only ever fixed the agent — never the model's work.

Phase 1 — Teething (runs 1–8). First converged run: +0.0010. Then one classic agent failure per run: an over-strict Auditor rejecting sound plans and burning strikes (fix: blocking-issues-only rejection); fall-back gates silently collapsing blends into the previous model (fix: no-op detection by prediction fingerprint); the Engineer citing a recipe it had never been shown (fix: knowledge notes injected verbatim); HTTP 429s killing runs (fix: a rate-limit-aware client with per-role model pools and resumable stops); reasoning models returning empty JSON because thinking ate the token budget (fix: budgets that grow themselves).

Phase 2 — The local-model gauntlet (runs 9–24, $0, a 32B local coder via Ollama). A local model makes recurring mistakes, and each became a permanent guard or hint: pandas-3 string-dtype fills, read-only views, hallucinated columns, f-string quote garbling. The most instructive error: "Sum of query counts (1,100,240) differs from #data (96,148,510)" — 96,148,510 is the sum of squared group sizes, because transform('size') builds a per-row group vector. One model ping-ponged between two errors for five straight repairs, so the repair loop now escalates to a full rewrite whenever an error signature repeats. These sixteen runs converged at baseline; their value was making the agent unbreakable.

Phase 3 — The free-tier climb (runs 25–41, $0). With stronger pool models on free-tier daily buckets the bests ratcheted: 0.6050 → 0.60512 → 0.60513 → 0.60537 → 0.60553 → 0.60560 — each step a real, logged discovery (9-seed ensembles, temporal target encodings, session features). Production incidents root-caused live along the way: an Ollama restart racing a zombie process still holding its port; an auditor wrongly branding label-free context features as leakage (fixed with an explicit "what is not leakage" section — the very next run it asked for order-robustness instead of rejecting); and a hard deadlock diagnosed by sampling the stuck process's stack — torch and LightGBM each bundle their own OpenMP runtime, and importing both parks the child at 0% CPU forever in __kmp_join_barrier. Fixed in the executor environment plus an engineer rule.

Phase 4 — The $2.55 experiment (runs 42–45). Under a hard $5 cap on a $2.55 balance, Claude Sonnet 5 took the core roles: the first attempt of its first iteration scored 0.6050 — a level free-tier runs needed 2–4 iterations to reach. Both runs were stopped by a per-tick cost meter, token-priced from the run log, exactly at the budget line.

What the plateau means: ten independent converged runs landed between 0.6011 and 0.6056; our own hand-tuned offline probes of the same method family measured about 0.6066–0.6074. The agent autonomously reached within ~0.002 of an expert — and every basis point above baseline is verified leak-free by construction.

HOW WE BUILT IT

Python 3.12. One package (reclab/): loop state machine, sandboxed executor, static and dynamic guards, scheduler, seven role prompts, one LLM client with three back-ends (Anthropic SDK, OpenAI Responses API, local Ollama), finalize/report/dashboard, a 12-attack adversary suite, 66 pytest tests. The organiser's starter kit is used unmodified as the locked harness. A mock-LLM mode replays a scripted trajectory so the whole loop runs offline in CI.

CHALLENGES

The honest list is the four phases above. The meta-challenge: doing this with almost no money. Free-tier daily buckets (50 requests per day per model, 10K tokens per minute) forced the client to become genuinely good at rate limits — per-role pools, rotation on 429, hopeless-request detection, resumable stops — and forced a local-model mode that hardened the repair loop far beyond what strong models alone would have required.

ACCOMPLISHMENTS

+0.0040 validation over the official baseline, all of it audit-clean: AST scan, bit-identical shuffle-invariance, 12/12 adversary attacks blocked
Zero manual interventions in the submitted run — and in any run's iterations; humans only fixed the agent, never the model
A repair loop that measurably learns: rewrite-on-repeat escalation, normalized error signatures, per-error hints
The whole project ran on a laptop CPU for about $2.55

WHAT WE LEARNED

Small deltas are hard and mostly engineering: the gap between "the plan is right" and "the code is leak-free, deterministic, and implements the recipe's exact hyper-parameters" is where autonomous agents live or die. Guards must be prescriptive (say how to fix). Auditors must know what is NOT a violation — over-rejection silently costs more score than leakage ever would. Repair loops need an escalation ladder — a model minimal-diffing the same broken file five times is a failure of the system, not the model.

WHAT'S NEXT

Per-user z-score blending of the GBDT with the FM (our best measured recipe, 0.6074 validation — the agent proposed it once the planners knew it existed; a deadlock stole its first attempt and the budget cap its second), KuaiRand-1K generalisation, and a self-updating knowledge base where each run's reflections become the next run's cited notes.

TOOLS, APIS, DATA

Development: Claude Code (the agent framework was itself developed and supervised in production by an AI assistant), VS Code, Python 3.12, pytest, git (80+ run-diagnosed commits). APIs: OpenAI Responses API (per-role pools across gpt-5.6-sol, gpt-5.5, gpt-5.4, gpt-5, gpt-4.1, o4-mini and minis under free-tier limits), Anthropic Messages API (Claude Sonnet 5 + Haiku 4.5), local Ollama (qwen2.5-coder 32B), arXiv API. Libraries: numpy, pandas 3, pyarrow, LightGBM 4.7, scikit-learn, PyTorch (CPU), anthropic SDK, requests, PyYAML, rich, pytest. Data: KuaiRand-Pure only (official archive). No external data, no pretrained weights.

Submission artifacts in the repo under runs/pure_openai_40/: submission_test.csv (170,588 rows, passes the kit checker), iterations.jsonl with per-iteration diffs and events, results.md, report.md, dashboard.html, resources.json, the harness SHA-256 manifest, the adversary report, and the baseline reproduction (0.6010, within 0.0006 of official).