Autonomous ML Research Agent — KuaiRand-Pure

Inspiration

The KuaiRand-Pure starter kit ships something unusual: a section titled "where to start" in which the benchmark authors list the things they already tried and measured as failures (more features, more embedding capacity — both flat), and then a ranked list of directions they had not explored. Ranking losses. User behaviour sequences. Multi-task supervision. Censored watch-time regression.

That reads like an invitation. It is, essentially, a research agenda with the dead ends already marked — the situation a human researcher faces at the start of a project, compressed into two pages.

Which raised the question we actually wanted to answer: the ML iteration loop is bounded by human attention, not by compute. A model here trains in 13 seconds. What is slow is a person reading results, forming a hypothesis, writing the next script, and doing it again. If an LLM can hold that loop on its own, the bottleneck moves.

So we did not set out to build a better recommender. We set out to build something that could do the research, and then find out honestly whether it worked.

What it does

An LLM-driven agent runs the complete ML research loop with no human inside it:

1. Proposes a hypothesis — one change, and why it should work
2. Writes a complete PyTorch training script, not a config or a diff
3. Executes it in a sandboxed subprocess with a timeout
4. Scores it on validation through the official evaluation code
5. Confirms anything that looks like an improvement across three random seeds
6. Reflects — the result, or the traceback, becomes the next prompt's context
7. Stops when the organizers' convergence rule says the run is done

Every iteration is logged with its hypothesis, code diff, metrics, and any error and recovery. The agent never sees the hidden test set — it cannot, by construction.

The division of labour is the design decision that matters. The agent owns the four things you could write a paper about: features, model architecture, loss function, hyperparameters. The harness owns everything that carries no research content: cached data, batching, the epoch loop, early stopping, scoring, sandboxing. Giving a researcher PyTorch is not doing their research; giving them the trained model would be.

How we built it

The harness/agent split. agent/loop.py imports exactly one harness module and reaches the benchmark through an injected scoring callable — enforced by an AST check in the test suite, not by a comment in a docstring. The loop knows nothing about KuaiRand. Retargeting it to another problem means writing a new adapter and system prompt against the same loop.

Leakage prevention is structural, not honour-system. The starter kit's loader hands back test rows with their labels attached, so every guarantee about "not peeking" would ordinarily rest on discipline. In our harness, candidate code that asks for test labels — or any of the eleven post-impression feedback columns — gets an exception, not data. Candidates never compute a metric at all: they write out prediction arrays, and the harness scores them. A static screen rejects any generated script referencing the raw CSVs or video_features_statistic_pure.csv, whose per-video counts span the test window and would be label leakage. One guarded entry point reads the test set exactly once, requires an explicit --confirm-final, refuses to run inside a candidate subprocess, and writes an indelible audit record to the run log.

A significance gate, because the noise floor is brutal. Seed-to-seed variation on this benchmark is about ±0.0005 while the convergence threshold is 0.002 — roughly 4σ. A single-seed "improvement" of 0.001 is a coin flip. So any candidate that beats the incumbent is automatically re-run on three seeds and accepted only on the mean.

Verification before the agent was allowed to run. The harness had to reproduce the published FM baseline (0.6016 validation) from an independently rebuilt data, encoding, training and scoring pipeline before a single LLM call was made. It lands at 0.6018. The full suite is 120 tests.

Challenges we ran into

We were working from a second-hand summary of the rules. Our planning document — written before we had the actual problem statement — asserted that teams could declare their own convergence parameters. The real brief contains no such clause; it fixes ε = 0.002, N = 3. We discovered this partway through, after two runs had used a modified rule. We reverted to the official rule, re-ran from scratch, and archived rather than deleted the non-compliant runs. Cheap lesson, pointed one: verify constraints against the primary source, not a summary.

Our own baseline lied to us on a lucky seed. It scored 0.6025 on seed 0 and 0.6018 across three. Had we selected checkpoints by best seed rather than mean, we would have manufactured roughly 0.0007 of improvement out of nothing — about half the size of our actual result. This is exactly the failure the significance gate exists to prevent, and it caught it on the first real use.

Infrastructure failure masqueraded as agent failure. Three iterations died in 2.3 seconds each with exit code 1 and completely empty stderr. Our loop's three-strike guard — designed to stop an agent looping on a bad idea — fired and killed the run. The cause was an API rate limit, and the CLI writes its failure reason to stdout, which we were not capturing. We added a distinct BackendUnavailable class, captured both streams, and gave outages their own stop reason with retry-and-backoff, so infrastructure can never again look like the agent being stuck.

Resuming an interrupted run nearly let us cheat by accident. When a spend limit killed the submitted run mid-flight, the obvious fix was to restart it with the best checkpoint seeded. But that hands the run a fresh three-iteration convergence window — letting it legitimately continue past the point the official rule would have stopped it. We implemented resumption that restores the prior scored trajectory into the window instead. The run then converged one iteration later.

An analysis we nearly published was wrong. Our first attempt at measuring whether the model's advantage decays over time sliced validation by date. But GAUC and nDCG rank within a user across the whole evaluation set — slicing by date changes the ranking task itself, so per-day deltas do not decompose into the overall delta. The flawed version showed decay that was not there, from single-seed comparisons that were mostly noise. Stratifying by user instead is valid, and gave a real answer.

Smaller ones: Windows caps a command line at ~32k characters, smaller than our prompt (it goes over stdin now); the claude CLI is a .cmd shim needing cmd /c; and the official submit.py prints a ✓ that crashes on Windows' default console encoding after successfully validating.

Accomplishments that we're proud of

Zero manual interventions inside any run. Every proposal, script, execution, score and log entry happened unattended. The five interventions we did make were all between runs, all harness or compliance fixes, and all documented with their reasoning.

The agent found the research agenda on its own. It worked down the authors' unexplored list — ranking losses, then user-history modelling, then multi-task supervision — and never once proposed either of the two dead ends they had already measured and published.

It wrote its own leakage guarantee, correctly. The submitted model needs strictly causal history, and the agent handled that itself. In the candidate's own words: "for a train row, the history is strictly causal — only positives with time_ms earlier than that row. The row's own label can never enter its own history." We verified it by hand. It is correct.

One iteration produced a genuine diagnosis rather than a tweak. After several iterations plateaued, the agent stopped tuning and reasoned about why:

the pure-ID FM is saturated as a function class, not that the signal is exhausted: an FM can only express additive first-order weights plus symmetric dot-product crosses over one-hot IDs, so it cannot represent a threshold on continuous duration. long_view is definitionally a play-time-vs-duration threshold, and the encoder's 10 quantile duration buckets smear that boundary; a tree splits it directly.

It then switched function class entirely. That is the difference between search and research.

We measured our own result honestly enough to undermine it. Repeating the run under identical rules gave +0.0002 and +0.0014 — a spread comparable to the improvement itself. We report the distribution rather than the better run alone.

What we learned

Run-to-run variance of an autonomous agent can exceed the improvement it finds. This is the finding we would most want other people to take away. Two runs of the same agent, same configuration, same rules, produced deltas of +0.0002 and +0.0014. Anyone reporting a single autonomous-agent run — ours included — is likely reporting one draw from a distribution they have not characterised. We would want ten runs; we managed two before the budget ran out.

Sequence models help exactly where their signal is fresh, and hurt where it is absent. Stratifying validation users by how stale their training history is:

user group	delta over baseline
no train history (1,219 users)	−0.0050
0 days stale	+0.0015
1–2 days stale	+0.0018
3–4 days stale	+0.0019
5–12 days stale	+0.0002

The history branch helps when history exists and is recent, does almost nothing once it is more than four days old, and actively harms the users who have none. Since every hidden-test row sits 8–17 days further from the training window than validation does, our validation gain is an upper bound on the test result, not a prediction of it.

A benchmark's noise floor should be a first-class component, not a footnote. Building the three-seed gate before the agent's first real run was the single highest-leverage decision we made, and we nearly skipped it as premature.

Guardrails are worth more than good intentions. Every constraint we enforced structurally — unreachable test labels, candidates that cannot score, a static screen on generated code — held. The one thing we left to discipline (checking the rules against a summary rather than the source) is the one thing that went wrong.

What's next for Autonomous ML Learning Agent

Fix the gap our own analysis found. Users with no usable history lose 0.0050 under the submitted model. Falling back to the plain FM for them is an obvious win. We deliberately did not hand-write it — that would be us doing the research the agent is supposed to be doing — so the right next step is to feed the analysis back into the agent's context and let it propose the fix itself.

Characterise the variance properly. Ten runs, not two, and report the distribution as the headline number rather than a single converged result.

Test whether the agent needs the hints. Our system prompt contains the authors' published list of unexplored directions. It is the same problem statement every entrant received, but we never measured whether the agent finds those directions without it. Removing the list and re-running would settle how much of the insight is genuinely the agent's — and it is the experiment we most want to run.

Let a run reach the 50-iteration cap. Every run so far ended on the convergence rule or an outage. Long-horizon behaviour — whether the agent keeps generating genuinely new ideas or collapses into micro-variations — is undemonstrated, and we saw hints of both.

Reach the censored watch-time direction. The label is literally a watch-time threshold, which makes one-sided censored regression the most on-target unexplored idea on the authors' list. No run got there.

---

Results

Submitted model: a DIN-style user-behaviour-sequence branch on the factorization machine — target attention over each user's last 20 long-viewed videos, strictly causal, built from the training split only.

	GAUC	nDCG@5	primary
official FM baseline (validation)	0.6674	0.5357	0.6016
our agent (validation, 3-seed mean)	0.6691	0.5368	0.6030 ±0.0001
absolute delta	+0.0017	+0.0011	+0.0014
oracle ceiling	1.0000	0.6968	0.8484

Found at iteration 4; the run then stopped on the organizers' convergence rule (ε = 0.002, N = 3) at iteration 8, using 7 of its 50 permitted iterations.

How big is that, honestly? Statistically the gain clears seed noise (t ≈ 4.0 against our own reproduction of the baseline). Practically it is small: it captures 0.57% of the headroom between the baseline and the oracle ceiling — for scale, the baseline's own advance over trivial item-popularity was 15× larger. And across runs rather than seeds it is not significant (+0.0002 and +0.0014, t ≈ 1.9, n = 2).

We think the methodological findings above are the more valuable contribution than the score.

Autonomy

Manual interventions during any run	0
Interventions between runs	5, all documented in interventions.md

Resource usage

	submitted run	all runs
iterations	7 of the 50 cap	19 scored + 3 failed
agent wall-clock	33 min	71 min
output tokens	~57k	223,034
total tokens (in + out + cache)	—	556,962
GPU-hours	<0.1	<0.5 (RTX 3060)

Runs used a Claude Code CLI subscription, so no API bill was incurred. Estimated equivalent API cost at Claude Opus 5 list pricing, derived from the measured token counts above: $0.47 per iteration (median; range $0.27–0.83), ~$3 for a converged run, ~$24 at the 50-iteration cap. Output tokens dominate — they are thinking-heavy.

---

Built with

Development tools: Claude Code (CLI) · Windows 11 · Python 3.11.4 · pytest · NVIDIA RTX 3060 Laptop (the benchmark does not require a GPU)

APIs: Claude Opus 5, driving every one of the agent's proposals, accessed through the claude CLI (Claude Code subscription) — invoked as a subprocess with the prompt on stdin and --output-format json parsed for the proposal and token usage.

No metered Anthropic API calls were made. The codebase also contains an APIBackend that speaks to the Messages API via the official anthropic Python SDK (prompt caching on the byte-stable system prompt, adaptive thinking, streaming), behind the same interface as the CLI backend. It is unit-tested but was never exercised against the live API — every run in this submission used the CLI path, and the API cost above is an estimate, not a bill.

Libraries and frameworks: PyTorch 2.3 (all candidate models) · NumPy 1.26 · pandas 2.1 (data cache and feature encoding) · pytest 9.1 (120 tests) · anthropic 0.91 (installed for the alternative backend; unused in the runs) · Python standard library — difflib for run-log diffs, subprocess for the sandbox, json for append-only logging.

The starter kit's evaluate.py, data.py, baseline.py, submit.py and ablation_features.py were used unmodified — evaluate.py is the sole authority on scoring.

Datasets: KuaiRand-Pure (Kuaishou), from https://kuairand.com — 1.14M training interactions, 27K users, 7.6K items. Fixed date-based splits per the starter kit: train 20220408–0421, validation 20220422–0428, hidden test 20220429–0508. No external training data, no pretrained weights, and the randomized-exposure log was never used for training.

Contributions: Solo project — all design, implementation, analysis and write-up by the repository owner. The candidate scripts under candidates/ and runs/*/candidates/ were generated by Claude Opus 5 during autonomous runs and are unedited.

Built With

• claude
• numpy
• pytest
• python
• pytorch