ForYouPipeline

Zero humans in the loop: an LLM agent that runs its own recommender-systems research cycle — hypothesis, implementation, leakage review, measurement — and beat the official KuaiRand baseline.

Comment

Inspiration

Recommender-system research is a loop: form a hypothesis, implement it, measure it, decide what to try next. Most of that loop is judgement, not typing — and it's the part that doesn't scale. A researcher can only run one experiment at a time, and the expensive resource isn't compute, it's the attention needed to design the next experiment well.

We wanted to find out how much of that loop an LLM could actually own. Not "write me a model" — the whole cycle, including the parts that are easy to get subtly wrong: deciding what's worth testing, noticing when a result is noise rather than signal, and catching your own data leakage before it silently invalidates everything downstream.

KuaiRand-Pure is a good testbed for that because it punishes shortcuts. The metric is within-user ranking, so anything constant within a user contributes exactly zero. The label is long_view, which is derived from watch time — so half the columns in the log are post-interaction and using them as features is leakage that looks like a breakthrough. And the headroom is narrow: the official baseline already captures ~31% of the attainable range (random 0.4753 → baseline 0.5946 → oracle ceiling 0.8645), so real gains are small and easily confused with seed noise.

What it does

ForYouPipeline is an autonomous ML research agent. Given only the task definition and the starter kit, it:

  1. Reproduces the official FM baseline from scratch to confirm its pipeline is sound
  2. Inspects the data once, and carries those findings into every later decision
  3. Repeatedly proposes one atomic, falsifiable hypothesis, writes the full training script for it, reviews that code for label leakage, runs it, and diagnoses the result
  4. Stops when the challenge's own convergence rule says to — ε = 0.002 over N = 3 consecutive iterations

Result on the hidden test set, with zero human interventions during the run:

GAUC nDCG@5 primary
Official baseline 0.6610 0.5282 0.5946
ForYouPipeline 0.6643 0.5306 0.5975
Δ +0.0033 +0.0024 +0.0029

Reached in 8 iterations of a 50 cap, 1.26 hours wall-clock, 217,595 tokens, 0 GPU-hours.

The winning solution wasn't one idea — it was a chain the agent built across three accepted steps: a 5-seed ensemble averaged in within-user rank space rather than score space (because the metric only reads ordering), then Bayesian-smoothed per-video and per-author CTR features, then BPR pairwise loss replacing pointwise cross-entropy — aligning the training objective with the ranking metric actually being scored.

How we built it

Six LLM-driven roles, each doing one job, coordinated by a search loop:

  • Planner decides the next experiment — one hypothesis, with a stated mechanism and a falsification condition. It can search the literature mid-decision.
  • Coder writes the complete standalone training script.
  • Reviewer reads the diff against the parent solution for exactly one defect: label leakage. A flagged node still runs — a false positive shouldn't cost an experiment — but it can never become the submission.
  • Debugger attempts a fix when a node crashes.
  • EDA and Baseline run once each at the start.

Around them sit the parts that make the search honest rather than merely busy:

  • Deterministic diagnosis (no LLM) classifies every result as under_trained / overfit / noise / regression / improvement. This matters more than it sounds: a model that never fit its own training data is not evidence the idea was wrong, and conflating those two makes an agent abandon directions it never actually tested.
  • An unbiased-exposure gate scores every candidate against the random-exposure log as well as normal validation, and rejects changes that improve validation while collapsing on randomly exposed items — a proxy for overfitting to the production recommender's biases instead of to user preference.
  • Mechanism memory, within a run and pooled across every prior run, so the agent can see which families of idea have historically paid off and which are 0-for-15.
  • Seed confirmation — any promising node is re-run on additional seeds and averaged, so a lucky draw can't be mistaken for a gain.

Generated solutions have numpy, scipy, pandas, scikit-learn, LightGBM, XGBoost and PyTorch available; the winning one uses numpy only.

Challenges we ran into

Our leakage reviewer was blocking good work. In one run it flagged 8 of 17 nodes, seven of them false positives, including the run's best score. Three rounds of prompt-tuning failed — one made it worse, because naming an objection made it more salient. What finally worked was structural: scope the review to the diff against an already-cleared parent instead of re-reading the whole file every iteration. It had been re-litigating identical scaffolding dozens of times.

And it was right when we thought it was wrong. Later, testing whether it would wrongly block multi-task learning, we hit three "false positives" in a row — and every one turned out to be a genuine defect in our test fixtures (an undefined helper, an unverifiable row alignment, a method that didn't exist). We had nearly weakened the one gate standing between a leak and the submission, on the basis of our own broken tests. We removed the suppression rule we'd added.

That gate earned its place: one run produced a validation score of 0.8482 against an oracle ceiling of 0.8484 by using play_time_ms / duration_ms as a feature. long_view is derived from play_time_ms — the model was reading the answer. It was caught and blocked.

Our seed confirmation was doing nothing. Every "confirmed" score in one run came back bit-identical across three seeds. The generated solutions were hardcoding their ensemble seeds and ignoring --seed entirely — so 51% of that run's wall-clock recomputed a number it already had, and the averaging that was supposed to catch a lucky seed could not have detected one. The score wasn't wrong; our confidence in it was unearned.

Our own tooling steered the agent away from its best idea. We built a cross-run table of which mechanisms had historically paid off, sorted by median. Pairwise ranking losses have a poor median here — mostly losing attempts — so the family sat near the bottom, reading like a dead end. But the same row's best result was the highest score any run had ever produced. A search that keeps one checkpoint should chase the ceiling, not the average. We re-sorted it.

And our documentation foreclosed a whole solution class. Writing a strict leakage warning, we said post-interaction columns were "legitimate in exactly one direction." That accidentally banned using them as auxiliary training targets — which is not leakage at all. Multi-task learning went untried across five consecutive runs because of one phrase we wrote. (We reopened it; the agent tried it twice and it lost. Refuted, rather than never asked.)

Accomplishments that we're proud of

  • A genuinely autonomous run. Zero manual interventions from baseline reproduction to converged result.
  • Robustness we can point at. Two of eight nodes failed — one a code AttributeError, one where the LLM call itself failed before producing any code — and the run recovered from both automatically and kept improving. The iteration log shows the real diff for the first and correctly shows no diff for the second, because no code ever existed.
  • A verified result, not a lucky one. The winning solution was re-run from a clean checkout at three independent seed offsets (0.6042 / 0.6044 / 0.6043) and reproduces the submitted file byte-for-byte.
  • The agent found a compounding chain, not a single trick — ensembling in rank space, then smoothed CTR priors, then a metric-aligned loss, each building on the last.

What we learned

Measurement discipline is most of the work. Nearly every mistake we made was a measurement error, not a modelling error: a confirmation that confirmed nothing, a comparison against the wrong file, a ranking that optimised the wrong statistic. When gains are ~0.003 against a seed noise of 0.0008, the harness has to be more trustworthy than the model.

"We didn't measure its variance" is not the same as "the number is wrong." We conflated those and talked ourselves into a plateau narrative that was simply false — the best result had been real the whole time.

You cannot prompt an LLM out of a systematic misjudgement. Three attempts failed; a structural change to what the model was shown worked immediately.

Instructions foreclose more than they forbid. Two of our biggest search dead-ends traced back to sentences we wrote ourselves.

What's next for ForYouPipeline

  • The bonus benchmarks. KuaiRand-1K and 27K aren't attempted. We profiled why: the FM's per-minibatch update is O(vocabulary size), which is fine at Pure's ~40K embedding table but measured 313ms vs 16.5ms per step at 1K's ~2.9M-entry table. The fix is a sparse update touching only the rows a minibatch references — a contained change that would make 1K tractable and is a prerequisite for 27K.
  • User behaviour sequences. Completely untouched, and the most obvious remaining lever — every user has hundreds of prior interactions, and DIN/SIM-family interest modelling uses exactly that signal.
  • Deeper literature grounding. The agent searched papers and cited one that shaped a decision, but this is the thinnest part of its reasoning. Making retrieved work genuinely influence design — rather than corroborate a choice already made — is where we'd invest next.
Share this project:

Updates

Submission history