RecAgent: Autonomous ML Research for KuaiRand

Inspiration

Track 2 challenged us to build something structurally different from a typical ML project: an agent that does the research itself, proposing hypotheses, writing the code, running the experiment, and deciding what to try next, rather than a human iterating with an ML toolkit. We were drawn to the honesty built into the task's own scoring: a negative or thin delta isn't a failure, it's just a lower score on one slice. That framing let us build something that reports what actually happened rather than what we wished had happened.

What it does

An autonomous ML research agent that runs a full Planner → Coder → Executor → Debugger loop against the KuaiRand-Pure recommender benchmark, with bonus attempts on KuaiRand-1K and a scale-transfer measurement on KuaiRand-27K. Each iteration, the Planner proposes a modeling hypothesis grounded in a curated knowledge base of prior findings, the Coder implements it as a code diff against the current best model, the Executor deterministically trains and validates the result and accepts or rejects it, and the Debugger repairs malformed diffs so a bad patch never crashes the run. The loop runs unattended until it converges (validation score plateaus within ε = 0.002 over 3 iterations) or hits a hard cap, then scores once, on test data the agent never accessed during iterative development.

Results

KuaiRand-Pure (required benchmark, official submission)

Metric	Baseline	Our result	Delta
Test primary	0.5946	0.5968	+0.0022 (≈2.76× baseline std)
Validation primary	0.6016	0.6041	+0.0025
Manual interventions	—	0	—
Iterations to converge	—	5	—
Wall-clock	—	22.75 min	—
LLM tokens	—	109,787	—

KuaiRand-1K (bonus benchmark)

Metric	Baseline	Our result	Delta
Validation primary (3-seed mean)	0.6439	0.6463	+0.0024 (3-seed replicated)
Test primary	0.6405	0.6399	pending verification*
Iterations	—	8 scored	—

*A validated, 3-seed-replicated improvement on validation. The test-side figure needs one more confirmation pass before we'd cite it as a win — see Challenges below.

How we built it

Python, numpy-only, matching the starter kit: no torch, pandas, or sklearn needed, and the FM baseline trains in about 40 seconds on a single CPU core, so compute was never the bottleneck. LLM backend: Sakana AI's Fugu model for the agent's Planner, Coder, and Debugger roles, with a parallel experiment using Llama-3.1-8B-Instruct. The knowledge base (knowledge_base.yaml) enforces a replication discipline: confirmed wins, controls, and known non-transfers are surfaced to the Planner first, trimmed to the decision-relevant subset to keep context focused. Beyond the required Pure benchmark, we built a KuaiRand-1K bonus adapter and a purpose-built paired-seed statistical stopping policy to guard against noise-driven false positives at that scale, and separately confirmed the full pipeline runs correctly at KuaiRand-27K's full scale (322M rows) via a direct scale-transfer measurement.

Challenges we ran into

The most persistent one wasn't a bug, it was a pattern. Across three independent runs, on different branches, by different team members, the Coder repeatedly implemented the same prevalence-weighted binary cross-entropy loss regardless of what the Planner's hypothesis actually asked for: a DIN sequence model, an SSM, a field-aware FM interaction structure. The scores were real and correctly computed, but the reasoning trail attached to them wasn't what shipped. Rather than editing the run logs to look cleaner, we documented this openly and treated it as a genuine finding about the current Coder's capability boundary. We also hit a real harness gap: a 1K-scoped run once drifted into rewriting our code for the 27K benchmark mid-run and crashed, since we had no guard against an agent proposing changes outside its assigned scope.

Our first KuaiRand-1K attempt came back a clean null result: no accepted improvement across 8 iterations, 5 of them literal no-op diffs. We responded by building a rigorous paired-seed statistical stopping policy, characterizing the 1K baseline's variance over 5 seeds (higher than we'd assumed) and requiring a 2σ margin over 3 matched seeds before accepting any candidate. A later, independent run under that same discipline found a validation-side improvement (+0.0024, 3-seed replicated); we're holding the test-side claim until the numbers reconcile cleanly, in keeping with the same honesty standard we held the Pure result to.

Accomplishments that we're proud of

A fully autonomous, zero-manual-intervention run that beats the Pure baseline by a reproducible margin roughly 2.76 times the baseline's own test-seed standard deviation, with an honest paper trail showing exactly how it got there, including where the agent's stated reasoning and its actual code diverged. We also confirmed the pipeline scales cleanly to KuaiRand-27K's full 322-million-row dataset without modification. We'd rather submit a true account of a real system's real limitations than a run log edited to look better than what happened.

What we learned

Published calibration work (MLE-Bench, MLRC-Bench) suggests a positive, reproducible delta with zero intervention is closer to the exception than the norm for this class of agent: one benchmark found the best agent/framework combination reached only bronze-medal performance on 16.9% of tasks, and it's common for agents to score worse than baseline entirely. That context reframed how we read our own results. The interesting story isn't just the delta, it's the systemic Coder limitation we found and could name precisely, three times, across independent runs.

What's next

A scope guard preventing an agent from editing outside its assigned benchmark, stronger triviality detection that inspects the effective computation graph rather than just the text diff, confirming the 1K test-side result now that a validated validation-side improvement exists, extending the paired-seed statistical stopping policy from 1K to Pure, and turning the 27K scale-transfer measurement into a full agent-driven run now that we've confirmed the infrastructure handles it.