TacoRank: an autonomous research agent for recommender systems

Inspiration

Anyone who has tuned a recommender knows the loop: think of an idea, write a patch, wait for training, stare at a number, repeat. It eats days, and most of it is mechanical. We wondered how much of that loop an LLM could actually run on its own. Not just suggesting ideas, but planning experiments, writing the code, training in a sandbox, judging its own results honestly, and knowing when to stop. TacoRank is our attempt at that: a self-driving research lab for the KuaiRand-Pure ranking benchmark. You start it in the evening and audit it in the morning.

What it does

Every iteration passes through five specialists.

A planner LLM (DeepSeek) reads a label-free profile of the data, a portfolio of "method cards" (short literature-grounded technique briefs), and the run's own past evidence. It writes one falsifiable hypothesis with an explicit falsification condition.
A coding agent (Trae) turns the hypothesis into a real code diff on an isolated commit.
The candidate trains inside a network-free Docker container with frozen CPU and memory quotas. Scoring uses the organizers' own evaluate.py, byte for byte untouched.
A trust layer decides accept or prune. Candidates climb a fidelity ladder (quick smoke test, cheap proxy, then full two-seed confirmation) and must beat their parent beyond a noise band we calibrated to the official baseline's own seed variance.
Everything lands in an append-only, hash-chained ledger (events.jsonl): the hypothesis, the diff, the metrics, every failure and every recovery decision. The whole run is replayable and tamper-evident.

Once the rules are frozen, no human touches the loop. We also built a Next.js dashboard that renders the ledger live, so you can literally watch the agent think.

How it addresses the problem statement

The challenge asks for an autonomous system that improves a recommender on KuaiRand-Pure and can prove its process: iterations, convergence, resource cost, auditable logs. Our overnight demo run delivers on each point.

The official FM baseline scores 0.60147 (GAUC 0.66713, nDCG@5 0.53580). Our submission is the run's validation-best model, exp_021, entirely produced and logged by the agent: 0.60385 primary (GAUC 0.67037, nDCG@5 0.53733), a +0.00238 delta over the baseline, with the CSV validated by the official submit.py --check. In offline analysis after the run, a per-user z-score ensemble of run-produced members reaches about 0.6047 on validation; we kept it out of the submission to keep the submitted artifact purely agent-made, but it supports our ceiling measurement below.

On convergence, the official rule ($\varepsilon = 0.002$, $N = 3$) is satisfied from iteration 4 onward, and anyone can recompute that from events.jsonl. The run finally stopped with no_legal_proposal, meaning the planner's deduplicated frontier was empty. That is actually a stricter stopping condition than the plateau rule.

The number we are proudest of is a rejection count: 19 of 21 explored directions were killed at the cheap proxy gate. We also measured the compliant ceiling ourselves in a manual lab study (about 0.6045 under strict train-split-only rules). The agent got there on its own, then established statistically that nothing more was left.

Total cost: 24 planner iterations, about 3 hours of laptop wall-clock, 25.1M LLM tokens, zero GPU-hours, zero manual interventions. All of it metered in the ledger.

Challenges we ran into

Compliance turned out harder than accuracy. Our strongest early recipes used rolling user-history features and scored 0.61+, but they implicitly touched score-period rows, which is leakage. We threw them away, rebuilt everything train-split-only, and enforced the rule in the planner prompt, the method cards, and static code gates. Deleting our best number was the hardest decision of the week.

Agents fail constantly, and the run must survive anyway. We hit coding-agent step limits, container OOM kills, Docker daemon flakes, and training jobs quietly running on one thread. Each failure class got a typed recovery policy: classify it, retry with the real diagnostic attached, tombstone it if it keeps failing. The overnight run recovered from every failure without waking anyone up.

We also learned the hard way about duplicated constants. Numeric policy bounds lived in several validation schemas plus the state machine, and five separate copies of one constant each crashed a live run before we hunted them all down.

And early versions of the trust gate accepted noise. We fixed that by calibrating the accept band to the baseline's five-seed standard deviation and requiring seed-confirmed improvements against the parent, not the baseline. That discipline is why the accepted chain ascends cleanly.

What we learned

The autonomy is not in the model, it is in the harness. The planner's ideas were only as good as the evidence loop around them: honest gates, machine-readable tombstones for dead ends, append-only provenance. And the most valuable thing the agent did was not the score gain. It was knowing when to stop, which is the genuinely expensive part of real research.

Built with

Development tools: VS Code, Claude Code, Git and GitHub, Docker Desktop on Windows, PowerShell, pytest for the harness test suite.

APIs: DeepSeek chat API (the planner LLM), Trae agent (autonomous coding), OpenAlex API (literature retrieval for method-card citations).

Libraries and frameworks: Python 3.12, LightGBM (LambdaRank and rankers), CatBoost (evaluated, then retired on evidence), NumPy, pandas, scikit-learn, Pydantic (event and schema validation), Next.js with React and Tailwind (the live run dashboard).

Datasets and assets: KuaiRand-Pure with the official competition split and evaluator; the organizers' starter kit (evaluate.py, submit.py) used byte-untouched; our own offline compliant-ceiling study in research/offline_ceiling_study/ as supporting evidence.