Autonomous ML Research Agent for Recommender Systems

Autonomous AI research agent that discovers, tests, and optimises recommendation models and features on real-world user behaviour data

Comment

How our solution addresses the problem statement

The problem statement asks the agent to do three things on its own.

Reproduce the official baseline. Every run opens by executing the pipeline as it stands and measuring its score in this environment, across five seeds, before proposing anything. We deliberately do not trust the published number as our reference: the published figure was measured on someone else's machine, and every accept-or-reject decision in the run is made against the value we measured. A run whose baseline verification fails aborts rather than reporting improvements over a reference that was never established.

Iterate autonomously across the full stack. The agent may rewrite two files

  • one owning features, encoding and sample construction, the other owning architecture, loss and training strategy - and it chooses which to target each turn, or both when a change needs them coupled. Across the reported run it proposed, among others: time-decayed user–author affinity features; duration-quantile survival metrics such as bounce risk and mid-watch survival; sequential retention stages with stage-to-stage transition probabilities; an intra-user pairwise logistic (BPR) loss over positive/negative impression pairs; Personalized Differentiable AUC Optimization with maximum violation; item–item and author transition-graph affinity mined from training sessions; greedy re-ranking to break up slates sharing an over-exposed author; and multi-task auxiliary heads on click, like and log watch-time.

That spans features, loss function, architecture, training strategy and a re-ranking stage. It is not architecture tuning, which is what the problem statement explicitly asks us to go beyond.

Improve over the baseline. +0.0044 on validation primary, computed by the agent and recorded in its own run manifest. The trajectory is not monotonic and our log shows that plainly: of all measured candidates, one was kept. Two scored below random guessing and were reverted automatically without human help.

What we think is interesting about it

The agent's code runs in a subprocess, never in-process. The code being executed was written by a language model seconds earlier. Run in-process, a segfault or an unbounded loop ends the entire search. Out-of-process, a crash becomes an exit code, a hang becomes a timeout with the process group killed, and leaked memory is reclaimed by the operating system. This is what let a five-and-a-half-hour run survive 31 pieces of generated code.

Guardrails run before the code does. A static check on the syntax tree rejects a candidate that does not expose the required entry point, or that writes its own training loop instead of delegating to the shared harness — a candidate with a private training loop escapes the shared early-stopping and learning-rate selection, so its score is no longer comparable to anything else we have measured. Rejecting it costs one prompt; discovering the same problem after training costs a full iteration.

A second guard catches the failure that would actually disqualify us. Any score above the oracle ceiling — what you get by ranking with the true labels — is not a breakthrough; it is a label that has leaked into the features. The run is rejected and the error names that cause, because on this task the difference between a real result and a disqualifying one is a single mis-indexed column.

Documentation is a byproduct of running, not a chore afterwards. One run is one directory, opened before the first iteration and finalised on every exit path including the crash path. Each iteration appends one line recording its hypothesis, the code diff, the resulting metrics, wall time, tokens, any error events and how they were handled, and an explicit statement of whether a human intervened. The schema refuses an incomplete line at write time rather than at judging time. Beside each line sits the evidence — the prompt as sent, the raw reply, stdout, stderr, the diff — so "the agent was told X" is checkable rather than asserted. Nothing is reconstructed from memory afterwards, and a value we could not measure is named as unmeasured and left null rather than defaulted to zero.

Development tools used

  • VS Code — primary editor
  • Claude Code — agentic coding assistant, used by us to build the harness; it is not part of the delivered agent and takes no part in a run
  • Git and GitHub — version control, one branch per experiment lineage
  • GNU Make — entry points (make agent, make agent-dry, make test)
  • pytest — 75 tests
  • Windows 11, CPU only — no GPU was used at any point

APIs used

  • OpenRouter — LLM gateway. The reported runs use google/gemini-3.7-flash; earlier runs used gemini-3.6-flash directly.
  • LiteLLM — provider-agnostic client, so switching model or provider is a config change rather than a code change.
  • arXiv API and OpenAlex API — literature retrieval, letting the agent look up published work while it is forming a hypothesis. Both are keyless.

Every LLM call in the codebase passes through a single wrapper, and a unit test greps the source tree to prove nothing else imports a provider SDK. Token accounting is part of how this challenge is scored, and tokens spent outside that wrapper are tokens nobody counted.

Libraries and frameworks used

  • PyTorch — the agent's model surface; every architecture it proposed is PyTorch
  • NumPy — the reference Factorization Machine baseline is pure NumPy, as is all encoding and bucketing
  • pandas and PyArrow — feature loading and the dataset catalog shown to the agent
  • Pydantic (with pydantic-settings) — schema validation on the agent's proposals, and typed configuration
  • LiteLLM — provider-agnostic LLM calls
  • json-repair — recovering malformed JSON from model replies
  • scikit-learn, SciPy, LightGBM — available to the agent as a non-PyTorch route via the tabular harness
  • PyYAML — configuration
  • Rich — terminal output during a run
  • pytest — testing

Python 3.11+, with exact versions pinned in a lockfile.

Datasets and assets used

  • KuaiRand-Pure (Kuaishou) — the required benchmark: 1.4M interactions, 27K users × 7.6K items from a short-video feed. We use the two standard interaction logs plus the basic and statistical video feature tables and the user feature table.
  • The organizer-provided Starter Kit — the official evaluation script (vendored verbatim and never modified), the split definitions, the submission schema, and the published baseline scores.

No external training data, no pretrained weights, and no access to the hidden test set at any point. The relevance label, the task form, the temporal split and the metrics are all the organizers' definitions, and scoring runs through their evaluation script — we did not reimplement it.

Built With

Share this project:

Updates