MARS-Rec

Visual flow of MARS-rec pipeline
MARS-rec primary score in comparison

Inspiration

The track asks for an autonomous ML research agent, and the obvious version of that is easy: wrap an LLM in a loop, let it edit a model file, print the scores. What we wanted to know was harder. Can an LLM actually do research, form a hypothesis, test it, be wrong, and use being wrong? Or does it just generate plausible-looking code until something sticks?

That question only has an answer if you can tell the two apart. So we built the measurement first: an evidence ledger, a hidden-test firewall, machine-verified intervention counting, and a set of predictions sealed in git before the scored run, recording what we expected the agent to find. If it matched our expectations we would know. If it didn't, we would know that too.

It didn't. That turned out to be the most interesting result we got.

What it does

MARS-Rec runs the full ML engineering loop on KuaiRand-Pure within-user ranking, unattended. The task is to rank each user's own candidate videos by the probability of a long view, scored by

$$ \text{primary} \;=\; \tfrac{1}{2}\bigl(\mathrm{GAUC} + \mathrm{nDCG@5}\bigr). $$

The loop:

Survey. The agent writes its own search queries from its current dead end and retrieves papers from arXiv and public solutions from GitHub.
Propose. It reads its evidence ledger, the metric's internals, the data distributions and its predecessors' training curves, then emits a complete rewrite of model.py and/or data.py with a falsifiable hypothesis.
Validate. The proposal is token-parsed and rejected before it runs if it touches the test split, references a test-window date, writes outside the sandbox, or repeats an experiment already in the ledger.
Train and score. Training happens in a killed-on-timeout subprocess, and scoring is done only by the organizers' own evaluate.py against the validation split.
Decide. Every experiment is trained on three seeds, two of them in parallel, and the mean of the three decides. Writing the three scores as $s^{(0)}, s^{(1)}, s^{(2)}$ and their mean as $\bar{s} = \tfrac{1}{3}\bigl(s^{(0)} + s^{(1)} + s^{(2)}\bigr)$, the change is committed when $\bar{s} > \bar{s}^{\,\text{parent}}$ and reverted with git checkout when it is not, surviving only as evidence.

Everything is logged: hypothesis, rationale, cited sources, diff, per-epoch curve, cost, and decision rationale. The run that produced our result took 20 iterations in one afternoon and stopped itself. A person touched it once, to restart the loop after the machine lost its network connection.

How we built it

The specification came out of a loop of its own, before the agent existed. NotebookLM drafted the PRD, ChatGPT reviewed the draft, and every piece of that critique was read and judged by one of us before it went back to NotebookLM for the next revision. Our role there was referee rather than author: deciding which objections were real, which were taste, and what got cut.

What survived was a spec for five components and nothing else (baseline runner, proposer, executor, evaluator, logger) with git as the entire state machine. Rollback is git checkout, lineage is the commit graph, and the required per-iteration diff is git diff. That decision removed a research memory DAG, two state machines and an AST dedup layer from the design before a line was written.

The harness was then implemented with an AI pair-programmer working from that PRD, in Python and NumPy with no ML framework, so the whole system runs on CPU, 0 GPU-hours. The agent's proposer calls Claude; the model it writes is plain NumPy.

Challenges we ran into

The agent was blind, and we blamed its intelligence. Runs A, B and C each produced three experiments and zero improvements, nine straight failures. We kept rewriting the prompt. Then we looked at the training curves and found every single failed experiment peaked in the first three or four epochs while the training loss kept falling: one mechanism, overfitting, nine times. The agent could not see it, because it received one scalar per experiment and never its own loss curves. Giving it the numbers, and only the numbers, never our interpretation, was worth more than every prompt rewrite combined.

A rule that could not be satisfied. The convergence rule stops a run when the best score has not improved by more than $\varepsilon = 0.002$ over $N = 3$ iterations. But across twenty experiments the running best moved by a total of $+0.00137$, and the largest single step ever measured was $+0.0012$. Both are smaller than $\varepsilon$, so no single iteration can clear it, and the rule fires after three experiments and never releases. We kept $\varepsilon$ and $N$ and let an accepted result reset the window, disclosed it in the README, printed it at every run start, and left --strict-convergence in place to reproduce the literal reading.

A one-hour hang against a thirty-minute timeout. Capturing subprocess output creates OS pipes and reader threads. The timeout kills the child, then the join waits for EOF, which never arrives while a grandchild holds the pipe open. The fix was not a bigger timeout. It was no pipes at all: output goes to a file, and the deadline is enforced by killing the whole process tree.

Knowing when the constraint was ours. We had capped training at $180$ seconds per run. Under that cap, $1$ of $17$ experiments was positive. Above it, $7$ of $11$. The cap was our own invention, sized by arithmetic that never survived contact with the convergence rule. Raising it to $600$ seconds is what made two of the six winning findings possible at all, and we say plainly in the README that the evidence is confounded, because expensive models are also more ambitious ones.

Staying honest while losing. After three barren runs it is tempting to loosen whatever is binding. We wrote down each deviation before running, recorded it on every ledger record, and left the strict setting reachable by a flag. When our budget accounting turned out to be stricter than the rules required, we did not relax it, because we discovered that after it started to hurt, the worst possible moment to reinterpret a rule.

Accomplishments that we're proud of

It beat the baseline, on the hidden test, on one shot.

Hidden test GAUC: $0.666422$ against a baseline of $0.6610$, a gain of $+0.005422$
Hidden test nDCG@5: $0.531392$ against $0.5282$, a gain of $+0.003192$
Hidden test primary: $0.598907$ against $0.5946$, a gain of $\mathbf{+0.004307}$

The baseline's standard deviation over five seeds is $\sigma = 0.0008$, so the gain is

$$ z \;=\; \frac{0.598907 - 0.5946}{0.0008} \;\approx\; 5.4 . $$

The test split was evaluated exactly once, on the checkpoint chosen by validation alone.

Six findings that compounded rather than competed: history pooling, field-weighted interactions (FwFM, arXiv 1902.09757), epoch-snapshot ensembling under within-user rank normalisation, DIN-style candidate attention, exponential recency weighting of training rows toward the last training day ($w \propto e^{-\lambda \Delta t}$, with $\Delta t$ the number of days back), and smoothed rate-bucket features. Every experiment was replicated on three seeds and judged on the mean, and git rolled back the thirteen that did not improve it.

One intervention, and the harness caught it rather than us reporting it. It hashes its own source and the config, watches the workspace for uncommitted edits, and counts session restarts against the declared run, so an intervention is detected rather than self-reported. The scored run records one event at level 1 of the ladder: at 13:34 the loop was restarted after four proposal calls failed on connection errors and it stopped itself with an environment fault. No code changed, and the file hashes prove it. Levels 2 and 3 are zero, so no hypothesis, parameter or acceptance decision came from a person.

0 GPU-hours. Everything, including 48 training runs, 20 iterations and the winning model, is NumPy on CPU.

We pre-registered, and then published the failure. The predictions were sealed in git before the run and the post-run comparison is in the repo unedited, including the three findings we never anticipated.

What we learned

Read the scorer, not the description of the scorer. We spent a day reasoning about a "pointwise loss vs user-averaged metric" mismatch. Then we opened evaluate.py and found gnum += npos * auc(...). GAUC weights each user by that user's positive count rather than uniformly. Writing $p$ for a user's positive count and summing over users:

$$ \mathrm{GAUC} \;=\; \frac{\sum p \cdot \mathrm{AUC}}{\sum p} \qquad \text{and not} \qquad \frac{1}{|U|} \sum \mathrm{AUC} . $$

Our central hypothesis was half wrong, and the experiment that assumed it had already lost $0.0018$. The same read showed that the oracle ceiling is pure arithmetic rather than a modelling frontier: $30.3\%$ of validation users have no positive label and score $0$ on nDCG@5 whatever the model does, so a perfect ranker reaches only $\tfrac{1}{2}(1 + 0.6968) = 0.8484$.

Silent failures cost more than loud ones. Our arXiv search returned zero results for a whole run and reported no error, because we sent a six-word phrase as an exact-phrase query, which matches nothing and is answered with a valid empty feed. The one-shot test evaluator would have printed a plausible wrong number, because it recomputed scores with plain factorisation-machine algebra, a bias plus per-feature weights plus dot products between latent vectors,

$$ \hat{y} \;=\; b + \sum w \cdot x + \sum \langle v, v' \rangle \, x \, x' , $$

while the winning model had learned a separate weight for every field pair and an attention mechanism that the plain form cannot express. Both were found by checking, not by anything failing.

Instrumentation beats persuasion. Every real jump in the agent's performance came from giving it something it could not previously see: its own loss curves, the scorer's source, the data's distributions, its predecessors' measured failures. None came from telling it to try harder.

The pre-registration failed, and that's the result. We predicted an objective-function-driven run: listwise loss first, features second, architecture barely featuring. The agent went architecture-first, with history pooling, then field-weighted interactions, then attention, while the entire objective family was proposed twice and rejected twice. Three of its six findings appear nowhere in our predictions, including the two largest. Because the predictions were sealed and specific, "we did not see that coming" is a measured result instead of a story told afterwards.

What's next for MARS-Rec

The self-ablation: the same harness with a random proposer and with a greedy hill-climber, on one chart. It is the only evidence that separates an agent from a loop, and its absence is the biggest hole in this submission. After that, cross-model replication, the same harness driven by Qwen or DeepSeek, to test whether these findings are properties of the task or of one model family. Longer term, the interesting direction is not a better score but a better ledger: an agent whose evidence store is good enough that a second agent can pick up where it stopped.