autorec lab

Inspiration

A new wave of "Autonomous ML Research Agents" (MLE-Bench, AIDE, AI-Scientist-v2) shows LLMs can now run a full ML iteration loop themselves — read the problem, engineer features, train, evaluate, reflect, repeat. That's exciting, but it raises a harder question than "can it write a training script": can you trust any number it reports? An agent that can silently leak test labels, call noise a "win," or quietly drift off its own best checkpoint isn't autonomous, it's just unaccountable. We wanted the boring, correct half of the system — the part nobody notices when it works — built as carefully as the creative half.

How we built it

One hard rule underneath everything: judgment and correctness never share code.

dataset profile + experiment memory + remaining budget
                         |
                         v
             structured LLM research proposal
                         |
                         v
 deterministic controller -> validate -> train -> predict
                                            |
                                            v
                              immutable official evaluator
                                            |
                                            v
                         SQLite + best checkpoint + reflection
                                            |
                                            +----> next proposal

The LLM proposes exactly one falsifiable hypothesis per iteration — it never touches the controller, the organizer's evaluator, credentials, or the dataset. A deterministic Python controller owns everything correctness-sensitive: sandboxed subprocesses, official metric execution, checkpoint promotion, and convergence. A second, deeper loop lets the LLM write actual model code (not just configs) inside a hash-guarded, path-restricted sandbox, gated by a compile/test/smoke pipeline before it's ever trained.

The model itself was built up one attributable change at a time, each one measured against the last:

Reproduce the official FM baseline
Switch to within-user BPR ranking loss (the single biggest win, since GAUC/nDCG@5 only ever care about ranking within a user, not raw scores)
Leakage-safe video metadata and field-aware interactions
A multi-task FM using click/like/profile-enter as auxiliary signal for the sparse long_view target
LambdaLoss, weighting training pairs by how much swapping them would move nDCG@5:

$$ \Delta_{ij} = \left| \text{nDCG@5}(\pi) - \text{nDCG@5}(\pi_{i \leftrightarrow j}) \right| $$

sampled only within one (user, date) group so it can't leak across days. A validation-selected rank ensemble of these checkpoints is the final result: primary metric 0.606015, +0.004546 over the reproduced baseline.

We layered two kinds of self-review on top late in the project. One teammate built an MLE-style diagnostics.json per attempt — outcome, promotion decision, artifact completeness, a recovery playbook keyed by failure type — so the next planning cycle gets an explicit review instead of a bare number. The other is a paired bootstrap over users:

$$ \hat{\Delta} = \text{primary}_A - \text{primary}_B, \qquad \text{CI}_{95\%} = \left[\hat{\Delta}^{\ast}_{2.5\%},\ \hat{\Delta}^{\ast}_{97.5\%}\right] $$

that turns "did this help?" into a real confidence interval instead of a flat epsilon check, run automatically after every experiment and, separately, across leave-one-feature-out model variants to decide which input columns are actually worth keeping. Both feed the same memory object the planner already reads before its next move.

What we learned

Getting the objective right beats almost everything else. Matching the loss function to how the metric actually works outscored every feature we added afterwards, combined.
A feature can lose on its own and still be worth keeping. The multi-task FM scored below baseline by itself (−0.0009) — it only helped once combined with LambdaLoss and blended into the final ensemble.
A flat threshold isn't statistics. Running the real bootstrap on our own "4 new features" experiment showed only 1 of the 4 had a confidence interval that excluded zero — the rest sat in noise about the size of the model's own seed-to-seed variance.

Challenges we faced

Proving "never sees the test set" structurally, not by promise. Our local research view materializes zero test rows, so there's nothing to leak even by accident, and every untrusted step is fingerprinted so nothing can quietly touch the controller, evaluator, or dataset.
Giving an LLM real code-writing power without real risk. The engineering loop lets the agent replace whole files, so every proposal is hash-guarded against stale edits, AST/secret-scanned, compiled, fully tested, and smoke-checked before it ever trains — and a worse candidate is kept only as an audit trail, never promoted.
Seed variance is a real adversary. More than once a "win" at one seed came back inconclusive on replication — the same size effect as an actual model change, which is a hard thing to trust from a single run.
Bootstrapping it correctly. Resampling rows instead of users would have violated the metric's own per-user structure and produced a confidence interval that lied.