Autonomous Recommender Researcher

• 

Inspiration

Machine-learning engineers repeatedly perform the same research loop: inspect the current result, form a hypothesis, change the pipeline, run an experiment, and decide what to try next. Track 2 asked whether an agent could own that loop for a real recommendation benchmark while requiring minimal human direction.

We were interested in more than producing a higher score. An autonomous researcher is useful only if its result is trustworthy, reproducible, and easy to audit. That led us to separate the creative researcher from a deterministic harness that controls data access, evaluation, budgets, provenance, and final submission.

We tested this through a completed KuaiRand-Pure submission and then a redesigned KuaiRand-1K campaign that applied everything we learned from the first run.

What it does

Autonomous Recommender Researcher lets a coding agent conduct an iterative ML campaign. For each iteration, the researcher proposes a hypothesis, writes a complete candidate pipeline, receives controller-computed validation metrics, records what it learned, and chooses the next experiment.

The official KuaiRand-Pure campaign completed 15 scored iterations. It reproduced the organizer's Factorization Machine baseline; tested pairwise and listwise losses, behavioral crosses, and auxiliary feedback; recovered from a NumPy state bug; introduced causal attention over user engagement history; checked seed robustness; diagnosed negative interaction between useful components; and finished with a score-level rank ensemble.

The submitted Pure model blends:

• a five-seed dual-gate DIN-lite/FM ensemble;
• a five-seed pointwise FM ensemble; and
• a watch-ratio auxiliary FM.

Every component trained only on the official 8–21 April 2022 training window. The validation-best checkpoint was selected before its final CSV was evaluated once on the hidden test.

After completing Pure, we built a much better harness and launched a separate KuaiRand-1K campaign. Its scientific ML loop was fully autonomous: GPT-5.6 Sol performed the EDA and research, wrote every candidate, interpreted results, and chose each next experiment with feedback from a Claude Opus 5 critic. The participant did not steer the ML work; Codex only supervised process health and trusted state. The campaign was externally aborted at the submission deadline after 13 attempts / 11 scored iterations, before its frozen minimum of 15. It never reached terminal state or opened the hidden test, so the 1K work is validation-only rather than a completed submission.

How we built it

The Pure system used a hosted Claude coding-agent session as the researcher, a Python harness as the controller, and OpenAI Codex as an independent read-only reviewer. Candidate models called no external API.

The redesigned 1K harness kept authority narrow without removing the tools needed for serious ML work:

• the complete stopping policy was frozen before launch, with no researcher override;
• controlled EDA, source-grounded research, and independent criticism were required before experimentation;
• candidates ran in a network-isolated Bubblewrap process while protected labels, evaluation, and stopping authority remained controller-only;
• the sandbox retained LightGBM, Polars, SciPy, scikit-learn, multithreading, caches, checkpoints, and bounded error diagnostics; and
• every attempt counted, entered the atomic event history, and had to pass frozen-checkpoint replay before its score was accepted.

The controller recorded immutable candidate snapshots and atomic events for every attempt. All 11 scored 1K attempts passed checkpoint replay.

The Pure run used 15 of 50 iterations, 3,781.3 seconds of agent wall-clock, 2,278.2 seconds of candidate execution, an estimated 466,000 LLM tokens, and zero GPU-hours. The 1K campaign ran for approximately 3.5 hours before the external deadline abort.

Challenges we ran into

The most important Pure challenge was governance. With the default epsilon = 0.002, N = 3 cumulative criterion, convergence became true after iteration 4 because the first hypothesis family had plateaued. The original harness nevertheless exposed --continue-past-convergence, which the researcher used while untested families remained. An independent reviewer correctly flagged this as a control violation. Logging an override made it visible, but did not give the researcher authority to change its own stopping instruction.

We disclosed the issue rather than retroactively calling it a custom policy. After reviewing the logs, the organizers instructed us to submit the iteration-15 validation-best result. The 1K redesign removes the override and freezes the full stopping policy before launch.

The agents also had to navigate genuine technical failures. In Pure, iteration 2's BPR fine-tune corrupted its intended fallback because NumPy arrays aliased the saved champion state. The researcher diagnosed the contradiction, fixed the state handling, and exactly restored the iteration-1 fallback in iteration 3.

In 1K, the researcher recovered from an unavailable inference feature and LightGBM's ranking-query limit, rejected harmful user weighting, removed an overfit feature bundle, developed causal session features, and found strong target-free user-context personalization. It also retained the incumbent when an unsupported added model received zero ensemble weight rather than inventing an improvement.

A temporary critic outage and an unrelated worktree interruption were recovered without changing any candidate, score, prompt, or scientific decision. The run ultimately stopped only because submission time ran out.

Accomplishments that we're proud of

The completed Pure submission improved both validation and the one-time hidden test over the published baseline:

Result	GAUC	nDCG@5	Primary
Published validation baseline	0.667400	0.535700	0.601600
Validation-best iteration 15	0.669094	0.536891	0.602992
Validation delta	+0.001694	+0.001191	+0.001392
Published hidden-test baseline	0.661000	0.528200	0.594600
Submitted hidden-test result	0.662562	0.529195	0.595879
Hidden-test delta	+0.001562	+0.000995	+0.001279

The final CSV contains all 170,588 test predictions and passed the organizer's header, row-count, alignment, and finite-score checker.

The incomplete 1K campaign also produced a strong validation trajectory:

KuaiRand-1K checkpoint	GAUC	nDCG@5	Primary
Attempt 3 — first scored candidate	0.439146	0.464725	0.451936
Attempt 11 — user-context personalization	0.685232	0.661519	0.673376
Attempt 13 — ordered causal history	0.690992	0.656925	0.673958

These are controller-measured validation metrics only. No official 1K baseline was supplied and the hidden test was never accessed, so we claim neither a baseline improvement nor hidden-test generalization.

We are especially proud that the 1K agent demonstrated real autonomous problem navigation rather than simply running a fixed tuning script. Its proposals, events, EDA, research, critic findings, policy, best candidate, and checkpoint evidence are all preserved in the repository.

What we learned

The main Pure modeling lesson was that new information mattered more than a new loss on the old representation. Pairwise BPR, listwise training, and explicit user-author affinity did not improve the baseline. Causal target-aware attention over engaged user history produced the largest gain.

The watch-ratio objective helped as a shared-representation regularizer, but its own head was not a better scorer. Jointly training useful mechanisms caused negative interference, while blending separately trained models at score level worked.

The main systems lesson was that auditability and authority are different. Logs can reveal an unsafe action, but the controller must prevent it. We also learned that security should restrict authority without removing the compute, libraries, diagnostics, and recovery paths needed for effective ML research.

The 1K campaign also showed that autonomy is demonstrated by navigation. The researcher recovered from failures, rejected harmful branches, retained its incumbent when an idea added no value, and continued without participant scientific steering.

What's next for Autonomous Recommender Researcher

The major governance improvements proposed after Pure are already implemented in vNext: immutable stopping rules, isolated execution, mandatory research, independent criticism, controller-only scoring, counted failures, automatic recovery, immutable snapshots, and exact checkpoint replay.

The next step would be to run a fresh 1K campaign through its normal terminal state and perform its hidden evaluation exactly once. We would also strengthen critic failover, isolate orchestration from shared-worktree changes, and explore deeper sequential and GPU-supported model families.

The repository contains the completed Pure submission, detailed report, convergence disclosure, full Pure journal, every Pure candidate source, final CSV, the deadline-aborted 1K evidence package, and the complete vNext harness.

Built With

• git
• github
• python