RecResearcher V3 — An Autonomous ML Research Agent for Recommender Systems

RecResearcher V3 is an autonomous research agent built for the KuaiRand-Pure recommendation benchmark in TikTok TechJam 2026. Instead of treating recommendation as a one-shot model-training task, we built a system that can inspect data, form bounded research hypotheses, run experiments, evaluate them with the official metrics, preserve the best checkpoints, and generate a final submission without using hidden test labels.

Inspiration

We were inspired by how recommendation research is usually iterative, evidence-driven, and expensive. A researcher does not simply choose a model once: they inspect data distributions, notice drift or cold-start patterns, test an idea, compare metrics, and revise the next experiment.

We wanted to turn that workflow into an agent. Our goal was not just to train a stronger recommender, but to build a reproducible research loop that can make and audit decisions with minimal manual intervention.

What it does

RecResearcher V3 works as a full research pipeline:

1. It loads the official train and validation splits and preserves the organizer-provided evaluator.
2. It performs leakage-safe data inspection, including label distribution drift, user-history coverage, feature availability, and post-exposure label checks.
3. It compiles bounded experiment recipes instead of allowing unrestricted code generation.
4. It trains and compares multiple candidate models using validation-only GAUC, nDCG@5, and their average, primary.
5. It saves experiment logs, checkpoints, recovery state, resource usage, and validation predictions.
6. It selects stable complementary models and creates an ensemble using validation data only.
7. It exports test predictions in the official CSV format without reading test labels.

The final agent supports CPU fallback and CUDA execution. We also included a ready-to-run Google Colab GPU notebook so that the full workflow can be reproduced on a Tesla T4.

How we built it

The project evolved from a lightweight CPU-based V2 into a GPU-enabled V3.

V3 keeps the official KuaiRand-Pure evaluation protocol unchanged and adds:

• CUDA-aware DeepFM and DCN training;
• DIN-style causal positive-history attention for user-interest modeling;
• multi-behavior auxiliary learning;
• deterministic, leakage-safe feature caching;
• bounded autonomous experiment recipes;
• auditable checkpoints and experiment artifacts;
• reproducible multi-seed DeepFM/DCN stability ensembling;
• label-free submission generation.

A key design decision was separating suggestion from execution. The agent can analyze evidence and propose research directions, but only whitelisted operations can become executable recipes. The official validation metrics remain the sole decision maker.

Results

On the public validation split, our verified V3 final ensemble achieved:

System	GAUC	nDCG@5	Primary
Official FM baseline	0.6674	0.5357	0.6016
V2 best ensemble	0.671813	0.538292	0.605052
V3 final ensemble	0.672020	0.538426	0.605223

V3 improves the validation primary metric by +0.003623 over the official FM baseline and by +0.000171 over V2.

These are public validation results only. We do not claim any hidden-test result. The generated submission contains 170,588 rows and passed the official submission alignment checker.

Challenges we faced

The hardest challenge was not simply making a model larger. The organizer’s baseline already captures much of the user-item interaction signal, while static feature expansion and larger FM embeddings bring limited gains.

We also had to address several practical issues:

• avoiding leakage from post-exposure signals such as click, like, comment, and play time;
• handling time-based distribution shift between train and validation;
• making user-history features causal;
• ensuring long Colab runs remain reproducible and recoverable;
• balancing autonomous exploration with controlled, auditable execution;
• ensuring the final inference path does not use hidden test labels.

GPU availability was another constraint. Our earlier versions focused on a CPU MVP first; V3 adds CUDA support and a Colab workflow so we can explore sequence models and multi-seed ensembles more efficiently.

What we learned

We learned that a useful ML research agent needs more than an LLM prompt and a training loop. It needs hard safety boundaries, a fixed evaluation protocol, reproducible artifacts, failure recovery, and a clear separation between hypothesis generation and metric-based decisions.

We also learned that small improvements matter in ranking tasks. A strong research process is not about claiming dramatic gains from one model; it is about producing trustworthy, repeatable improvements through disciplined experimentation.

What’s next

Our next steps are to run additional independent seeds, further investigate pairwise/listwise ranking objectives, expand sequence modeling, and explore richer multi-task architectures such as MMoE and ESMM. We will keep all future model selection validation-only and continue treating hidden-test evaluation as a final, one-time measurement.

Built With

• llm
• vscode