KuaiLab: Autonomous RecSys R&D

Inspiration

Recommender-system research often becomes a maze of notebooks, scores, and experiments that are difficult to reproduce. Worse, subtle data leakage can make an improvement look real when the model has accidentally learned from the future.

We built KuaiLab to explore a different approach: an autonomous research agent that can generate hypotheses, run controlled experiments, reject unsafe ideas, preserve the evidence, and clearly explain why a model became—or did not become—the champion.

What it does

KuaiLab autonomously improves recommendation models through a structured research loop:

Inspect → Research → Hypothesize → Implement → Train → Evaluate → Reflect

The agent reviews previous experiments, proposes a falsifiable idea, implements it through trusted executors, evaluates it on fixed temporal splits, and records the outcome in a searchable candidate tree.

The dashboard provides:

A frozen champion with a traceable score and model lineage
Accepted, rejected, and failed experiment branches
Temporal validation and leakage checks
Reproducible model artifacts and starter downloads
Resource, runtime, and intervention tracking
An evidence-grounded AI chatbot for model explainability
Live ranking demonstrations using a compact deployment surrogate

KuaiLab does not promote a model simply because one metric improves. Candidates must pass the predefined evaluation gates without weakening the other official metrics.

How we built it

We used the KuaiRand-Pure recommendation benchmark with long_view as the prediction target.

Every experiment follows the same fixed temporal protocol. Historical features are constructed causally, validation histories are frozen using training data only, and future test-like labels remain sealed during model selection.

The research system combines:

An LLM researcher for hypothesis generation and reflection
Typed experiment configurations rather than unrestricted generated code
Trusted training executors for reproducible model families
Fixed GAUC and nDCG@5 evaluation
Experiment memory containing successes, failures, and exhausted ideas
Compute budgets and convergence-based stopping
Checksum-verified champion artifacts
A candidate tree that preserves model lineage
A server-side explainability layer grounded only in checked-in evidence

The current frozen static champion reaches a primary validation score of 0.612858, compared with the reproduced baseline of approximately 0.601469—an absolute improvement of 0.011389, or about 1.89%.

Challenges we ran into

The hardest challenge was preventing leakage. Recommendation features often depend on user history, video popularity, repeat exposure, and delayed behaviour. A seemingly harmless aggregation can expose the current row or future outcomes to the model.

We therefore separated causal training features from frozen validation histories and avoided using the hidden test-like period during feature selection.

Another challenge was keeping model identity clear. We had several legitimate scores from different protocols: the frozen static champion, a historical autonomous FM run, an online research teacher, rejected candidates, and a lightweight live surrogate. These numbers were not directly comparable, so we built an explicit model registry and evidence trail.

We also learned that promising research ideas must sometimes be rejected. One ranking candidate improved against its matched baseline on both screening and confirmation splits, but became harmful when integrated with the stronger champion. KuaiLab retained the champion and stored the failed experiment as useful research memory.

Accomplishments that we're proud of

We are proud that KuaiLab demonstrates scientific restraint, not just optimization.

The system:

Produced a measurable improvement over the reproduced baseline
Preserved an untouched future evaluation boundary
Recorded complete accepted and rejected experiment lineages
Automatically rejected a candidate that failed the champion-integration gate
Recovered from invalid experiments without losing the campaign
Created a reproducible, downloadable champion starter package
Made every headline score traceable to a checked-in artifact
Added an AI assistant that explains results while clearly stating evidence limitations

Most importantly, KuaiLab can say, “This idea did not generalize, so we did not deploy it.”

What we learned

Historical user behaviour contains most of the useful recommendation signal, but how it is represented matters more than simply adding more history.

User-by-context interactions—especially user-by-tab behaviour—were consistently valuable. Duration preference and auxiliary behaviours also contributed useful information.

Naive short recency windows, such as fixed one-, three-, and seven-day aggregates, did not improve generalization. More features were not automatically better.

We also learned that model explainability must distinguish between different levels of evidence. Experiment results can explain why a model was selected, but they are not automatically faithful explanations for an individual prediction. KuaiLab exposes candidate-level contribution terms only where the deployed surrogate supports them and refuses to invent unavailable attributions.

What's next for KuaiLab: Autonomous RecSys R&D

Next, we plan to:

Complete the final sealed benchmark evaluation after freezing the submission
Expand strictly causal session, fatigue, preference-lift, and last-N features
Add stronger sequence and graph-based recommendation executors
Improve automatic user and video segmentation
Support faithful per-prediction attribution for full champion checkpoints
Generalize the autonomous research loop to additional recommendation datasets
Add experiment comparison, collaboration, and remote execution
Turn the candidate tree into a reusable knowledge base across research campaigns

Our long-term vision is a recommendation research partner that does more than train models: it designs careful experiments, protects evaluation integrity, learns from failure, and makes every decision understandable.