About the Project

Inspiration

Recommender engineers spend most of their time in one loop: read the data, engineer features, train, evaluate, repeat.

This challenge asks us to automate that loop — build an agent that iterates on its own and beats the official FM baseline on KuaiRand-Pure.

We started from ML-Master, an MCTS-driven agent that is state-of-the-art on MLE-Bench, and adapted it to recommender ranking.

What It Does

The agent runs the full loop autonomously.

The task is within-user ranking: for each user, rank only that user's already-logged impressions by the probability of long_view == 1.

The scored metrics are:

• Per-user GAUC (grouped AUC, weighted by each user's positive count)
• nDCG@5 (gain 2^rel - 1, position discount 1/log2(rank + 1))
• Primary = (GAUC + nDCG@5) / 2

Starting from a hand-written AutoInt/DIN reference, the agent self-discovered that the winning recipe is not a deep model but a LightGBM LambdaRank ranker + past-only historical aggregate features:

• Per-user exposure counts
• Per-video exposure counts
• Per-author exposure counts
• Per-tab exposure counts
• Long-view rates
• Click rates
• Time-of-day features
• Weekend features

Final Results

Metric	Validation	Test
Our Agent	0.6186	0.6120
Official FM Baseline	0.6016	0.5946
Absolute Delta	+0.0170	+0.0174

The final test delta is approximately 20× the baseline's 5-seed standard deviation of 0.0008.

The agent found and built this stack itself over 50 autonomous iterations.

How We Built It

Three layers on top of ML-Master:

1. Ported It to KuaiRand

• Wrote the date-based splits
• Implemented the GAUC/nDCG@5 evaluation
• Implemented the submission format
• Patched it to run on an Apple-Silicon Mac with the DeepSeek v4 API

2. Gave the Agent Hints, Not Answers

We provided:

• The organizer's headroom directions as a map
• Validated reference starting points (DIN, AutoInt)

Every improvement was left to the agent.

3. Validated Independently

Every hypothesis was scored on the held-out validation split, and the final submission was checked against the official submit.py harness.

What We Learned

1. The Ranking Loss Matters Most

With pointwise BCE, the agent got:

primary ≈ 0.478

which is near random.

Switching to a listwise ranking objective (LambdaRank) — which directly optimizes NDCG by weighting each pair's gradient by the NDCG change it would cause — jumped the same signal to:

primary = 0.6186

2. Feature Engineering Beat Model Depth Here

AutoInt (self-attention field interaction) and DIN (sequence attention) both plateaued below a gradient-boosted ranker fed historical statistics.

This matched the organizers' hint that:

"the bottleneck is not model capacity".

3. Negative Results Count

A learnable recency-decay was learned negative — the model preferred older history, because long_view reflects long-term taste, not recent spikes.

Challenges We Faced

1. Environment

ML-Master is Linux/CUDA-only.

Running on a fanless Mac M4 required:

• Patching CPU-affinity code
• Trimming 470 dependencies to ~15

2. Reasoning-Model LLM

DeepSeek v4's default "thinking" mode burned the whole token budget.

Disabling it cut per-call latency from approximately ~287 s to ~21 s.

3. Label Leakage

The agent learned to feed is_click/play_time_ratio as features and "scored" 0.84, past the 0.8645 oracle ceiling.

We had to forbid it at the source in the task description.

4. Autonomy vs. Results

Too much guidance kills the Autonomy score, while too little wastes the 50-iteration budget.

"Hints, not answers" was the balance.

Built With

Development Tools

• VS Code
• macOS Terminal
• Git

APIs Used

• DeepSeek v4 API (deepseek-v4-pro, via the OpenAI-compatible endpoint)

Libraries & Frameworks

• ML-Master (MCTS agent framework)
• LightGBM
• PyTorch
• NumPy
• pandas
• scikit-learn
• OmegaConf
• OpenAI SDK

Datasets & Assets

• KuaiRand-Pure — https://kuairand.com
• Official KuaiRand starter kit:
  • evaluate.py
  • data.py
  • submit.py

Built With

• deepseek
• lightgbm
• numpy
• pandas
• pytorch
• recommender
• scikit-learn