CookedRecommender

An autonomous ML research agent for KuaiRand ranking

Problem statement

The objective is to design and implement an Autonomous ML Research Agent that:

reproduces a fixed organizer baseline,
iterates on the full recommendation pipeline using only the training split and public validation feedback,
and improves validation performance until it exceeds the official reference baseline before a final submission is selected for the hidden test set.

This project implements that idea as a closed research loop: a Python pipeline executes the real experiment, metrics are measured with the official evaluator, and a Gemini model acts as the text-only ML engineer that proposes hypotheses, suggests code changes, diagnoses errors, and decides whether to continue or stop.

Inspiration

The TechJam challenge isn't "train one model" — it's to design and build an autonomous ML research agent that:

Reproduces a fixed organizer baseline.
Iterates on the full recommendation pipeline using only the training split and public validation feedback.
Improves validation performance until it exceeds the official reference baseline, then selects a final submission for the hidden test set.

That framing is what CookedRecommender implements: a closed research loop where Python executes the real experiment, the official evaluator scores it, and a Gemini model acts as a text-only ML engineer — proposing hypotheses, suggesting changes, diagnosing errors, and deciding whether to continue or stop.

What it does

This is a research loop, not a static training script. It:

Records historical states and metrics
Uses an LLM (Gemini) to propose the next experiment
Executes those experiments in the real codebase
Evaluates with the official recommendation metric
Decides whether to continue or stop

The pattern:

The model and data pipeline produce measurable outcomes.
The language model interprets the outcomes and writes the next hypothesis.
Python executes the next experiment.
Results feed back into the next prompt.

Gemini never executes code — it only provides ideas and text recommendations. Python is the sole execution layer.

What judges get:

Official KuaiRand-Pure splits, evaluated on long_view within-user ranking (GAUC, nDCG@5, primary = average of both)
A heterogeneous neural trainer: Compact SENet DeepFM + Low-Rank DCNv2 DeepFM, multi-seed, then power-rank blended on validation
An agent loop that persists history and writes a contest submission.csv
evaluate.py is never altered

How we built it

The stack has three layers.

1. Data & evaluation

kuairand-starter-kit/data.py loads KuaiRand-Pure, applies the official train/validation/test split, and encodes categorical fields (user_id, video_id, author_id, tab, dur_bucket) to integer IDs, each with its own vocabulary plus an "unknown" slot. kuairand-starter-kit/evaluate.py is the organizer protocol — GAUC, nDCG@5, primary = mean of the two — treated as ground truth and never modified.

Official splits: train runs 2022-04-08 to 2022-04-21, validation runs 2022-04-22 to 2022-04-28, and test (hidden) runs 2022-04-29 to 2022-05-08. The agent stays on the public split at all times — it never uses the hidden test set to decide what to try next.

2. Training & modeling

train_heterogeneous_blend.py trains a heterogeneous ensemble of MultiTaskResidualSENetDeepFM and MultiTaskLowRankDCNv2DeepFM. Both share multi-task outputs (long-view logit, click logit, watch-ratio prediction), shared embeddings and feature-interaction modules, SENet-style channel attention for feature reweighting, and low-rank DCNv2 for structured feature crosses.

Each architecture trains over multiple seeds; the best validation checkpoint per seed is kept. Seed predictions are aggregated, then power-rank normalization and blend weights are optimized on validation — the final submission is chosen by validation score, never test leakage. SENet contributes compact deep interactions and feature recalibration; DCNv2 contributes explicit, low-rank feature crosses; the blend combines both inductive biases. Supporting training utilities live in autotrainer.py (SENet / low-rank DCN modules, official-metric early stopping, seed ensembles).

3. Research-agent layer

kuairand-starter-kit/kuairand_agent.py is the autonomous loop (Gemini client + runner in one module). It loads the current pipeline and data, builds context (metrics, baseline gap, hypothesis history), asks Gemini for the next config/hypothesis, runs the real trainer, parses validation GAUC / nDCG@5 / primary, stops when validation primary is stagnant for 3 trials (ε = 0.002), and ensembles the top configs into a final submission.csv plus a JSON history. Gemini is the "ML engineer in the loop" — ideas, code/config suggestions, failure-mode diagnosis, continue-vs-stop calls — while training and evaluation always stay in Python.

Tech stack: Python 3.9+, PyTorch, NumPy, Pandas, SciPy, python-dotenv, and the Google GenAI SDK for Gemini, built on the official KuaiRand-Pure starter kit (data.py, evaluate.py, submit.py, baseline.py). Data comes from KuaiRand-Pure logs and video metadata (Zenodo / kuairand.com), with is_click and watch time powering the multi-task heads. Secrets live in .env, never in source.

Challenges we ran into

Keeping Gemini as advisor and Python as executor, so the loop stays reproducible and safe
Matching the training objective to within-user ranking (GAUC / nDCG@5) rather than global classification
Combining two architectures without leaking test labels — blend weights and power exponents are fit on validation only
Recovering gracefully when an API call or a trial fails, so the loop still writes a submission

Accomplishments that we're proud of

A closed research loop that matches the problem statement: baseline protocol frozen, LLM proposes, Python executes, official metrics decide
A heterogeneous multi-task ranker (SENet DeepFM + low-rank DCNv2) with validation-driven power-rank blending
Full persistence of hypotheses, metrics, and errors — the "agent" is inspectable, not a black box
Strict split hygiene: test data is used for the final CSV only

What we learned

"Autonomous" here means a thinking layer plus an execution layer — not an LLM that trains models by itself
Ranking quality in this challenge is about user-level order, so evaluation must stay tied to the organizer script
Two different inductive biases (attention recalibration vs. explicit crosses) combine better when ranks are calibrated before blending

What's next

Richer code diffs in the agent prompt
Sequence features over user history
Using the random-exposure log as an extra unbiased validation check — still without touching the hidden test set for decisions

The Python trainer does the work; Gemini is the research advisor across iterations — keeping the run safe and reproducible while still delivering the autonomous ML researcher the problem statement asks for.