Autonomous ML Research Agent — KuaiRand-Pure

TikTok TechJam 2026, Track 2 — solo entry

What it does

An LLM-driven agent that autonomously improves a recommender-systems pipeline on the KuaiRand-Pure benchmark. Each iteration it looks at the current best pipeline's real source code, a ground-truth profile of the data, and its own recent history, then proposes one focused code change with a stated hypothesis and research direction. The candidate runs, gets scored against the official evaluator, and the agent decides whether to keep it, reject it, or move on — including auto-rejecting its own results when they look like they came from data leakage rather than a real improvement. No human touches a single iteration once the run starts.

Inspiration

Track 2's framing — automate the MLE loop of read the problem → inspect data → engineer features → train → evaluate → reflect — is a genuinely different kind of challenge from "build the best model." The interesting part isn't picking a model architecture, it's whether a system can drive its own research process: propose an idea, test it honestly, and know when to trust or distrust its own result. That's the part I wanted to actually build, rather than a hand-tuned pipeline dressed up as an agent.

How we built it

The loop is deliberately split into two layers: a generic propose → execute → evaluate → reflect machine, and a thin, swappable layer of KuaiRand-specific context (the field schema, the organizers' own confirmed dead-ends and candidate directions, pulled straight from their starter kit).

• Bootstrapping is deterministic, not LLM-guessed. The starting pipeline is built by reading the organizers' real baseline.py off disk at runtime — never a hand-retyped copy — so iteration 0 is guaranteed byte-identical to the official reference before the LLM ever runs.
• Every proposal is grounded in real, computed facts, not just prose instructions: a data profile (dtypes, shapes, class balance) is computed fresh each run from the actual loader and handed to the model directly, since prose warnings alone weren't reliably followed.
• The candidate script format is plain text, not JSON. Early on, asking the model to return code inside a JSON string field broke constantly on unescaped quotes and newlines — switching to a DIRECTION:/HYPOTHESIS: header plus a fenced code block removed that failure class entirely.
• Both failure paths — a bad LLM response and a bad candidate script — are caught and logged, not fatal. A single malformed proposal or a runtime error just gets recorded and the loop moves to the next iteration.
• An anti-repetition rule stops the agent from re-opening a hypothesis with "since the top-ranked direction is loss mismatch..." after that exact direction already failed twice in a row — a pattern that showed up repeatedly in early runs.
• The submission file is never reimplemented — the winning iteration's raw scores are handed to the organizers' own submit.py:write_submission(), so the output format can't drift from what their checker expects.

Convergence matches the brief exactly: the run stops when validation primary hasn't improved by more than \( \varepsilon = 0.002 \) over \( N = 3 \) consecutive iterations, or the iteration/wall-clock caps are hit — whichever comes first.

Challenges we ran into

The honest headline challenge was catching my own agent cheating — twice.

The scoring metric is:

$$\text{primary} = \frac{1}{2}(\text{GAUC} + \text{nDCG@5})$$

and Task Requirement 4 scores, on the hidden test set:

$$\Delta(m) = \text{score}_{\text{agent}}(m) - \text{score}_{\text{baseline}}(m)$$

That means a validation number that looks great is worthless — worse than worthless — if it doesn't reflect something the model actually learned from train data.

A run early on committed a result that looked like a breakthrough: validation primary jumped from ~0.60 to 0.62 in a single iteration. It turned out the "user history" feature that iteration invented was computing each row's feature from other rows in the same validation/test split — an expanding accumulator that leaked labels forward. The first fix (a validation-minus-test gap check) closed the obvious version of this, since a naive leak inflates validation while leaving test flat. A later run found a subtler variant of the same bug: a feature that leaked symmetrically into both validation and test, so the gap stayed normal and the first guard missed it entirely. Closing that took two more layers: a static scan for any loop over validation/test rows that reads a label, and a behavioral backstop that auto-rejects any single-iteration jump implausibly larger than every legitimate gain observed across the whole project.

Getting the agent's response format itself to be reliably parseable — without falling back to something rigid like JSON, which broke on code containing ordinary quotes and newlines — took a few iterations to land on plain delimited text.

What we learned

That an "autonomous agent" is only as trustworthy as its own skepticism toward its own results. The scaffolding around the model — what ground truth it's shown, what it's forbidden from doing, and what gets automatically distrusted rather than accepted — mattered at least as much as the model's raw capability at proposing ideas. A model that's good at ML but has no guardrail against leakage will confidently hand you a number that looks like progress and isn't; the interesting engineering problem turned out to be building the part that catches that, not the part that proposes changes.

Result

Final result: validation primary 0.6043, hidden-test primary 0.5970, versus the official baseline's 0.5946 — a delta of +0.0024, independently verified against the organizers' own submit.py --score checker.

Development tools used

• VS Code, with Claude Code as the coding agent for implementation
• Claude (Anthropic) for design discussion, debugging, and code review

APIs used

• An OpenAI-compatible chat completions endpoint (school-provided, served via vLLM), model qwen3-coder-next

Libraries and frameworks used

• numpy only for the ML pipeline itself (per the competition's own constraint — no torch/pandas/sklearn)
• openai (Python client) and python-dotenv for the agent's own infrastructure (LLM calls, credential loading)

Datasets and assets used

• KuaiRand-Pure, as provided by the organizers' frozen starter kit

Built With

• numpy
• openai
• python
• qwen
• vllm