Autonomous Recommendation ML System

Inspiration

Everyone's shown that LLMs can write ML code. We wanted to answer a harder question: can an agent do honest research on its own — make a bet, run the experiment, accept the result, and never grade its own work?

Track 2 judges the run logs and intervention count, not just the score. That's a challenge about trust, and we liked that.

What it does

Our agent starts from the official baseline and improves it by itself:

an LLM Researcher proposes one idea per iteration,
an LLM Engineer writes the code,
a plain Python harness does everything that matters — runs the code in a sandbox, scores it with the organizers' sealed evaluator, decides what's promoted, and stops by their convergence rule.

Every improvement also gets a leak test: we flip some users' labels and check the model doesn't care.

Final run: baseline → converged champion in 4 iterations · 32 minutes · ~$1.30 · zero human touches. It beat the official baseline on validation — and by more on the test split.

How we built it

Python, numpy, Gemini via OpenRouter, and a lot of tests (135) on the loop itself. Most of the work was making the agent hard to fool:

code that doesn't compile never runs,
summaries that invent numbers get rejected,
the agent reads a structured logbook of everything it has tried — not a chat history.

Challenges we ran into

Our best score turned out to be fake. We hit 0.6092, celebrated — then found the model was quietly reading validation labels through a counting feature, and our first leak test was too weak to notice. We built a much stricter test, re-measured the honest number, and threw the big one away.

That hurt — and it was the most important thing that happened to the project.

Accomplishments that we're proud of

Catching our own leak instead of shipping it.
A fully autonomous run with zero interventions, where every idea, code change, score, and leak check is committed for judges to read.
The agent independently found the same winning idea we would have picked ourselves.

What we learned

An LLM will happily cheat if the system lets it — honesty has to be built into the plumbing, not requested in the prompt.
Real gains are small and noisy — measure everything, trust nothing unmeasured.
Sometimes the simple model with the right objective beats every fancy architecture.

What's next for KuaiRand Research Agent

Smarter search (a tree of experiments instead of one chain — already prototyped), using the dataset's random-exposure log for less biased decisions, and pointing the same harness at other benchmarks. The agent knows nothing about this dataset that judges could read as hand-tuning — that's the point, and it should travel.