Tippytop

Inspiration

ML research is a loop someone runs at night. Have a hunch, write the code, train, squint at a number, decide what to try next. Repeat forty times.

We wanted to know if the loop could be the program.

But we treated that as a question about people before we treated it as a question about code. The agent is meant to replace a human doing a specific job, so we started by asking how that human actually works.

Where does a researcher's next idea come from? Never from nowhere. An experienced ML engineer staring at a dataset and a score to beat draws on two things, and every decision they make combines both.

The first is feedback from the system. You run an experiment, read the result, react to it. Score went up, push further in that direction. Score dropped, back out and try something else. This is knowledge produced during the work.

The second is background knowledge. What you already knew before you started: your training, your intuition, the methods you carry in your head, plus what you go and read. Papers, public solutions, prior work on similar problems. This is knowledge brought into the work.

The insight that shaped our whole design is that these are not separate stages. They operate together, at every step. Choosing a feature, choosing a model, and choosing a learning rate are all the same act: read the feedback so far, combine it with what you know, decide the next move.

What it does

Give it the benchmark and walk away.

How it addresses the problem statement. The brief asks for an agent that runs the full ML iteration loop on its own: inspect the data, engineer features, choose and tune a model, evaluate, reflect, try again, and keep improving until it beats the official Factorization Machine baseline, with as little human intervention as possible. Tippytop does exactly that. It runs deterministic EDA over the splits, proposes a named falsifiable concept, writes the code, trains, scores itself on validation only, judges the result against its own measured noise floor, and decides what to do next. It stops by the stated rule and submits its validation best checkpoint. The hidden test set is computed for reporting and never enters a prompt.

Result. One run, converged by the stated rule, zero manual interventions.

Validation best 0.6045 against the baseline's 0.6015. Five iterations of fifty. 0.30 hours of a six hour ceiling. 37,379 input and 26,104 output tokens. Zero GPU hours. Zero manual interventions.

Five iterations is the rule, not the ceiling. The brief defines convergence as no validation improvement beyond 0.002 across three consecutive iterations. The largest honest single step gain this benchmark offers is about +0.001, which we measured across seven independent runs. So any agent that actually obeys the stopping rule finishes in single digits here. Ours stopped because its own best score had not moved for three straight experiments. It used 0.30 of a six hour budget because it had converged, not because it ran out of road. We could have disabled the rule and filled fifty iterations with noise. We report the converged result instead.

From eleven candidate directions in its context, it picked field aware factorization machines on iteration 2, implemented it from scratch, then pivoted on iteration 4 to within session position reconstructed from an unused timestamp column. Combining the two was its own call. Nothing suggested it, and it took five iterations to reach what took us weeks by hand.

How we built it

What we studied. To ground the design we looked at how existing autonomous ML agents structure this loop, including the autoresearch line of work, which frames ML engineering as an agent that edits a training script, runs it, reads the metric, and keeps or discards the change on repeat. That gave us the skeleton: propose a change, apply it, run it, score it, decide, go again.

Mapping the human workflow onto the agent. We broke the recommender workflow into the stages a person moves through. Exploratory data analysis, to understand the data's shape before touching a model. Feature selection, to decide what the model gets to see. Model selection, to choose the architecture that fits the problem. Hyperparameter tuning, to refine the chosen model's settings.

At every one of those stages, the thing informing the next change is the same pair of sources. Feedback supplies the tweaks and test results, telling the agent what happened when it last changed something. Background knowledge supplies the space of things worth trying: the model's own parametric knowledge of architectures and losses, extended by the papers and dataset specific findings we give it as context. So the agent's job each turn is to fuse the two.

Our design: explore and exploit. Feedback tells the agent whether the last move worked. It still needs a rule for what to do about that, and ours is the classic one. When an idea is working and the score is climbing, the agent exploits: it stays on the concept and pushes further, refining rather than replacing. When an idea has failed and the score dropped, the agent explores: it abandons that direction for something genuinely different. When an idea worked but plateaued, the agent expands to an adjacent concept that builds on the win. And when code simply breaks, the agent repairs and retries before treating it as a failed idea.

That gives it a coherent, legible policy for driving its own research. Feedback is the input, explore and exploit is the decision, and the loop is the machine that runs them together, the same way a person moves between doubling down on a promising lead and cutting a dead end loose.

Crucially, the policy decides only the kind of move. The specific change is the agent's own call, from its background knowledge and the run so far. That separation is what keeps the process genuinely autonomous rather than a fixed script: the agent reasons about its own results instead of executing a human written checklist.

Development tools. VS Code on Windows 11, PowerShell, Git and GitHub, LangGraph Studio for stepping through the graph visually, pytest for the test suite (200 plus tests), and uv for dependency management. No notebooks. Every number in this project comes from a script that can be re-run.

APIs. The Anthropic Messages API (claude sonnet 5) for hypothesis generation and code writing, called through tool use with a strict JSON schema so proposals arrive structured rather than parsed out of prose. The OpenAI Chat Completions API is supported as an alternative provider but was not used for the submission run.

Libraries and frameworks. LangGraph, so the loop is a graph rather than a while statement. NumPy for all models. LightGBM and scikit learn for the tree lane. SQLite for experiment checkpoints, JSONL for the run log, python dotenv for configuration.

Datasets and assets. KuaiRand Pure only: the six CSVs published by the organisers (Zenodo record 10439422), used as given. Train 2022 04 08 to 04 21 (1,141,112 rows), validation 04 22 to 04 28 (124,909), hidden test 04 29 to 05 08 (170,588), label long_view. No external data, no pretrained weights, no manual labelling. Item and author level engagement aggregates are computed from the training split only, leave one out per row. We deliberately did not use the shipped video_features_statistic_pure.csv, because its aggregation window is undocumented and may span the test period.

Architecture. Propose, experiment, critic, router, check convergence. The experiment stage is its own sub graph so every step has an explicit failure edge rather than a try/except that swallows things. Each experiment gets its own folder, so reverting means never reading that folder again.

What we got wrong first was the judging. Our early version had two outcomes: better, or failed. It proposed field aware FM, scored dead level with the incumbent, was told it had failed, and walked away from what turned out to be the best model in the project. A good idea's first implementation arrives untuned and lands at a tie. So we added a third outcome, parity, and taught the router to give it one bounded refinement before pivoting. That fix is the reason the final run works.

Challenges we ran into

Almost every hard problem was one we had created ourselves.

We were leaking the test set through a stack trace. The agent never sees test metrics, which we enforced carefully. But when an experiment crashes, the tail of its output becomes the error message, and that message goes back to the model so it can learn what broke. Our training script prints its summary, including the test score, before some crashes happen. One badly timed crash would have put the answer into the prompt that chooses the next experiment.

The agent could have overwritten its own scorer. The model chooses the filenames it writes. Nothing stopped it writing a path that escaped its folder, or replacing evaluate.py with its own version. "Score yourself 1.0" is the highest reward move available to anything optimising the number we read, and the log would have recorded it as a triumph.

Our first real run produced no submission at all. The starter kit's submission builder hardcodes the baseline model class. The moment our agent renamed it, the whole thing died. The silent version was worse: keep the name, change the configuration, and it writes a perfectly valid file containing the original model's predictions. We would have submitted the baseline and never known.

We were stopping the agent for the wrong reasons. Crashes counted toward the "no progress" counter, so three broken experiments in a row ended the run. The agent was quitting because it hit bugs, not because it was done.

Accomplishments that we're proud of

Zero interventions, measured rather than claimed. Our first version reported a hardcoded zero. The field existed, reached the report, and nothing ever incremented it. A zero you assert is worth nothing. Now a restart is detected whether or not the operator admits to it, and every reason is written into the report.

We closed three of the organisers' own suggested directions with evidence. Ranking losses win seven of seven controlled comparisons and still lose, because the grouped batching they require costs more than the objective returns. Multi task heads fail six of six against their own control, because is_click correlates 0.760 with the label and is nearly a duplicate task. User side features score exactly random GAUC of 0.5000: a feature constant within a user shifts all their scores equally and cannot reorder anything.

And it built something we hadn't. We had listed both ingredients as candidate directions and never tested either. It implemented both and combined them, unattended, in eighteen minutes.

What we learned

The ceiling isn't 1.0, it's 0.8645. On the test split, 27.1% of users have no positive label and 9.2% are all positive. Their nDCG is fixed regardless of any model, and GAUC excludes them entirely. Every headroom estimate we made before working that out was wrong.

We ran out of measurable, not out of ideas. Seed variation is about 0.0003, but the uncertainty from which users happen to be in the eval set is about 0.0008. More seeds cannot touch that. Below roughly 0.002 we cannot tell a real gain from luck, which is why we report paired bootstrap confidence intervals rather than differences of scalars.

Best of N lies. Pick the best of several noisy runs and you manufacture an improvement out of nothing. We caught ourselves publishing an inflated figure and had to correct it in our own notes.

Our agent's short runs aren't laziness. The convergence threshold we were given is larger than any single step gain this benchmark offers. An agent obeying the rule finishes in single digits. That is the rule working, not the agent giving up, and realising it changed how we read every result.

What's next for TippyTop

Ensembling. Worth +0.0013, replicated on disjoint seed halves, and it does not fit inside the ten minute per experiment budget at our model's cost. The agent tried, timed out, recovered and re-scoped, which is the right behaviour, but the gain stays uncollected. Cache the encoding across members, or use a cheaper base model, and it becomes reachable.

Align the loss with the metric. Our training loss weights a 40 impression user forty times more than a 3 impression user, while nDCG@5 counts every user once and GAUC excludes the inert ones entirely. About a quarter of the gradient comes from users who cannot move the score. Weighting each row by 1/n_u is a weight vector away, and nobody has run it.

The bonus benchmarks. KuaiRand 1k and 27k are scoped out for now. The loader hardcodes the Pure filenames, 27k is roughly 230 times the data for a CPU pipeline, and neither ships an official baseline to compute a delta against.