RankForge

Inspiration

Machine learning research is mostly a person sitting down and trying things. You look at the data, you get a hunch, you write some code, you check the score, and the score tells you your hunch was wrong. Then you do it again. Most of the ideas fail.

We wanted to know whether an AI could do that job — not the coding part, which is easy now, but the deciding part. Could it look at a result, work out what the result implies, and choose what to try next for a reason it could explain?

So we built one and pointed it at a real problem, then got out of its way.

What the problem actually is

TikTok-style apps show you a feed of short videos. For each video the app already decided to show you, we predict whether you will watch it all the way through — the dataset calls this a long view.

The important detail is that we only sort your own feed. We are not picking videos from a catalogue of millions. We are given the handful you actually saw, and we put them in the best order.

The data is KuaiRand-Pure: 27,000 users, 7,500 videos, 1.4 million impressions from the Kuaishou app over a month in 2022.

Two things are scored, and the final number is their average:

GAUC — for each person, how often did we rank a video they finished above one they skipped?
nDCG@5 — did we get the top of their list right?

$$\text{primary} = \tfrac{1}{2}\left(\text{GAUC} + \text{nDCG@5}\right)$$

The organisers gave us a starting model scoring 0.6015. A perfect model would score about 0.86. So there is real room, and it is genuinely hard.

What we built

Two things, and both are judged.

An agent. An LLM that reads the problem, asks questions about the data, forms a hypothesis, writes real Python, runs it, reads the score, and decides what to do next. It repeats until a stopping rule fires.

A model. Whatever that loop produced.

How we built it

The whole design is one idea: split the system into a part that must never break and a part that is free to fail.

Think of a chemistry lab. There are rules you do not touch — the fume hood, the fire extinguisher, the labels on the bottles. And there is the actual experiment, where you are supposed to try things and mostly be wrong.

We built the lab. The agent runs the experiments.

The harness (the lab). Loads the data, checks the answers, saves the results, counts the budget, and sandboxes anything the agent writes. The agent is forbidden from editing any of it, and a validator rejects any change that touches a protected file.

The agent (the researcher). Gets five parts of the pipeline it may rewrite: how the model learns, the model itself, the features, how data is fed in, and how results are combined. It writes the actual code for each one.

The model underneath is a factorization machine. It learns a short list of numbers for each user and each video, and multiplies them together to guess whether they are a match. The important thing is that it learns those numbers from the data rather than being told them.

Everything runs on a laptop CPU. Zero GPUs.

Challenges

The answers were sitting in memory. The organisers' data loader hands you the hidden test set with its true labels attached. Nothing stops you reading them. Nobody would ever know.

So our loader strips them at load. The test rows come back six fields wide instead of seven, so our own feature code physically cannot reach a label — it crashes instead. Twenty-five tests hold that in place. It is the first thing we would point a judge at.

Almost everything failed. The agent tried 43 different ways of changing how the model learns. Not one beat the simple version we started with. That is not a bug — it is what research looks like — but it means the interesting output is why each one failed, not the score.

The agent got stuck on one idea. Two-thirds of its attempts went into the same part of the pipeline. Partly its own bias, and partly ours: that was the only part with a proper interface behind it. When we later opened up features, its behaviour changed immediately.

Broken features are worse than missing ones. Twice we shipped something the agent could reach but that did not quite work. Both times it found the bug by using the feature, drew the wrong conclusion from the crash, and wandered off in a worse direction. A capability that exists but fails costs more than one that was never there.

What we learned

A negative result with an explanation beats a lucky number. We closed eleven directions. Each one has a measurement and a reason. That is a far better answer than "we tried some stuff and this worked."

Check the premise before building on it. Several times we spent twenty minutes measuring whether an idea could work before spending three hours building it. Every single one of those probes came back negative. That is hours saved, and each one became a finding.

The best thing we found, we were not allowed to use. Models normally get retrained on all available data before making their final predictions. The rules were ambiguous about whether we could. We took the strict reading and turned it off — then measured what it would have been worth. The answer was larger than our entire result. We wrote it down and still did not use it.

The agent surprised us once, properly. The organisers had published that facts about the user cannot help here, and the logic is airtight: you are sorting one person's videos, so anything describing that person is the same for all of them and cannot change the order.

The agent worked out this is only half true. Alone, useless — but paired with something that does vary between videos, it becomes useful. Knowing someone is patient tells you nothing. Knowing they are patient and this video is long tells you a lot.

So it deliberately built a feature it knew was worthless on its own, purely so it could be combined. Its own code comment says it is there "only to be crossed."

Nobody told it to do that. That is the moment the project stopped feeling like automation and started feeling like research.

Results

	validation score
Official baseline	0.6015
Our scripted control (no AI)	0.6025
Our agent	0.6036

The middle row is the one we are proudest of building. It is a plain script that tries thirty settings in a row, with no AI at all — built so we could ask honestly whether the agent was thinking or just searching. It was thinking, but only by a little, and we would rather report that than hide the comparison.

Zero manual interventions. Seven experiments out of a fifty-experiment budget. Forty-five minutes. No GPU.