Inspiration
Early in the project our agent surfaced a candidate that scored +0.0314 over the official baseline, roughly ten times what we eventually shipped. It would have looked extraordinary on a leaderboard.
We measured it properly and it was worth +0.0023.
The gap was winner's curse. We had taken the best of many noisy per-group estimates, and the act of selecting a maximum is itself biased upward, because the group that wins is disproportionately the one whose noise happened to be positive. Once each group's effect was estimated out of fold instead of read off the same data that selected it, 92% of the improvement vanished.
That number never made it into a report, and the reason it did not is the entire project. We stopped trying to build an agent that finds improvements and started building one that tries to prove its own findings wrong.
What it does
Falsifier is an autonomous ML research agent for recommender systems. It proposes a hypothesis, writes the code to test it, runs a controlled experiment with negative controls, evaluates against a held out split it is not allowed to tune on, and keeps or rolls back the change.
The task is ranking videos within each user on KuaiRand-Pure, scored by GAUC and nDCG@5:
$$\text{GAUC} = \frac{\sum_u n^{+}_u \cdot \text{AUC}_u}{\sum_u n^{+}_u}, \qquad \text{primary} = \tfrac{1}{2}\left(\text{GAUC} + \text{nDCG@5}\right)$$
Result: validation primary 0.6049 against the official FM baseline's 0.6016.
baseline Falsifier delta GAUC 0.6610 0.6720 +0.0046 nDCG@5 0.5282 0.5378 +0.0021 primary 0.6016 0.6049 +0.0033 We claim no hidden test score anywhere. The hidden test is scored once, by the organizer, and our pipeline is built so the agent can never read it.
How we built it
The agent runs on Gemini, with a documented fallback tier after a quota 429 during the graded run. It converged in 7 hypotheses using 91,430 tokens and 0.196 hours of a 6 hour ceiling, with 2 manual interventions.
Four pieces of machinery do the real work, and all four were built before we needed them.
A selection and confirmation split. Validation users are partitioned by a hashed user ID under a salt fixed before the first run, giving 11,270 selection users and 11,107 confirmation users. Every hypothesis, sweep and early stopping decision sees selection only. You cannot tune against a number you refuse to look at.
Negative controls on everything. Permuted, randomized, quality matched and same model seed groups. A result that does not beat its own control is not a result.
Knowing our own noise floor. We had recorded a 3 seed standard deviation of 0.0001. Running 20 seeds showed the true single seed value is 0.00049, so we had been underestimating our own noise by a factor of 8.
A rule against outcome features. play_time_ms correlates 0.64 with the label and predicts it at 84.7% accuracy from a threshold alone. It is not a feature, it is the label in disguise, an outcome of the very impression being predicted. It is banned as an input, along with profile_stay_time, comment_stay_time and is_profile_enter.
The single structural insight everything rests on is that only within user order is scored. Anything constant within a user cannot reorder that user's candidates, which we verified empirically at GAUC exactly 0.500000. That fact killed more ideas than any experiment did.
Challenges we ran into
A leak we introduced ourselves. A per user statistic scored $-0.0048$, actively harmful, while the permuted and randomized controls came back free. That is not how a failed hypothesis behaves, it is how a leak behaves, because a useless feature costs nothing and noise is free. The statistic was computed from each user's own training labels, so a training row's feature contained that row's label. It was an in fold target encoding one level above where we had already applied the out of fold rule. Rebuilding it so each row draws from that user's other folds recovered 75% of the drop.
A control that would have lied to us. To test whether user_id contributes identity or merely capacity, we permute the user to row assignment. Our first instinct was a bijective user to user remap, which would have been wrong: a bijection is a symmetry of the model, so it trains to a numerically identical result and produces a clean, convincing, completely false null. The permutation has to be at the row level. Done correctly, identity accounts for 108% of the effect and capacity for $-8%$.
Choosing the smaller number. Late on, we found our shipped models were early stopping over all validation rows, including the confirmation half we had promised to hold out. Fixing it moved our headline from 0.6053 down to 0.6049. We shipped the lower number and kept the superseded file hashes in the repo so the swap is auditable.
Auditing our own audit. One of our commit messages claimed a grep returned no consumers of a variable. The consumer was four lines below the definition, because the grep had excluded the file itself. History cannot be amended after pushing, so the correction is written into docs/COMPLIANCE_NOTE.md. An audit that misdescribes its own diff is worth less than no audit.
Accomplishments that we're proud of
Reporting +0.0033 when +0.0314 was sitting there, and being able to show exactly why the larger number was false.
Catching a leak we wrote ourselves, from the shape of the evidence rather than from a stack trace.
Proving the oracle ceiling is not a ceiling. The starter kit publishes a label revealing oracle at 0.8645, and every team looking at that gap sees enormous headroom. We fit calibrated probabilities $q$, then simulated a world where the model is exactly right by drawing $y \sim \text{Bernoulli}(q)$, so that by construction nothing is left to learn:
primary perfect probability model vs simulated labels 0.5915 label oracle vs simulated labels 0.8471 irreducible gap, pure coin flip noise 0.2556 our model vs real labels 0.6053 oracle vs real labels 0.8484 observed gap 0.2431 The observed gap is smaller than the gap a perfect model faces in a world with nothing left to learn. The size of the gap to the oracle is therefore not evidence of remaining headroom.
A repo where a judge can clone, run, and get our exact bytes back. The smoke fingerprint holds at 0.4498908751631786 across every patch, the metric decomposition agrees with the official evaluator to $1.6 \times 10^{-14}$, and the submission CSVs regenerate byte identical from a fresh clone.
What we learned
Features that look powerful often cannot move a within user metric. tab has a long view base rate running from 0.004 to 0.489, which makes it look like the most important variable in the dataset. But 60% of users appear in only one tab, and for them it contributes +0.0001. Its entire apparent power is the level gap between tabs, which the metric never sees. Similarly, 87% of authors and 98% of music IDs own exactly one video, correlating 0.985 and 0.987 with the video prior, so they are video_id wearing a hat.
Matching the loss to the metric can cost you. GAUC is a within user AUC, and AUC is exactly $P(s^{+} > s^{-})$ for a pair from one user, so pointwise cross entropy is a genuine objective mismatch. We implemented within user BPR, $\mathcal{L} = -\log \sigma(s^{+} - s^{-})$, and it lost monotonically at every weighting from $\alpha = 0.1$ to pure pairwise. The mechanism was in our own sampler log: only 33.5% of training rows belong to users holding both a positive and a negative, and a pairwise loss cannot consume the rest. Those discarded rows still teach the model what a good video looks like. The metric ignores single class users, the representation must not.
Fitted blend weights overfit, and we caught it live. We split the selection half again under a different salt into selA, the only rows weights are fitted on, and selB, the only rows results are reported on. The greedy blend scored 0.61194 on selA against base's 0.61174, and 0.60393 on selB against base's 0.60481. It wins where it was fitted and loses by 0.00089 where it was not.
17.5% of the metric is structurally unmovable. 3,917 validation users have a single impression, where model, prior and oracle score identically.
What's next for Falsifier: an ML research agent that audits itself
A longer agent run. The graded run converged in 7 hypotheses. Its convergence rule also discarded a validation best checkpoint worth roughly +0.0005 because it did not clear the acceptance threshold. That is disclosed in the README and fixed in the code for future runs.
Attack the representation, not the objective or the ensemble. Fourteen arms on the final night produced zero wins at two standard errors, with two clean monotone dose response curves pointing away from the ranking loss and away from extra capacity. Both are now measured dead ends rather than untested guesses.
Port the protocol, not the model. The selection and confirmation split, the negative control requirement, and the out of fold rule at every level are dataset independent. The model is worth little outside KuaiRand. The machinery for not fooling yourself transfers everywhere.
Log in or sign up for Devpost to join the conversation.