Building an ML Researcher That Works Without Us
TikTok TechJam 2026 — Track 2: Autonomous ML Research Agent
Inspiration
Every "autonomous AI agent" demo we'd seen before this hackathon had the same shape: an LLM in a loop, a green checkmark at the end, and a claim of credit for whatever improved. The results look impressive and are almost impossible to check. If a model gets better, was it because the agent understood something real, or because it got lucky on attempt 40 and nobody logged attempts 1 through 39?
We wanted to build the opposite of that — not the flashiest agent, the most auditable one. Something where every number in the final report traces back to a line in a ledger file, where "the agent found X" means we can point at the exact iteration, the exact diff, and the exact metric that proves it, and where a negative result gets written down with the same care as a positive one.
Track 2's own judging criteria pushed us the same direction — autonomy is scored on manual-intervention count, and a self-reported "trust us, it was autonomous" doesn't survive contact with that. So the rule we set for ourselves on day one was: nothing gets claimed that a stranger with no access to our chat history couldn't independently verify from the repo alone.
The one-line version
We built a system that improves a recommendation model on its own — proposing ideas, writing real code, testing it, keeping what works and writing down why — and then we walked away. Zero human interventions. Zero GPU-hours. Zero dollars.
It beat the provided baseline, and it documented every step of getting there.
champion val_primary 0.6051 (baseline 0.6016)
budget used 10 / 50 iterations
wall clock 37.3 min / 360 min ceiling
human interventions 0
degraded states handled 100 = 98 provider + 2 experiment (all recovered)
experiment failures 9 RUNTIME, in ledger.jsonl
GPU-hours 0
cost $0.00
Two counters, deliberately separate — a judge comparing them should not read a contradiction:
| counter | where | what it counts | in this run |
|---|---|---|---|
| degraded states | degraded.jsonl |
infrastructure the controller routed around without a human: free-tier rate limits, provider failover, auth failures | 100 — 57 failovers, 38 rate-limit cooldowns, 2 auth failures, 1 full-chain exhaustion, plus 2 experiment-level (1 repair, 1 route-around) |
| experiment failures | ledger.jsonl (error_category) |
candidate programs that failed to run — the agent's own proposals | 9, all RUNTIME |
They overlap by exactly 2 records (the repair and the route-around, which are both). Neither is a human intervention: that count is 0, and it is the one Autonomy is scored on.
What the task actually was
The instinct is to read Track 2 as a modelling competition. It isn't. The deliverable is an agent that does the loop an ML engineer does: form a hypothesis, implement it, measure it, keep it or bin it, repeat, and log the reasoning so a human can follow it.
The recommendation problem is the homework we set our agent to prove the loop works. That reframing drove every decision below.
How we built it
The objective is fixed and small on purpose:
$$ \text{primary} = \frac{\text{GAUC} + \text{nDCG@5}}{2} $$
measured by the organisers' own frozen evaluate.py, against the official FM baseline.
Four parts of the model are editable — features, model, objective, training loop. The
agent rewrites them in real Python. Everything else is off-limits.
PREFLIGHT firewall deletes the hidden test labels before anything runs
validate the row ordering we will have to submit in
reproduce the published baseline (0.6014 vs 0.6016)
INIT establish a measured starting point, not an assumed one
REFINEMENT ┌─ which block should I change? UCB1 over blocks
├─ write 3 variants LLM, free tier
├─ validate before spending anything deterministic checker
├─ train all 3 CPU, ~70s each
├─ score with the official metric FROZEN evaluate.py
└─ promote only if it survives a gate OUR CODE, not the LLM
─────────────── repeat ───────────────
ENSEMBLE blend what the search produced
SUBMISSION write the CSV, validate it, stop
The critical boundary: the LLM writes code and nothing else. It never sees the data, never sees a score, never decides what is kept. Every keep/discard decision is made by deterministic code we wrote against a metric we measured.
The model itself is a hand-written factorisation machine, not a library call — no PyTorch, no scikit-learn, just NumPy:
$$ \hat{y}(\mathbf{x}) = b + \sum_i w_i x_i + \frac{1}{2}\sum_{f=1}^{k}\left[\left(\sum_i v_{i,f} x_i\right)^2 - \sum_i v_{i,f}^2 x_i^2\right] $$
That second term is the standard FM trick — it lets you compute every pairwise feature interaction in time linear in the number of features instead of quadratic, which is the entire reason a candidate can retrain from scratch in about 70 seconds on a CPU and the search loop can afford dozens of them.
Four things we think matter
1. We measured our own selection rule instead of assuming it
Every team has to pick some way to rank variants. Most use a single validation split, because that's what the starter code hands you.
We asked a different question: how do we know our selector is any good?
So we ran an experiment to measure the selector itself, using rolling-origin validation, which respects the fact that this data is a time series and the test set is a later week than validation:
| Selection rule | Spearman correlation with true ranking |
|---|---|
| Single split (the default) | 0.833 |
| Rolling-origin h(s) | 0.905 |
And it was not a cosmetic difference: there were candidates where the single split selected the wrong champion and h(s) selected the right one.
This is the finding we would point at first. It is not a better model; it is evidence that our process for choosing models is sound.
2. We found the gate was wrong, and have the receipts
Our confirmation tier required a candidate to beat the champion by more than a measured noise floor. Sensible in principle. In practice:
iter 06 llm_features_v2 val_primary 0.6038 (+0.0004)
GATE REJECT — below noise floor (+0.0004 <= 0.0013)
iter 10 ENSEMBLE uses that same rejected candidate → 0.6044
Our gate was rejecting candidates that our own ensemble then depended on.
The fix was not to lower the floor — that just moves the problem. We changed the question from "is the gain big?" to "does the gain show up consistently?": require improvement on every rolling-origin fold. A real effect appears in both time windows; noise does not.
The repaired gate proved itself in the final run:
h(s) confirm: 0.6115 vs champion 0.6111 (+0.0004, all-folds=True) → PROMOTE
h(s) confirm: 0.6127 vs champion 0.6115 (+0.0012, all-folds=True) → PROMOTE
Both of those are below the old threshold. Both were real, and the run kept both.
3. We put a number on something most teams will describe in words
The judging brief specifically flags impressions vs. negatives, exposure bias from the old policy, and ranking rather than accuracy.
We measured the exposure bias directly:
| Impression source | long_view rate |
|---|---|
| Algorithmically exposed | 0.3133 |
| Randomly exposed (same week) | 0.0806 |
A 3.9× gap. The recommender that produced this data was already good at surfacing videos a user would watch for a long time. That is not the model's achievement; it is the logging policy's.
This has a consequence worth stating plainly, because it is the opposite of the obvious move:
The hidden test set carries the same exposure bias. So optimising an unbiased objective would actively hurt our competition score.
We therefore use the unbiased (random-exposure) estimate as a veto, not as the objective — it can stop a candidate that is gaming the bias, but it does not steer the search. Knowing why to optimise a biased objective is a different thing from not noticing the bias exists.
4. Two independent implementations converged on the same number
Two of us built this separately, with separate codebases, and compared notes late.
We measured eight directions. Nearly all were flat or negative. The instructive case:
duration cross-feature Approach 1: −0.0047
Approach 2: −0.0045
Two strangers' codebases, same number to four decimal places. And a ninth direction that one of us found valuable (+0.0024) precisely because the other's prior had closed that direction off too broadly — which is the best available argument for keeping two independent searches running.
The benchmark is close to saturated. The reachable frontier is around 0.604–0.605, which is why we spent the remaining effort on the loop rather than the model.
What did not work
Recorded as carefully as what did:
| Direction | Δ |
|---|---|
| Censored watch-time | −0.0289 |
| Watch-time auxiliary objective | −0.0144 |
| User × author cross | −0.0063 |
| User × duration cross | −0.0045 |
| Learning rate 3e-3 | −0.0030 |
| MLP head | −0.0006 |
| Recency weighting | +0.0004 |
Per-video target-encoded like-rate (train-fit, smoothed, bucketed) |
+0.0001 on valid, but rolling-origin folds it +0.0006 / −0.0004 — the exact failure mode the fold-consistency gate exists to catch, not just a flat no-op |
The like-rate row draws on the problem statement's Appendix A.3: KuaiRand's other
feedback signals are fair game as features even though only long_view is scored. It was
implemented with train-only fitting and smoothing toward the global rate rather than a raw
aggregate, tested with the same rolling-origin check the real gate uses, and found not to
survive it. A single-split read would have called +0.0001 harmless and possibly promoted
it; the fold check shows it moving in opposite directions on two different windows. That's
one data point, not a verdict on dynamic, history-based features as a category — a
distinction we come back to in "what we would do with more time" below.
The firewall is structural, not a promise it makes
The downloaded dataset secretly contains the hidden test labels. Anyone could score against them.
We delete them before the agent starts, and the check is verified rather than asserted:
rows_standard_dropped_by_rule1 170,588
That is exactly the test-set row count. A patch that tries to cheat does not get caught by a checker that "thinks it looks wrong" — it crashes, because the data it needs is not there.
evaluate.py and submit.py are used from the starter kit as shipped, hash-verified at
startup and never edited by the agent. 21 automated
checks (python3 verify.py, exit code = failure count) cover submission format, row
ordering, and firewall integrity.
Challenges we ran into
Free-tier APIs that don't want to cooperate
A six-hour autonomous run against no-credit-card free tiers means rate limits are not an
edge case, they're the normal operating condition. Our first router design marked a
provider permanently dead on a single 429 — one transient rate-limit and a perfectly good,
still-working provider was retired for the rest of the run. We rebuilt it around
time-boxed cooldowns instead: a provider that gets rate-limited comes back after the free
tier's own window resets, rather than being written off. That's most of where the 100
"degraded states" in the stats above come from — not failures, recoveries. It's also how
we found out three of our original candidate providers (Cerebras, DeepSeek, SambaNova)
authenticate but never actually serve a request on our accounts — tools/check_providers.py
checks that live rather than assuming a key that authenticates is a key that works, and all
three got dropped from the chain once that was confirmed.
The bug we found in our own submitted log
This happened on our first full run (now kept as runs/final_v1/, champion
0.604632). The run finished and wrote a submission. Reviewing it afterwards, we found
the run had submitted the wrong model:
- It shipped a cross-candidate blend scoring 0.60410
- A champion scoring 0.604632 had been promoted two iterations later
The blend was computed once, on entering the ENSEMBLE phase, and never re-checked against champions promoted afterwards. Fixed: the blend is now recomputed at submission and always compared against the current champion.
We then re-evaluated properly over the full candidate pool:
best blend (it006 + it008) 0.60471 (+0.00008 over champion)
trivia floor 0.0002
baseline 5-seed std 0.0008
The gain sat inside both the floor and the noise. So we respected our own threshold
and submitted the single champion — and the fix held on the run that actually shipped:
our submitted run (runs/final/, champion 0.605146) went through the same fixed
submit() path and needed no correction.
We could have taken the +0.00008. We didn't, for the same reason we didn't refit on validation — see below. The correction is appended to the submitted log, not deleted from it.
The rule we followed even though it cost us
The judge said: "should only use train set to train the model."
There is a standard trick here: after using validation for early stopping, refit on train + validation together. More data, closer in time to the test week. We measured it at +0.002.
We switched it off.
The trade is asymmetric. +0.002 is about 2.5× the baseline's own seed noise, so it barely
registers, while being caught training on validation after being told not to would cost
credibility across every other criterion. The flag exists in the code, defaults to
False, and its measured value is documented, so anyone can check the work.
We use time_decay = 0.90 instead: it up-weights recent training rows, captures much of
the same "newer data is more useful" signal, and never touches a validation label.
What we learned
A model getting better and a process getting more trustworthy are different projects. We spent more engineering time on the gate, the selector, and the ledger than on the model itself, and in hindsight that was the right allocation — the model was never the bottleneck here. Knowing whether to believe a result was.
"Zero interventions" only means something if it's checkable. We were tempted, early on, to just report it the way most agent demos do — a number in a slide. Instead the count comes from parsing our own commit history against the one file a human is allowed to touch mid-run, so anyone can rerun that check themselves rather than take our word for it.
Rejecting a candidate is not the same as the candidate being wrong. The gate bug above taught us this the hard way — a threshold that's too strict doesn't fail loudly, it fails by quietly discarding real signal, and you only find out if you go looking for the cases where "rejected" and "actually used later" overlap.
Negative results are cheap to log and expensive to hide. Every row in the "what did not work" table cost us one iteration of budget, once. If we hadn't written it down, some future run — ours or anyone else's — would burn another iteration re-discovering the same dead end.
Cost, and why it is zero
Free tiers only — Gemini, Groq, NVIDIA NIM, OpenRouter (two routes) and Mistral, in a
fallback chain. Six hops means more free daily quota than any single provider gives, which
is what a multi-hour autonomous run needs. No credit card required to reproduce this.
Cerebras, DeepSeek and SambaNova authenticate but never serve on our accounts
(tools/check_providers.py confirms this live), and were dropped from the chain rather
than left as dead hops.
| API cost | $0.00 |
| GPU-hours | 0 |
| Compute | MacBook Air, CPU only |
| Tokens (submitted run) | 97,660 in / 39,978 out, across 4 live providers |
The context packet sent to the LLM is 1,895 bytes. It never sees data, labels, or scores — only structural descriptions. That is what keeps both the cost and the leakage surface negligible.
Reproducing it
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python3 -m aura preflight # must print PREFLIGHT OK
python3 verify.py # 21 checks, 0 failures
python3 -m aura run --run-dir runs/demo
Full setup, including the 194 MB dataset download, is in the repository README.
Runs with no API keys at all, driven by a deterministic prior queue — so the loop, gating and logging can be verified without any external service.
Keys go in .env (gitignored). Setup takes about five minutes.
Built with
- Language: Python 3.9, NumPy (the model is a hand-written factorisation machine — no ML framework, which is why it trains in ~70s on a CPU)
- LLM routing: LiteLLM across the Gemini, Groq, NVIDIA NIM, OpenRouter and Mistral free tiers, with per-provider RPM pacing and rate-limit cooldowns (not permanent retirement)
- Validation: rolling-origin temporal splits; official frozen
evaluate.py - Safety: AST-based static checks on generated code + filesystem-level label removal
- Dataset: KuaiRand-Pure (Kuaishou), 1,436,609 interactions over 27,077 users and 7,551 videos in the two standard logs (1,141,112 train / 124,909 valid / 170,588 test)
- Development tools: VS Code, Claude Code (AI pair programming, used throughout — disclosed here because the Devpost asks for tools used)
What we would do with more time
The feature space is the part we explored least, and we know it
Our agent is genuinely good at tuning what's already there — objective, learning rate,
batch size, blending, regularisation. It is bad at asking whether the input is the
actual bottleneck. Almost every candidate it has ever proposed edits model, losses,
or training; in the whole run, essentially nothing touched features beyond the config
it was seeded with. Our own "what did not work" table says why: we only ever gave it five
static categorical fields to work with, and every attempt at adding more of the same
kind of field measured flat (organisers' own numbers: 0.5940 vs 0.5950). The one time we
tried something structurally different — a per-video target-encoded engagement rate — it
looked promising on a single split and then failed the rolling-origin check. That's one
data point, not a verdict on dynamic features as a category, and we've deliberately
written it that narrowly rather than either oversell it or close the door on the whole
direction.
That leaves an honest gap: we don't know how much score is sitting in temporal structure — a user's or a video's interaction history up to the point of the impression, not just their static id — because we never gave the agent, or ourselves in the search loop, a real way to explore it. Handing it more raw CSV columns and hoping wouldn't fix this. The two hard parts are (1) computing anything history-based without leaking the future into the past, which is exactly the kind of bug that stays silent until someone goes looking for it, and (2) making that safe enough to expose to something that proposes structure without a full code review every time. If we had more time, here's specifically what we'd build, in order:
A feature store, not a feature file. Right now
build_featuresassembles a fixed list of columns from a static npz. We'd add a second, precomputed layer — running counts and rates per user, per video, per (user, video) pair, computed once with a strict read-then-write ordering so a row can never see its own outcome — and let the agent choose which of those to include, instead of writing the aggregation logic itself. The dangerous part stays deterministic code we write and unit-test once; the part the agent gets to search over is small and structurally safe to hand it.A declarative feature spec instead of free-form code, for exactly that reason.
{"scope": "video_id", "signal": "like", "stat": "rate"}is something we can validate before it ever touches a training run; a hand-written aggregation function is something we'd have to review for a subtle ordering bug every single time. This is the same trade our whole proposal-validation layer already makes for hyperparameters — structure the LLM can propose freely, deterministic code decides if it's even legal to try — just extended one level into feature space instead of stopping at hyperparameters.A cheap pre-screen before paying for a full retrain. Every idea currently costs a complete model fit to evaluate — ~70 seconds at minimum. Most of what's in our "did not work" table would have shown up as flat on a five-second correlation check against the label, no training required. A relevance filter ahead of the expensive path would let us spend iteration budget on the minority of ideas actually worth a full measurement.
Let the agent see where it's wrong, not just how wrong. Every proposal today is grounded in one scalar — the current primary metric. If the context we hand it also included a breakdown (is the champion weaker on cold-start users, on rarely-seen videos, on short sessions), its hypotheses about what to try next would be grounded in an observed weakness instead of a guess dressed up as one.
None of this is a guarantee the score moves — the benchmark may genuinely be close to saturated on this data, and we'd rather say that honestly than promise a number we haven't measured. What we are confident of is narrower and, we think, more useful: right now, feature engineering is the one lever in this whole system that only we can pull, by hand, before a run starts. Closing that gap is what would take the "autonomous research agent" claim from accurate-with-an-asterisk to complete.
Everything else on the list
- Cross-run memory. Each run starts cold. Nine iterations is short; a diary that persisted across runs would compound instead of re-deriving the same good ideas every time.
- Hypothesis-level search. The agent mutates parameters well but does not yet form structural hypotheses — "the bias is in the sampling, so change the sampling" is a different kind of proposal than "try lr=0.0015," and we don't currently give it room to make the first kind.
- Tighter proposal validation. It rejected 11 proposals pre-flight this run, and every one of those was an experiment we did not have to pay for. There are more classes of "obviously wrong" we haven't taught the checker yet.
The score shows the agent works. The log is what we would actually hand to another engineer.
Log in or sign up for Devpost to join the conversation.