IterateML -Write. Run. Fail. Fix. Repeat — No human in loop

r IterateML -Write. Run. Fail. Fix. Repeat — No human in loop

Inspiration

Machine learning engineers spend most of their time in the same loop: read the problem, inspect the data, engineer features, train and tune, evaluate, reflect, repeat. It's structured and repetitive — which is exactly what makes it a good target for automation. TikTok TechJam's Track 2 challenge asked for an agent that could run this loop autonomously on a real recommender-systems benchmark (KuaiRand-Pure), and I wanted to build one that didn't just try random things, but actually learned from its own failures the way a real engineer would.

What it does

IterateML is an autonomous ML research agent that:

Reads the official baseline and problem definition Proposes a hypothesis for how to beat it Writes a complete Python training/evaluation script Runs it, and reads back either a real score or the exact error Uses that outcome to decide what to try next — with zero human input during the run

Over multiple runs, the agent tried hyperparameter tuning, feature changes, and more ambitious ideas like pairwise (BPR-style) and listwise ranking losses — theoretically the most promising direction, since the benchmark's metrics (GAUC, nDCG@5) are ranking metrics, but the baseline trains with plain pointwise logloss.

How we built it

The core is a loop written around the OpenAI API: each iteration, the model gets a system prompt describing the exact data-loading and evaluation interface, a summary of what's already been tried (and what failed), and is asked to propose one hypothesis and write a full script for it. The script is validated (syntax check + banned-import check) before execution, run in a sandboxed subprocess with a timeout, and the result — success or a specific error — is logged and fed back into the next prompt.

Two design choices mattered a lot once real runs happened:

Topic-level banning: if the agent fails 2+ times on a similar approach (e.g. a hand-rolled pairwise-loss gradient), that approach gets explicitly banned from future attempts rather than being retried indefinitely. Anchoring on the best script, not just its score: later iterations are shown the actual code of the best-performing attempt so far, so the agent builds incrementally rather than reinventing a working solution from scratch every round.

Solo project — Python, numpy (matching the starter kit's numpy-only constraint), and the OpenAI API (gpt-4o) as the agent's reasoning/code-generation engine.

Challenges we ran into

Environment setup ate more time than the ML problem itself did — getting a working Python/PowerShell environment, then hitting free-tier LLM API rate limits (both per-minute and per-day caps) before switching to a different provider entirely. Silent numpy/JSON incompatibility: evaluate.py's math returns numpy scalar types (float32/float64), which json.dumps() can't serialize — this quietly broke the agent's structured-result parsing until it was traced back and fixed by forcing explicit float() casting in every generated script. Hand-implemented ranking losses are genuinely hard to get right in one shot. Multiple iterations failed on subtle gradient/shape bugs in custom pairwise and listwise loss implementations — a good reminder that "theoretically the best idea" and "reliably implementable by an LLM in a single pass" aren't the same thing. Timeouts on later runs: some generated training loops were slower than expected and hit the 5-minute per-iteration cap, cutting a run short before it could explore as many hypotheses as hoped.

Accomplishments that we're proud of

Built a fully autonomous loop that runs end-to-end with zero manual intervention during any individual run — every fix and improvement happened between runs, not by touching the agent mid-execution. The agent's best result (primary = 0.6001, iteration 1 of the second run) landed within 0.0015 of the official baseline's validation score (0.6016) — inside the range of normal seed-to-seed variance, given the baseline's own published standard deviation across 5 seeds is 0.0008. Diagnosed and fixed a genuinely subtle bug (numpy types breaking JSON serialization) that was silently causing failures across multiple runs, by tracing the exact line a traceback pointed to rather than guessing. Built real self-correction into the loop: after the first full run showed the agent repeatedly failing on hand-rolled pairwise/listwise loss code, the prompt strategy was redesigned to ban repeated failed approaches and anchor new attempts on the best working script — and the very next run's first iteration immediately produced the best result of the whole project.

What we learned

The most useful realization was that autonomy isn't just "let the model try things" — it's building the scaffolding that turns raw LLM output into a system that actually gets better at trying things over time: catching cheap errors before wasting a full run, remembering what's already failed, and giving the next attempt real context instead of starting cold each round.

What's next for IterateML -Write. Run. Fail. Fix. Repeat — No human in loop

A "critic" pass that reviews generated code for likely bugs (shape mismatches, undefined variables) before execution, instead of only discovering them at runtime Generating and pre-screening multiple hypotheses per round instead of committing to one A cost-aware stopping policy that weighs expected improvement against remaining token/time budget, instead of a fixed convergence rule Investigating why some generated training loops ran slower than expected, to reduce timeout-related wasted iterations Results GAUC nDCG@5 primary Official baseline (validation) 0.6674 0.5357 0.6016 IterateML best result 0.6654 0.5347 0.6001

Combined across both runs:

First run: 15 iterations, fully converged naturally, best primary = 0.5987, 28.6 minutes wall-clock, 47,114 tokens, 0 manual interventions during the run Second run (with topic-banning and best-script anchoring added): 9 iterations completed before being manually stopped given time constraints, best primary = 0.6001 (iteration 1), 0 manual interventions during the run itself Total: 24 agent iterations across both runs, 0 manual interventions during any individual run — interventions occurred only between runs, to fix environment issues and improve the agent's prompting strategy based on the first run's results

Full per-iteration hypotheses, code diffs, results, and errors are in run_log.jsonl / run_log_attempt1.jsonl in the repository, alongside every script the agent generated.