swe hell

Inspiration

We began with a challenge that sounded simple: beat the Factorization Machine (FM) baseline on the KuaiRand-Pure short-video ranking benchmark, where the primary score is calculated as:

primary = (GAUC + nDCG@5) / 2

However, the starter kit ships with a warning that most people probably skip: the FM baseline scores 0.5946, and the oracle (perfectly ranking every impression with the true labels) only reaches 0.8645. Since 27.1% of users have zero positive labels, so their nDCG is pinned to 0 no matter what you do. In other words, the real headroom is 0.27, not 0.41, and the baseline has already eaten 30.7% of it, resulting in very little room for improvement.

That framing is what made this a good agent problem rather than a good Kaggle problem. On a benchmark like this, the FM's 5-seed standard deviation is 0.0008. Every improvement worth having lives in the third or fourth decimal place. At that scale, it is dangerously easy for an eager LLM to happily hand you a hallucinated +0.03 of noise as progress, leak information from the test set, or make a no-op change and report its parent model's score as an improvement.

So we stopped trying to build an agent that gets a high score. We built one that cannot lie to us about its score, and then let it search autonomously.

The name swe hell is affectionate. Anyone who has worked on machine learning experiments knows the feeling of staring at a 0.00021 improvement late at night and wondering whether it is real. Our project automates that difficult, repetitive, and surprisingly human part of ML engineering.

What it does

swe hell runs the full research loop autonomously, with one human intervention in six hours:

Diagnose — a Diagnostician persona reads the run trajectory (per-iteration running best, the plateau signal, prior evidence) and commits to one bottleneck, with a confidence and an edit_radius.
Ground — a Literature persona does a web-search-backed lookup for published techniques addressing that bottleneck, returning a structured evidence card. Cached to disk by normalized bottleneck string, so the same question is never paid for twice.
Hypothesise — a Hypothesis persona emits 1 proposal normally, or 3 when confidence is low or the trajectory looks flat. Each one must carry a mechanism, a pre-registered paired success criterion, and an implementation sketch that fits the compute budget.
Deduplicate — a semantic fingerprint checks the hypothesis against a typed evidence store. Exact matches are rejected deterministically; only genuinely ambiguous near-duplicates escalate to a lightweight model call.
Write — a code-gen layer produces a unified diff against baseline.py or data.py, but never both. The original starter kit remains untouched.
Gate — a static, deterministic, bias-to-block scan of the added lines only. It blocks test-split references, non-causal statistic columns without an explicit point_in_time=True marker, external weights/downloads, and same-row auxiliary signals (is_like, is_follow, …) used as input features instead of loss targets.
Execute — the diff is staged into a per-candidate temp directory and run as a subprocess with a hard wall-clock cap, with every test-named file physically omitted from the sandbox and the split passed only via an environment variable.
Repair — if a candidate crashes, a debugging persona gets the traceback and up to 2 retries, and classifies whether the repair was semantic or cosmetic. A result above the 0.8645 oracle ceiling, or a >0.02 single-edit jump, is routed to a skeptical reviewer persona as a suspected leak.
Judge — this is the part we care about. A candidate is compared to its parent by a paired per-user bootstrap (500 resamples over thousands of users, matched on user id and seed), not by two noisy scalars. Only a candidate that is significant against its parent and clears the reigning champion is allowed to spend one query against the sealed valid_confirm split, paired against that split's own baseline.
Remember — every outcome is filed by type: invariant, refuted_under_context, failed_implementation, timeout, or no_op. We also pre-seeded the memory with dead ends we had already tested, so the agent never spends an iteration rediscovering them.

Selection is UCB1 over typed nodes with a wall-clock cost penalty, with a multi-fidelity triage tier (1 seed, 240 s cap) in front of the full 3-seed runs.

What actually happened

From a cold start, with the loss function nowhere in its instructions, the agent independently converged on the starter kit's own untested direction #1: the objective is pointwise log-loss, but both scored metrics are pure within-user ranking metrics. Its diagnostician committed to component = loss_function at confidence 0.90 in iteration 1 , and never left it for all 11 iterations.

It then found four successive new bests on valid_search, each one a distinct mechanism on that axis:

Iter	Mechanism	primary	Δ vs baseline	Δ vs prev best
0	untouched FM baseline	0.5946	—	—
1	within-user BPR, one uniform negative per positive	0.5951	+0.0005	+0.0005
2	negatives drawn from the user's own logged impressions, so every gradient contrasts two co-observed items	0.5955	+0.0009	+0.0004
6	hybrid loss: BPR pair term + residual pointwise term as a calibration regularizer	0.5958	+0.0012	+0.0003
7	2×2 grid search over (lr × 0.5, 1.0) × (1, 2 negatives), best-of-4 by validation	0.5961	+0.0015	+0.0003

Read that last column: +0.0005, +0.0004, +0.0003, +0.0003. Textbook diminishing returns. And note what the fourth "new best" actually was not a new mechanism, but a hyperparameter grid search over the third one. The agent had exhausted its ideas within the loss family and started tuning. We think that transition is one of the most interesting things in the run, and we're reporting it rather than dressing it up.

Two candidates cleared local significance and were allowed to spend a sealed-split query:

Candidate	Paired mean Δ	p(Δ>0)	lower 95%	Sealed-split verdict
`98fedeee` (iter 2)	+0.0016	0.956	+0.0001	failed the `lower_95 > 0` gate
`bbe18131` (iter 6)	+0.0019	0.972	+0.0006	failed the `lower_95 > 0` gate

Nothing was promoted. The run stopped with global_best still equal to the baseline, and print_final_summary said so in as many words: ** STILL THE BASELINE — nothing was promoted **.

Then we spent the single hidden-test measurement, after the run had terminated:

	baseline	champion `3769a939`	Δ
primary	0.5946	0.5953	+0.0007

The +0.0015 on valid_search became +0.0007 on held-out test, which is below the baseline's own 0.0008 seed standard deviation. The correct reading is "at baseline within noise," not "a confirmed improvement." Against the oracle ceiling our candidate captures 30.9% of the attainable range where the baseline captures 30.7%.

Eleven iterations, 5.86 hours, zero GPU, ~35 candidates attempted, zero test-set accesses during the search , and an honest negative at the end.

We think that's the result, not the absence of one.

How we built it

Our team of four built approximately 9,400 lines of code across four packages, supported by 230 tests. We froze the interfaces between the packages on the first day, allowing everyone to work in parallel without blocking one another.

harness/ — make the data impossible to use wrongly. Wraps the organizer's data.py / evaluate.py / baseline.py without editing a line. It splits the official 7-day valid window into a freely-usable valid_search (5 days, 96,609 rows) and a sealed valid_confirm (2 days, 28,300 rows). The split is contiguous, no gap, no overlap, proved by a test that compares rows as whole tuples including date, because (user_id, video_id) is deliberately non-unique here (3.06% of test pairs repeat, up to 12×), so a pair-level comparison would report false leaks. validated_evaluate raises ValueError instead of returning a wrong number on unequal lengths, NaN/Inf scores, non-binary labels, 2-D input, or a row count that doesn't match the named split. check_provenance rejects all 50 post-hoc engagement-aggregate columns as label leaks.

The test split is one-shot per process. We were honest with ourselves that this is not enforceable: the gate is module state, and codegen.execute runs every candidate as a subprocess by design, so a child can pull test after the parent already has and no library can stop it from the inside. What it can do is make it undeniable: every sealed access appends a JSON line (timestamp, pid, calling stack) to an append-only audit log, so the real end-of-run check isn't "did the gate raise?" but assert harness.count_sealed_accesses('test') == 1.

llm_calls/ — five personas behind five typed functions. Diagnostician, Literature, Hypothesis, Refiner, Auditor, plus a cheap Dedup escalation. Every call is schema-validated with retry; a model that can't produce a valid array of exactly the requested length raises rather than returning something plausible. All prompts live in one personas.py — the single tuning surface — and each is deliberately self-sufficient, repeating the dataset context, the oracle ceiling, the measured dead ends, and above all the implementation budget: numpy and lightgbm, nothing else; one file; one CPU core; 240 seconds.

codegen/ — write, gate, sandbox, repair. Diffs only, staged into throwaway directories, applied via patch -p1 → patch -p0 → git apply → git apply -p0, because models emit headers with and without the a/ prefix and the wrong strip level makes a perfectly good diff look broken. stdin=DEVNULL is load-bearing: without it, patch prompts File to patch: and hangs an unattended run forever.

orchestrator/ — the loop. Node tree, UCB1 selection, triage ranking, paired bootstrap promotion, plateau convergence, typed memory, and two live artifacts: progress.json rewritten atomically each iteration (so you can tail a 6-hour run mid-flight) and nodes.jsonl appended one self-contained JSON object per candidate, carrying the verbatim diagnosis and hypothesis, which serves as the record a judge reads to assess whether the autonomy is real.

We also implemented a second phase — MLE-STAR-style ablation-guided refinement, which ablates registered components (features, regularization, capacity), picks the weakest by measured delta, and refines it in a component-scoped edit. The search measurably regressed with it in the loop: our ablation set is too narrow to reliably identify the weakest component of a 5-field FM, and its refinements tended toward no-ops. It ships behind REFINE_ENABLED = False with its registry, harness, prompt and tests fully intact, and the comment above the flag says exactly that. Shipping a negative result in the on position would have been a nicer demo and a worse project.

Challenges we ran into

Almost every real challenge was the same shape: a measurement that looked fine and was structurally rigged. We found them one at a time, and each fix is documented in the code where the bug lived.

The tree could not grow, and nothing said so. The root node was seeded with a synthetic {u0..u9: 0.5946}. The paired bootstrap matches on user id, so the intersection with a real candidate's 10,894 users was empty, every delta list came back [], p_positive was always 0.0, should_continue_locally was always False, and no candidate ever entered open_nodes. The agent ran, spent money, printed scores, and was mathematically incapable of building anything on top of anything. Fix: measure the real baseline on 3 seeds, cache it, carry real per-user vectors.

The promotion test compared two different splits. Promotion checked a candidate's valid_confirm primary against a global_best carried on valid_search. That "delta" is a real effect plus the level difference between two different date ranges, added together. Fix: a separately measured, separately cached valid_confirm baseline, paired on the same split.

The promotion trigger was set four times higher than anything the search could produce. local_best_score > global_best + 0.003, against a search whose largest observed delta is +0.0019. Not one confirm query was ever eligible to be spent. Fix: trigger on the paired bootstrap (p ≥ 0.9, lower-95 > 0) plus a scalar champion check, which is how both 98fedeee and bbe18131 earned their sealed-split runs.

The stop rule demanded a brand-new gain every three iterations, forever. iter_history is a monotone running best, so max(h[-N:]) - max(h[:-N]) collapses to h[-1] - h[-4]. A gain can never count twice since three iterations later it is the h[-4] being subtracted. An early run cleared its iteration-3 check by 8.8e-6 and then stopped with the window delta at exactly 0.0, having used 28 minutes of a 6-hour budget. Fix: ε = 0.0005 (below the 0.0008 seed noise floor, i.e., "no gain larger than measurement error") and N = 8.

The root got a free head start in every comparison. The root runs 3 seeds; most candidates finish only the 1-seed triage. Comparing maxima hands whichever side ran more seeds an E[max of n] bonus which is measured on our own cached root, max 0.594966 vs mean 0.594640, a bias of +0.00033, roughly a fifth of the champion's entire reported gain, paid by the candidate every time. Fix: rank on the mean, which is unbiased in seed count.

Candidates that were right were being recorded as candidates that were broken. The triage cap was 120 s against an 18 s valid_search baseline, which is only 6.7×. Every pairwise and listwise loss runs a second backward pass, so correct implementations of the single most promising direction on the board were timing out and being filed as failed_implementation, throwing the hypothesis away with the implementation. Fix: 240s (13×), and timeout became its own evidence type, because "the writer can't build this" and "this is too slow to measure" call for opposite responses.

A candidate can pass every check and change nothing. A new helper function that nothing calls applies cleanly, passes the gate, runs to completion, and reproduces its parent's per-user scores bit for bit. Silent "no improvement", while the memory store files the mechanism as refuted when it was never tried. One candidate in the submitted run did exactly this. Fix: compare per-user vectors at 1e-9, name it no_op, feed that fact back and rewrite once.

Then the run itself found two failures we had not anticipated:

A reasoning model at temperature 0 makes schema-retry useless. Iteration 3's hypothesis call exceeded max_output_tokens = 2000 mid-write and returned a truncated response. The retry logic assumes sampling variance, but at temperature 0 the model produced byte-identical truncated output on all three attempts, and the run died on LLMSchemaError. This is our one manual intervention: we raised the token ceiling to 6000, raised MAX_FIX_ATTEMPTS, and resumed from the cached checkpoint. The agent could not have made that configuration change itself, so we count and report it as human input. Retry budgets are worthless against deterministic failure.

Our own logging ate the champion's record. During the resumed session _append_nodes_log wrote literal \n character pairs instead of newlines, concatenating ~35 records into one malformed line. Post-run recovery via json.JSONDecoder.raw_decode salvaged 18 records; 17 were unrecoverable, the parser skipped to the next {"iter": marker on each corruption. Among the losses was 3769a939, the highest-scoring candidate in the run: we still have its diff and its score from terminal output, but its hypothesis text and diagnosis are gone, so the mechanism in the table above is reconstructed from the diff rather than quoted from the agent. We built an append-only log specifically so a crash could not corrupt prior records, and then corrupted them with a formatting bug in the append itself. It is the most avoidable thing that went wrong.

The LLM auditor hallucinated leaks from thin context. We kept it, as its concerns are recorded verbatim into each node's diagnosis, and one of them (negatives sampled without timestamp filtering, so a positive may be contrasted against an impression that had not happened yet) is a genuinely sharp catch, but it is advisory only. The deterministic static gate is the gatekeeper. A blind reviewer with veto power over a search that only produces third-decimal gains simply stops the search.

Accomplishments that we're proud of

The agent found the right idea by itself, and then worked it out. The starter kit names changing the loss function as untested direction #1 and explains why. That text is not in the diagnostician's prompt. The agent reached it from the trajectory alone — "objective mismatch: pointwise loss on a within-user ranking metric", confidence 0.90 — and then produced four genuinely distinct mechanisms on that axis across 11 iterations, each an incremental new best over the last.

It reported the honest negative, in bold, on its own. print_final_summary exists for exactly one reason: global_best only advances on a confirmed promotion, so a run that found nothing prints a "best" numerically identical to the baseline — and read quickly, 0.5946 looks like a result rather than the absence of one. The function says ** STILL THE BASELINE — nothing was promoted **, and separately reports the best unconfirmed valid_search score, because the gap between those two numbers is precisely "we found something and it didn't replicate."

The sealed split earned its keep. Two candidates cleared local significance with p = 0.956 and p = 0.972 — numbers most projects would ship. Both failed the sealed gate, and the hidden test later agreed with the sealed split, not with valid_search: +0.0007 against a 0.0008 noise floor. The guardrail didn't just exist, it fired, and it was right.

Zero test accesses during the search, provable rather than claimed. Backed by an append-only audit log recording pid and call stack, plus a sandbox that physically omits test-named files. The single hidden-test measurement was taken after plateau termination and is reported as such.

Every constant is justified by a measurement, in a comment next to it. ε = 0.0005 sits below the 0.0008 seed std. The 240 s cap is 13× the measured 18 s valid_search baseline. The +0.0005 promotion margin is set against the largest delta the search has ever produced. 230 tests, 111 running in 17 seconds as the default gate, with the expensive split-integrity and baseline-reproduction checks behind -m slow.

We shipped a phase in the off position because the evidence said off, and a negative result we can defend beats a positive one we can't.

What we learned

On a low-signal benchmark, the statistics are the agent. Almost none of our engineering time went into prompts; it went into "is this number real?" A paired per-user bootstrap resolves a +0.0016 that a two-scalar comparison buries entirely — and comparing maxima across unequal seed counts is systematically biased toward whichever side ran more seeds. Without the pairing we would have shipped a leaderboard of noise and believed it.

valid_search gains at p = 0.97 still didn't generalize. This is the finding we did not expect and would most want another team to take from us. A paired bootstrap over thousands of users, significant at the 95% lower bound, on a 5-day held-out window, twice — and the sealed 2-day window disagreed both times, and the hidden test agreed with the sealed window. Local significance is not generalization when the effect size is smaller than the seed noise.

Retry budgets are worthless against deterministic failure. Temperature 0 plus truncation equals three identical failures and a dead run. Any agent that retries should either perturb its input or fix the actual cause.

Half of all failures were budget failures, not reasoning failures. Early candidates proposed SAM, ASAM, NAS, attention layers, "using known deep learning libraries." The fix was not a smarter model; it was writing numpy and lightgbm. NOTHING ELSE. One CPU core. 18 seconds baseline, killed at 240 into the persona in capital letters, with the mechanism-level reasons the dead ends are dead. Constraint text was the highest-leverage prompt engineering we did.

Failure taxonomy is a feature. failed_implementation, timeout, no_op and refuted_under_context are four completely different signals wearing the same "didn't improve" costume, and they call for four different responses: rewrite the code, make the same mechanism cheaper, feed back that nothing changed, or abandon the idea. Collapsing them silently threw away our best hypothesis.

A single-bottleneck commit is a double-edged design. Forcing the diagnostician to name exactly one component gave us a clean, auditable search — and it never left loss_function for 11 straight iterations, because the running best kept inching up. The honest post-run read is that the loss family was not exhausted, it had reached a local optimum with sub-ε gains, and by iteration 7 the agent was tuning hyperparameters rather than proposing mechanisms. An agent that can say "this family is done, look elsewhere" is a different and harder design than one that follows the gradient of its own running best.

What's next for swe hell

Make the log durable, and make resume real. A formatting bug cost us 17 records including the champion's reasoning. Write-and-fsync per record, a schema check on append, and a proper --resume flag are the first commits — the recovery path we used was raw_decode in a shell, not a feature.
Break the single-bottleneck loop. Add an explicit exhausted_families signal with teeth: when a component's last k proposals all land within seed noise, forbid it and force the diagnostician elsewhere. The one component the agent never touched is the one the starter kit rates second: user behaviour sequences, completely unused today, with hundreds to thousands of interactions per user available.
Act on the auditor's real catch. It flagged that negatives are sampled from all same-user rows without timestamp filtering, so a positive may be contrasted against an impression that had not yet occurred. Point-in-time negative pools are a correctness fix and a plausible source of genuine gain.
Unbiased validation. log_random_4_22_to_5_08_pure.csv is 1.18M rows of random exposure. Wiring it in as a second, unbiased confirmation tier would let the agent distinguish "learned the ranking" from "learned the logging policy" — directly relevant, since our gains failed to generalize.
Re-earn Phase 2. Ablation-guided refinement is written, tested, and off. The ablation set (features, regularization, capacity) is too coarse for a 5-field FM; a set that includes the loss and the sampler is the experiment that would let us turn it back on with evidence.
Measure the agent, not just the model. One run is an anecdote. Repeated runs across seeds and sampling settings, reporting mean, variance and success rate, is what would turn "our agent found +0.0015" into a claim about the agent.
Then scale. KuaiRand-1k and 27k are untouched: a single 27k baseline exceeds the 6-hour ceiling before any search can begin, so cheaper per-iteration evaluation (warm-started candidates, shared encode passes, reused parent per-user vectors) is the prerequisite, not a nice-to-have.