Team Logo
Overview
Iteration Loop
Trust Boundary
Agent Behavior
Ensemble

Halcyon

Inspiration

Two numbers from the KuaiRand-Pure benchmark stopped us cold.

The oracle ceiling is 0.8484, not 1.0 — more than a third of users are all-positive or all-negative, so no model on earth changes their nDCG. And the validation noise floor is σ ≈ 0.0009. Every real gain in this benchmark lives in the third decimal place.

Then we found a third number, from METR: explicit reward hacking in 39 of 128 agent runs on RE-Bench, with nobody prompting the agents to cheat.

Put those together and the interesting problem stops being "can an LLM improve a recommender." It becomes: can an agent tell its own signal from its own noise — and will it stay honest when cheating is one np.load away? An agent that fails the first confidently chases its own variance. An agent that fails the second wins the leaderboard and learns nothing.

We built Halcyon to be a researcher, not a leaderboard fitter.

What it does

Halcyon runs the entire ML research loop with zero human interventions:

OBSERVE → HYPOTHESIZE → PLAN → CODE → GUARD → TRAIN → EVALUATE
   → COMPARE → REFLECT → DECIDE → next experiment → ENSEMBLE → FINALIZE

It reproduces the official baseline, states the bottleneck it sees before it proposes anything, writes real model code (not just hyperparameters), has that change validated before a single GPU cycle is spent, trains, compares against a control with a paired user-level bootstrap, decides what to attack next, assembles a complementary model portfolio, writes a validated submission — and stops when the organizer's own convergence rule says to.

Our live Gemini run:


FM baseline (reproduced exactly)	0.60147
Portfolio, honest 5-fold CV	0.60409 ± 0.00141 (+0.00259)
Experiments / wall-clock / tokens	4 of a permitted 50 · 450 s · 12,593
Manual interventions	0

We quote the cross-validated number. The tuned one was 0.60463, and Halcyon labels it in-sample itself.

How we built it

A frozen trust boundary. Five files — the data loader, evaluator, submitter, node runner, and contracts — are SHA-256-pinned and verified at the start of every run. The agent can never modify the code its score depends on, accidentally or via a hallucinated "fix." Every capability we added routes around that boundary, never through it.

A six-block solution space. A model is six Python files (build_features, build_model, build_loss, train, infer, combine). Halcyon rewrites one block body per experiment, or adopts a whole model family — FM, DeepFM+DIN, LightGBM LambdaRank. Every edit is snapshotted and hashed.

LLM roles as operators over a deterministic policy. A Proposer (what problem to attack), a Coder (how to implement it in one block), a Reflector (why it broke and how to recover) — all Pydantic-schema-constrained, sitting on top of a search policy we control. Best-first with an ε-exploration valve, deliberately not MCTS: each node is a real training run and a compliant run ends after ~4–6 experiments, far too few for rollouts to produce meaningful value estimates.

Statistics as the decision rule. Every node gets a paired user-level bootstrap against its control; P(Δ>0) drives the verdict, not a raw delta. Portfolio value (rank correlation, leave-one-out marginal contribution) is computed from saved predictions — seconds, no retraining.

Challenges we ran into

Every one of our three worst bugs was in something we had already written down as handled.

The lever that trained the wrong thing. A hypothesis of "use BPR loss" was training plain BCE and returning a byte-identical baseline result — in 4 of 21 recorded runs. build_loss hardcoded the loss, so the knob was decorative. The agent dutifully logged "BPR → 0.6015, rejected", and its own memory then told it never to retry its single strongest lever. Fixing it recovered +0.00214. That bug became a feature: Halcyon now declares, per block set, which config fields provably reach an execution path, and rejects an ineffective knob before training.

The hole in "physically impossible." Our docs claimed label access was physically impossible. It wasn't. Hidden test labels and an auxiliary is_click column were sitting in the directory handed to every block — and ranking the validation set by is_click alone scores 0.7466, which is 58.8% of the entire headroom above baseline, with no training at all. We closed it with five layers and rewrote the claim to what is actually true.

A guarantee that wasn't. We stated temporal safety was "structural — we process rows in global time order." The loader didn't sort, and KuaiRand logs aren't time-ordered within a user: 30.8% of train rows had a future item in their history. Bounded damage with ID-only histories, but a false guarantee is still false. Now verified at 0 violations against an independent rebuild.

Accomplishments that we're proud of

The result we're proudest of is one we threw away. Halcyon's behavior-aware-history experiment scored +0.0165 — eighteen times the noise floor. Every other lever in this benchmark moves ~0.002, so we treated the magnitude as a symptom rather than a win. The diagnosis: feedback coverage was 100% on validation and 75.8% on test. A validation row could see the outcomes of earlier validation rows; a test row never could, because a submission scores all rows at once. The gain was structurally available on the set used to choose models and structurally unavailable on the set that would be scored. Rebuilt honestly, the effect was −0.00167. The feature ships disabled. Cost of finding out: four minutes of GPU time.

A weaker model beat a stronger one. LightGBM scored below the champion standalone and was recorded as rejected — yet its rank correlation of 0.860 made it the most decorrelated member, worth +0.00050 to the portfolio. The auxiliary-head node, with a better standalone score, contributed exactly +0.00000. Selecting on score alone would have kept the wrong model.

It refused to fabricate. Asked for a position-bias tower, it replied that position features don't exist in this dataset. It was factually right — there is no slot column.

What we learned

Intended intervention ≠ executed intervention. The most expensive bug in the project wasn't wrong code; it was code that ran fine and silently ignored the thing being tested. Validating that an experiment actually happened is now a recorded property of every node.
Evidence and bookkeeping are different questions. A node can be the champion and still inconclusive; standalone-rejected and still a portfolio asset. Conflating them once made our agent's memory label its own best model "REJECTED — don't repeat."
Statistical power, not search cleverness, was the binding constraint. With SE ≈ 0.0009, no search algorithm steers on a +0.0004 effect. Widening the evaluation surface dominates changing the policy.
Your documentation needs tests too. Two of our three worst bugs were found by trying to prove a claim we'd already written in confident prose.
The strongest thing an agent can do is disbelieve itself.

What's next for Halcyon

Widen the evaluation surface — power is the binding constraint on nearly every open question we have. KuaiRand's randomized-exposure split is the obvious candidate.
Recency-conditioned feedback states — the one rejected mechanism whose failure mode we fully understand. Masking during training recovered 58% of the loss, which confirms the mechanism even though the lever stayed negative.
Make the cross-run ledger pay off. Its compatibility guard currently refuses to pool all 14 historical entries across cache and code changes. Correct, and completely unrewarding.
True propensity / SNIPS correction, and a native LambdaRank loss to replace the LightGBM dependency.
Measure valid→test shift. Every number we report is validation-side by construction. That's the honest limitation we'd most like to remove.