Inspiration
Every ML project I've worked on has had more promising directions than I had time for — a feature idea, an architecture swap, a paper worth trying — that just sat on a list because attention is finite. This track was a chance to build (or at least learn how to build) an agent that explores those branches for me: not a chatbot that writes code on request, but something that proposes its own hypotheses, tests them, and decides for itself what to try next.
What it does
kujira-ai runs the full ML engineering loop — EDA, feature engineering, model training, evaluation, reflection — as a chain of autonomous iterations on the KuaiRand-Pure recommendation benchmark. Each iteration proposes a hypothesis, implements it, runs it, and either accepts or rejects the result against the current best, then feeds that outcome into the next proposal. A crash doesn't end the run: the traceback goes back to the agent as repair context, it patches its own code, and the iteration re-runs. Every decision — the hypothesis, the category of change it belongs to, before/after metrics, accept/reject — is logged per iteration, so progress is fully auditable rather than a black box.
How we built it
The core is an orchestrator loop (Python) driving Claude as the proposing/repair LLM, with a few deliberate constraints on top of "let the model do whatever":
- Grounded proposals. A deterministic, no-LLM EDA pass runs once up front over every raw column (not just the fields the starter kit encodes) and gets summarized into a report every proposal call reads — so the agent reasons from this dataset's actual quirks (a sentinel value disguised as real data, a heavily skewed popularity distribution) instead of generic priors about "recommendation datasets."
- Literature grounding. A local BM25 retriever over a small curated note corpus (FM, DeepFM, DCN, AutoInt, xDeepFM, ESMM, MMoE, popularity-debiasing methods) surfaces relevant papers based on the EDA findings themselves, so retrieval is steered by the data, not a generic keyword search.
- A restricted live-research role. A separate, sandboxed Claude call can search and fetch the web — but only that; no code execution — and its answers are discarded unless the cited source resolves to an allow-listed domain (arXiv, ACM/IEEE, OpenReview, PapersWithCode, GitHub). This is what found the single highest-leverage change in the whole run: switching the training loss from pointwise BCE to a BPR pairwise-ranking loss, because the evaluation metrics are inherently pairwise within a user's session.
- A lever-category taxonomy. Every hypothesis must declare which of 9 fixed categories it belongs to (feature engineering, model architecture, training procedure, negative sampling, popularity debiasing, multi-task, hyperparameter/capacity, ensembling, data-quality preprocessing). Once the agent plateaus, the next prompt names the specific untried categories and requires the next proposal come from that list.
- Diagnostic probes and mid-run agentic EDA. The agent can also ask a read-only numpy question about the data instead of proposing code — and, when it's truly stuck, request a budget-capped round of exploratory probing to look for a genuinely new signal, rather than guessing blind.
- Parallel hypothesis racing. A proposal can also come as 2–3 independent variants run concurrently, with only the winner carried forward into hyperparameter sweeps.
The whole trained pipeline (FM/DeepFM models, BPR training, evaluation) is plain numpy — CPU-only, no GPU needed, each candidate trains in well under a minute.
Challenges we ran into
- Plateauing. After the first real win, the agent spent many iterations proposing plausible-sounding tweaks that all landed within the metric's own sampling noise — none of them cleared the acceptance margin.
- Category tunnel vision. Early on, a softer "try something different" instruction didn't work: the agent kept proposing variations of the two categories it had already succeeded with once, and never touched the others even though the EDA and literature context mentioned them constantly. That's what forced the lever-category taxonomy into existence — a much harder instruction to quietly ignore than "be different."
- Running out of runway. Between LLM call cost, wall-clock time, and turn budgets on sub-tasks like mid-run EDA, the agent's own resources — not just its ideas — became the binding constraint before the search formally converged.
Accomplishments that we're proud of
As an experiment, I gave the same task — autonomously improve this benchmark — to plain Claude Code with no scaffolding, just the instructions and freedom to iterate. It didn't reach the score my orchestrated agent found. That's the result I'm most proud of: the scaffolding around the LLM — grounding it in real EDA findings, forcing category coverage, giving it a restricted-but-real way to pull in literature — measurably outperformed just pointing a frontier coding agent at the same problem and letting it go.
What we learned
Mostly how much the harness around an LLM determines whether it actually explores or just converges on the first plausible idea and polishes it. Concretely: grounding proposals in real data findings beats generic priors, a plateau needs a structural nudge (not a vaguer prompt) to break out of a rut, and giving the model narrow, verifiable tools (a restricted web role with a source allowlist, read-only diagnostic probes) is safer and more useful than giving it broad, unconstrained access.
What's next for kujira-ai
- A genuinely different model family. Everything tried so far is a numpy FM/DeepFM variant; a PyTorch model zoo (DCN, AutoInt, xDeepFM) is sitting unwired and untried.
- Pretrained models from the Hugging Face Hub as a candidate feature source or base model — the agent currently has no way to reach outside what it can hand-derive in plain numpy.
- Other KuaiRand splits (1K, 27K) to check the pipeline actually generalizes beyond KuaiRand-Pure, not just in principle.
- Real token-usage logging, not just cost-derived estimates, for a more precise resource accounting of each run.
