🍳 Let Him Cook

Your own personal ML engineer

Inspiration

What does a machine learning engineer actually do when given an unfamiliar dataset?

They don't immediately start tuning a learning rate.

They first understand the data. They draw on what they already know, read papers and documentation, identify possible weaknesses in the current approach, and form hypotheses about what could improve it. They implement an experiment, evaluate the result, learn from what happened, and decide what to investigate next.

Then they do it again.

And again.

Until they find something better or their tired or their boss tell them to stops, or for my AI engineer, when it runs out of credits.

Most AutoML systems automate pieces of this process, such as model selection or hyperparameter optimization. For Let Him Cook, I wanted to explore a different question:

Can we automate the research process itself?

The challenge asks for an Autonomous ML Research Agent capable of reproducing a baseline, autonomously modifying the ML pipeline, learning from validation results, recovering from failures, and ultimately improving upon that baseline with minimal human intervention.

So instead of building an agent that asks "Which learning rate should I try next?", I built one that can ask:

"What have I learned so far, what hypothesis should I test next, and is that experiment worth the resources it will consume?"

🧑‍🍳 What is Let Him Cook?

Let Him Cook is an autonomous ML research system that reads, hypothesizes, experiments, remembers, and iterates.

The system maps the workflow of a human ML researcher into an agentic loop:

Research
   ↓
Understand the Dataset
   ↓
Generate Hypotheses
   ↓
Plan Experiments
   ↓
Select Research Direction
   ↓
Write + Execute Code
   ↓
Evaluate Results
   ↓
Reflect + Remember
   ↓
Repeat

For the hackathon, we used KuaiRand-Pure, a short-video recommendation dataset containing roughly 1.4 million interactions. The task is to rank each user's logged impressions according to long_view, evaluated using GAUC and nDCG@5.

The provided baseline is a Factorization Machine with five categorical fields. Its published validation primary score is approximately 0.59, from my own recreation.

The agent's job was simple:

Beat it.

🧠 How It Works

1. Research before experimentation

A human ML engineer enters a project with knowledge accumulated from courses, papers, documentation and previous experiments. I wanted the agent to have something similar.

Instead of simply dumping entire research papers into an LLM context, Let Him Cook builds an actionable research knowledge base inspired by ideas from autonomous ML and scientific-agent frameworks.

Research is transformed into structured information describing:

What an ML technique does
Why it might improve a model
When it should be considered
How it could be translated into an experiment

When its existing knowledge is insufficient, the research layer can also retrieve additional external information.

This lets papers become experimental ideas rather than passive text.

2. From knowledge to hypotheses

The research agent combines this knowledge with information about the current dataset, model and previous experiments to generate hypotheses.

For example:

Observation:
The Factorization Machine treats interactions between feature
fields uniformly.

Hypothesis:
Different feature interactions may contribute unequally to
ranking quality.

Experiment:
Introduce learnable interaction gates.

Crucially, the system does not generate one idea, test it, forget about everything else, and repeat.

It maintains multiple competing research directions.

An idea that initially performs poorly does not necessarily disappear forever, while an early promising result does not force every subsequent experiment down the same path.

That creates a tree of research:

                         Baseline
                       /    |     \
                  Features Model   Training
                     /       \
                 Gating      Other architecture
                /     \
           Context    Field-level
             |
        Shared groups

The problem then becomes deciding where to spend the next experiment.

3. Resource-Aware Monte Carlo Tree Search

Experiments aren't free.

A brute-force researcher could test every possible idea, but an autonomous system needs to reason about the value of an experiment relative to its cost.

I therefore use a resource-aware Monte Carlo Tree Search policy that conceptually balances:

Score(i) = Δ̂ᵢ + c√(ln(N) / nᵢ) − λCᵢ

Δ̂ᵢ = expected improvement c√(ln(N) / nᵢ) = exploration λCᵢ = experimental cost

The first term rewards research directions that have produced promising improvements.

The second rewards exploration. A branch that has barely been investigated receives an exploration bonus, preventing the system from prematurely committing to the first successful idea.

The final term penalizes expensive experiments.

This changes the question from:

"Which experiment has the highest expected score?"

to:

"Which experiment gives me the most valuable next piece of evidence given what I know and the resources I have left?"

That matters because the challenge explicitly evaluates both autonomous improvement and the resources required to reach the converged result.

4. Let the coding agent cook

Once the research system chooses an experiment, it hands the experimental plan to a coding agent.

The coding agent modifies the existing ML pipeline, executes the experiment and returns:

Hypothesis
     ↓
Code Modification
     ↓
Training
     ↓
Validation
     ↓
Metrics / Errors

The researcher therefore decides what to investigate, while the coding agent handles how to implement it.

Failed code is also part of the research process rather than automatically ending a run. Errors can be captured and fed back into subsequent decisions so the system can recover or change direction.

This is particularly important for long-running autonomous experimentation: the challenge explicitly values recovery from errors, timeouts and unexpected inputs rather than expecting an agent to magically never fail.

5. Experiments become memories

A human researcher doesn't just write:

Experiment 17 = 0.6032

They remember why they tried it and what they learned.

Let Him Cook therefore maintains experimental memory.

After an experiment, the agent records:

WHAT DID I TRY?
        ↓
WHAT HAPPENED?
        ↓
WHY MIGHT IT HAVE HAPPENED?
        ↓
WHAT SHOULD I BUILD ON OR AVOID?

For example, an experiment might produce a lesson such as:

Observation: Context-conditioned interaction gating improved over static gating. Lesson: A single global weighting of feature interactions may be overly restrictive. Preserve interaction gating while investigating context-specific parameter sharing.

This memory is available to other branches of the research process.

A failed experiment therefore isn't necessarily wasted compute. It can tell the agent what not to do again.

The result is:

$$ \text{Experiment} \rightarrow \text{Evidence} \rightarrow \text{Reflection} \rightarrow \text{Memory} \rightarrow \text{Better Next Experiment} $$

🔬 What Did the Agent Actually Research?

The most interesting result wasn't simply the final metric. It was the research trajectory that emerged autonomously.

The agent began from the Factorization Machine baseline and investigated whether all feature interactions deserved equal treatment.

Stage 1 — Selective interactions

The first promising hypothesis was that different feature interactions should have different importance.

The agent experimented with interaction gating.

The positive result gave the search evidence that this family was worth investigating further.

Stage 2 — Context matters

Rather than simply repeating the same experiment with different hyperparameters, subsequent hypotheses asked a deeper question:

Should the importance of an interaction depend on context?

The agent explored tab-conditioned interaction gates, finding evidence that static global interaction weights were too restrictive.

Stage 3 — Share or specialize?

This led naturally to another question:

Should each context receive completely independent parameters, or should related contexts share information?

Experiments with shared tab groups and context-specific deviations explored the trade-off between shared representations and specialization.

Stage 4 — Find the right granularity

The agent then investigated where gating should occur.

Aggressive pair-level interaction reweighting was not consistently beneficial, while per-field gating proved more reliable.

Negative results were useful here: greater granularity did not automatically mean a better model.

Stage 5 — Refine the winning mechanism

Finally, the system explored the range over which successful field gates should operate.

Wider but bounded per-field gating produced the strongest validation result.

The agent subsequently challenged this incumbent with alternatives including additional pair reweighting, duration-conditioned mechanisms, calibration and loss modifications. None established a stronger direction.

The research process had effectively moved from:

$$ \text{Do interactions matter?} $$

to:

$$ \text{Does context matter?} $$

to:

$$ \text{How should contexts share information?} $$

to:

$$ \text{At what granularity should interactions be controlled?} $$

That progression is what I wanted Let Him Cook to demonstrate: not merely automated tuning, but iterative hypothesis refinement from experimental evidence.

📈 Results

The autonomous campaign completed:

25 iterations 25 real training runs

The reproduced validation baseline was:

$$ \text{Primary}_{baseline}=0.59 $$

and the best model discovered by the agent achieved:

$$ \boxed{\text{Primary}_{best}=0.60422} $$

The complete run consumed approximately only 2 dollars of api credits because codex subscription handled most of the coding cost, only the research agent itself needed to consume credits.

The challenge requires each iteration to retain its hypothesis, code changes, metrics, and error/recovery information specifically so that the autonomous research trajectory can be inspected rather than judging only the final number.

🛠️ Tech Stack

The project combines several components rather than relying on a single monolithic agent:

LLM Research Agents — hypothesis generation, reflection and research planning Codex — autonomous experiment implementation and code modification Monte Carlo Tree Search — research-direction selection and exploration Experimental Memory — cross-branch lessons and research history Research Knowledge Base — converting published ML research into actionable experimental knowledge Python / NumPy — recommendation pipeline and experimentation KuaiRand-Pure — required recommendation benchmark

The architecture deliberately separates reasoning, search, coding, evaluation and memory, allowing each component to evolve independently.

🚧 Challenges

Autonomy vs. blindly experimenting

The hardest problem was not getting an LLM to suggest experiments.

It was getting it to decide which experiment was worth running next.

LLMs can generate dozens of plausible ML ideas, but compute and API calls are limited. Without a search strategy, an autonomous agent can easily spend its entire budget pursuing one attractive direction or repeatedly rediscovering similar ideas.

The resource-aware tree search and experimental memory were introduced specifically to address this.

Turning failures into information

Autonomous coding inevitably produces failed experiments.

Instead of designing the system around the assumption that every generated experiment succeeds, I treated failures as part of the research trajectory. The challenge became distinguishing:

a bad hypothesis,
a bad implementation, and
an experiment that simply needs refinement.

That distinction is essential if an autonomous researcher is expected to operate for dozens of iterations without human supervision.

Context and token consumption

Research agents accumulate context extremely quickly.

Papers, experiment history, source code, metrics and reflections all compete for the same context window and API budget. My 36-iteration run alone consumed roughly 3.56M tokens.

This made memory compression and selective context an important engineering problem rather than just an optimization.

🧬 What I Learned

The project changed how I think about AutoML.

Initially, I thought the difficult part would be generating good ML ideas.

It wasn't.

The harder problem was deciding:

What should I try next given everything I have already learned?

An autonomous researcher needs more than an LLM and a coding tool. It needs a mechanism for exploration, evidence accumulation, resource allocation, failure recovery and memory.

I also learned that failed experiments are surprisingly valuable. Several of the most useful decisions in the run came not from discovering something that worked, but from discovering that an apparently reasonable direction didn't.

That is precisely what separates research from simple optimization.

🚀 What's Next?

Two days was enough to make the agent cook.

It was not enough to build the whole kitchen.

More efficient coding

Currently, coding can consume substantial context even when an experiment requires only a localized modification.

Future versions could use patch-based coding approaches inspired by systems such as SWE-agent and AIDE, presenting the coding agent primarily with the relevant portions of the pipeline rather than repeatedly reasoning over unnecessary context.

Tool evolution

Today, the agent primarily evolves through its accumulated experimental knowledge.

A future system could also create reusable ML tools.

If several experiments discover that a particular interaction-gating mechanism is useful, for example, the system could turn that implementation into a reusable tool available to future research tasks.

Prompt and policy evolution

The research agents themselves could improve.

Successful and unsuccessful research trajectories could be used to optimize how agents generate hypotheses, diagnose bottlenecks and allocate experimental resources.

Eventually, even parameters governing exploration and resource sensitivity could adapt from previous AutoML trajectories instead of being manually configured.

Cross-task learning

The longer-term goal is for experimental lessons to transfer between completely different ML problems.

Instead of:

$$ \text{New Dataset}\rightarrow\text{Start From Zero} $$

we want:

$$ \text{Past Research} \rightarrow \text{Reusable ML Knowledge} \rightarrow \text{Better Initial Hypotheses} \rightarrow \text{Faster Discovery} $$

A better knowledge base

Finally, research retrieval itself can improve.

A curated knowledge base organized around dataset characteristics, model families, failure modes and interventions could give the research agent stronger priors before experimentation even begins.