-
EurekaML autonomously diagnoses, selects hypotheses, runs experiments, evaluates results and learns what research direction to try next.
-
A cost-aware Multi-Armed Bandit uses Thompson Sampling to balance exploration and exploitation across competing ML hypotheses.
-
Research DNA preserves experimental knowledge while the lineage tree tracks every model, result and failure across the autonomous search.
-
EurekaML verifies gains across seeds, guards against leakage, and measures which model components genuinely contribute to improvement.
-
Every research decision is traceable, from hypotheses and experiments to gate rejections, failures and autonomous recovery.
Inspiration
Machine learning research is rarely a straight line. An ML engineer must understand the problem, inspect the data, reproduce a baseline, form hypotheses, implement experiments, evaluate results, analyse failures, and decide what to investigate next.
This inspired the question behind EurekaML:
What if we automated the ML research scientist, rather than just model training?
Instead of building a traditional AutoML system that searches through predefined models or hyperparameters, we designed EurekaML to behave like a researcher. It reasons about competing hypotheses, decides where to spend its research budget, learns from both successful and failed experiments, and can even create new modelling capabilities when its existing ones are insufficient.
What it does
EurekaML is an autonomous ML research agent for recommender systems that automates the complete machine learning research loop:
Understand → Diagnose → Hypothesize → Prioritize → Build → Train → Evaluate → Reflect → Learn → Repeat
For the KuaiRand-Pure benchmark, EurekaML begins by reproducing the official Factorization Machine baseline, giving the agent a trustworthy reference point before autonomous experimentation begins.
It then investigates improvements across feature engineering, behavioural history, ranking objectives, hard-negative sampling, multi-task learning, interaction modelling, hyperparameter optimisation and model architecture.
Instead of following a fixed list of experiments, EurekaML continuously uses evidence from previous experiments to determine what should be investigated next.
Every experiment — including an unsuccessful one — becomes research evidence.
How we built it
We designed EurekaML around several components inspired by how a research scientist works.
Research Portfolio — Reinforcement Learning-inspired research allocation
A researcher may have many hypotheses but limited time, compute and experimental budget.
We model this as a reinforcement learning-inspired Multi-Armed Bandit (MAB) problem, where each research direction represents an arm and every completed experiment provides new evidence about that direction.
EurekaML uses cost-aware Thompson Sampling, a Bayesian Multi-Armed Bandit strategy, to decide which hypothesis deserves the next research experiment.
This balances:
Exploration — investigating uncertain or less-tested hypotheses that could reveal a breakthrough.
Exploitation — continuing to investigate hypotheses that existing evidence suggests are promising.
The process creates a feedback loop:
Select hypothesis → Run experiment → Observe evidence → Update beliefs → Select next hypothesis
The strategy is also cost-aware, allowing EurekaML to consider the resources required to obtain that evidence instead of blindly running every possible experiment.
This gives EurekaML an adaptive research strategy rather than a predefined experiment checklist.
Research DNA — learning from research history
We did not want EurekaML to simply remember that "Experiment X failed."
Researchers remember the underlying mechanism they tested and use that knowledge when evaluating future ideas.
EurekaML's Research DNA therefore records mechanisms behind experiments — such as candidate conditioning, behavioural history, interaction modelling and ranking objectives.
This creates a growing scientific memory that helps EurekaML recognise related approaches and avoid repeatedly rediscovering the same unsuccessful mechanism under a different experiment name.
Dynamic Architecture Factory — expanding its own research space
Traditional AutoML systems are generally limited to models and search spaces predefined by their developers.
We wanted EurekaML's research space itself to evolve.
When EurekaML proposes a promising research direction that its existing capabilities cannot test, its Dynamic Architecture Factory can generate a model specification, construct an executable model, connect the required data and training pipeline, and introduce that capability into the research process.
EurekaML therefore does not only select between models we originally gave it — it can expand the set of experiments it is capable of performing.
Generated architectures must still pass the same scientific evaluation process as existing models before their results can influence future decisions.
Cheap falsification — fail cheaply before failing expensively
Not every interesting hypothesis deserves a full training run.
Before expensive training, generated models pass through representation sanity checks and inexpensive experimental probes.
If an idea cannot demonstrate basic viability, EurekaML can falsify it early rather than wasting the research budget on full training.
This led to one of our core design principles:
The objective is not to run the most experiments — it is to extract the most useful evidence from each research decision.
Generalization Guardian — making sure improvements are real
Machine learning metrics are noisy.
A model scoring slightly higher once does not necessarily mean EurekaML has discovered something better.
Our Generalization Guardian verifies promising candidates before they are promoted as trusted improvements. It also enforces integrity checks around the evaluator, training split and forbidden leakage patterns.
This prevents EurekaML from blindly chasing small validation fluctuations and helps distinguish genuine progress from statistical noise.
Failure recovery — scientific failure vs engineering failure
Another important design decision was teaching EurekaML that not every failure means the same thing.
If a model trains successfully but performs poorly, that is a scientific result. The hypothesis was tested and produced useful evidence.
A crash, invalid implementation, numerical issue or persistence error is an engineering failure.
EurekaML tracks these separately.
Scientific failures become evidence for future research decisions, while engineering failures can trigger bounded recovery mechanisms.
This allows the agent to learn from negative results rather than simply retrying until something scores higher.
Research Tree — preserving experimental lineage
Every experiment branches from a known parent rather than continuously mutating one model.
EurekaML maintains a Research Tree that records the lineage of models, interventions, scores, failures and promoted candidates.
Together with Research DNA, this provides an auditable history of how the agent's research evolved.
Plateau detection and research retrieval
Researchers also look outward when their existing ideas stop working.
When EurekaML detects that research has plateaued, it can trigger targeted literature and code retrieval based on the diagnosed bottleneck rather than simply searching for another model name.
Retrieved ideas must still enter EurekaML through its controlled experiment and evaluation pipeline before they can influence the research portfolio.
How our solution addresses the problem statement
The challenge asks for an autonomous ML research agent capable of reproducing the official baseline, iterating on the ML pipeline, improving over the baseline, recovering from failures and minimizing human intervention.
EurekaML addresses this by automating the complete ML research lifecycle, rather than only model training or hyperparameter tuning.
EurekaML can:
- reproduce the official benchmark baseline;
- inspect and diagnose the ML problem;
- generate falsifiable research hypotheses;
- maintain competing research directions;
- use cost-aware Thompson Sampling to allocate research effort;
- engineer features and ranking objectives;
- implement and generate model architectures;
- train and evaluate experiments;
- promote or reject hypotheses using experimental evidence;
- recover from engineering failures;
- preserve research knowledge through Research DNA;
- expand its own modelling capabilities;
- retrieve new research directions when progress plateaus; and
- use accumulated evidence to decide what to investigate next.
Throughout development, model selection is driven only by training and validation information. The hidden test set remains isolated from the autonomous research process.
Challenges we ran into
One of our hardest challenges was balancing autonomy with scientific integrity.
Giving an AI agent the ability to generate architectures and expand its own experimental space introduces difficult engineering problems. Failed generated models must not contaminate unrelated hypotheses, duplicate experiments should not waste research resources, and generated architectures must pass the same evaluation process as existing models.
Another challenge was research allocation.
Always selecting the currently best-performing idea would over-exploit known approaches, while random exploration would waste limited compute. Formulating research allocation as a Multi-Armed Bandit problem allowed us to use Thompson Sampling to continuously balance exploration and exploitation.
We also had to distinguish scientific failure from system failure. A poor-performing experiment is valuable evidence, while a software or training failure requires recovery.
Finally, after exploring many different approaches, we discovered that dramatically different modelling methods often converged within a surprisingly narrow performance range.
This forced us to ask a deeper research question:
Was the bottleneck our search strategy, or was the available predictive information itself becoming the limiting factor?
Accomplishments that we're proud of
Our proudest accomplishment is that EurekaML became more than a scripted AutoML search.
It can autonomously move through:
Research question → Hypothesis → Research allocation → Implementation → Experiment → Evidence → Reflection → Next hypothesis
with minimal human intervention.
Its research priorities evolve as experimental evidence accumulates rather than being fixed beforehand.
EurekaML can also generate modelling capabilities that were not originally present in its experiment catalogue, allowing its research space itself to evolve.
Across our development, EurekaML investigated ranking losses, feature engineering, sampling strategies, multi-task learning, behavioural modelling, interaction architectures and dynamically generated models.
We are also proud that unsuccessful experiments remain valuable. Instead of discarding negative results, EurekaML incorporates them into its future research decisions.
We did not just build another recommendation model — we built the system that researches the models.
What we learned
Our biggest lesson was that autonomous ML research should not mean blindly running more experiments.
After exploring many different modelling directions, we observed that performance frequently converged within a narrow range.
We investigated whether previously unused features contained the missing predictive signal. Even strong platform-level behavioural aggregates provided limited incremental information beyond what the existing model had already learned.
This changed how we thought about autonomous research.
The best research decision is not always another training run.
Sometimes evidence indicates that a direction has been sufficiently explored, that different signals contain largely redundant information, or that another expensive experiment has low expected value.
This reinforced one of EurekaML's central ideas:
Knowing what not to pursue is also a research result.
A capable autonomous research scientist should not only know how to explore and exploit. It should eventually recognise diminishing returns and determine when it needs to change the kind of question it is asking.
What's next for EurekaML
Our next goal is to evolve EurekaML from an autonomous experimenter into an even more complete autonomous ML scientist.
We want EurekaML to automatically diagnose why research has plateaued and distinguish between:
- optimisation bottlenecks;
- inadequate model representations;
- unexplored research directions;
- redundant features;
- missing predictive information; and
- insufficient data.
We also want to extend the Multi-Armed Bandit research portfolio with explicit information-gain reasoning.
Instead of valuing an experiment only by its expected performance improvement, EurekaML could also consider how much uncertainty the experiment is expected to remove.
This would allow the agent to ask:
"Which experiment would teach me the most?"
Finally, we want to extend EurekaML beyond KuaiRand-Pure and recommender systems so the same autonomous research framework can adapt its hypothesis space, capabilities and research strategy across different machine learning problems.
Ultimately, EurekaML should autonomously decide whether the best scientific action is to:
exploit, explore, invent, investigate, or stop.
That is our vision for an autonomous ML research scientist.
Development tools
- Visual Studio Code — primary development environment
- Git / GitHub — source control and project management
- Command Line / Terminal — autonomous research campaigns, testing and evaluation
APIs
- Anthropic Claude API — research hypothesis generation, reflection, model specification and research planning
Libraries and frameworks
- Python — primary implementation language
- NumPy — numerical computation and baseline modelling
- pandas — dataset processing and experiment analysis
- PyTorch — neural model architectures and training
- pytest — automated testing and integrity checks
Dataset and assets
- KuaiRand-Pure — required recommender-system benchmark
- Official challenge Starter Kit — official dataset splits, Factorization Machine baseline, evaluator and submission schema
EurekaML reproduces the official Factorization Machine baseline before autonomous experimentation and uses the official evaluation procedure throughout development.
Log in or sign up for Devpost to join the conversation.