Autonomous AI Research Agent

Inspiration

Machine learning engineers spend a huge amount of time repeating the same cycle: form a hypothesis, modify a model, run an experiment, analyze the results, and decide what to try next. We wanted to ask: what if an AI agent could take over this research loop?

Instead of simply asking an LLM to generate code, we wanted to build an agent that could genuinely experiment, learn from failures, and make evidence-based decisions about what to try next.

What it does

Our Autonomous AI Research Agent iteratively improves a recommender system on the KuaiRand benchmark.

The agent:

Generates hypotheses based on previous experiments
Researches potential ML improvements
Writes and modifies model code
Validates its proposed changes before running them
Trains and evaluates candidates using GAUC and nDCG@5
Accepts improvements and rejects regressions
Recovers from implementation failures
Maintains experiment memory to avoid repeating failed ideas
Determines when further experimentation is unlikely to produce meaningful gains

The result is an autonomous hypothesis → code → experiment → evaluation → reflection loop.

How we built it

We built the agent in Python using LangGraph for orchestration and the Gemini API as the reasoning engine.

The ML pipeline uses PyTorch, NumPy, pandas, scikit-learn, XGBoost, LightGBM, and TensorFlow across different experiments.

Our architecture separates the agent into components for planning, research, code generation, validation, evaluation, reflection, memory, recovery, and convergence.

A major part of the system is our experimental validation framework. Because small changes in recommender metrics can come from random seed noise, we introduced multi-seed evaluation and minimum-effect thresholds so the agent does not mistake a lucky experiment for a genuine improvement.

Challenges we ran into

Our biggest challenge was that not every improvement suggested by the agent was actually an improvement.

Early experiments showed promising gains from techniques such as multi-task learning and attention, but these disappeared when tested across multiple random seeds.

We also encountered practical agent failures, including malformed LLM-generated code, parser failures, quota issues, and an early bug where the agent compared candidates against the previous experiment instead of the actual best model.

These failures forced us to make the agent more robust through validation, checkpointing, recovery mechanisms, and better experiment memory.

Accomplishments that we're proud of

We are proud that our agent can perform a meaningful ML experimentation loop with zero manual interventions during the completed autonomous run.

The agent independently discovered a strong interaction-based architecture, improving the validation primary score from approximately 0.60144 to 0.60422.

Our final three-seed ensemble achieved:

GAUC: 0.6658
nDCG@5: 0.5312
Primary: 0.5985

This improved the official baseline primary score of 0.5946 by +0.0039.

More importantly, we demonstrated that an autonomous ML agent needs to learn from failed experiments as well as successful ones.

What we learned

We learned that building an autonomous research agent is much harder than simply connecting an LLM to a code interpreter.

The hardest part is creating a reliable feedback loop: the agent needs to understand whether an experiment actually worked, distinguish signal from noise, recover from failures, and remember what it has already tried.

We also learned that better experimentation methodology can be more valuable than simply trying larger or more complex models. Multi-seed validation and rigorous acceptance criteria prevented several misleading improvements from being accepted.

What's next for Autonomous AI Research Agent

Our next step is to make the agent a more capable general-purpose AI research assistant.

We want to improve its ability to automatically discover useful features, reason about temporal patterns, select experiments based on expected information gain, and allocate its compute and LLM budget more intelligently.

Ultimately, we envision an agent that can take a dataset and objective, independently design experiments, implement and evaluate them, learn from the results, and continuously improve its own research strategy — turning ML experimentation from a repetitive manual process into an autonomous research workflow.