TAISP Baddies Track 2

An autonomous research agent that reads the problem, writes the pipeline, and iterates — no human in the loop. Proposes research ideas, tests them, and logs everything by itself.

Comment

Inspiration

We took inspiration for our agent from such projects as AIDE and MARS. While AIDE is given as a prior work in the task description, we would elaborate more on MARS.

Where AIDE searches greedily over whole-file rewrites and learns only a scalar per node, MARS adds three things: budget-aware Monte Carlo tree search that penalises expensive nodes, comparative reflective memory that distils why a change helped, and diff-based editing so changes stay localised. On MLE-bench with a fixed backend, that combination is worth roughly +20 points of medal rate over AIDE.

Because MARS wasn't open-sourced we took AIDE as a base model and started our journey into building an agent that can solve ML tasks. We called it AgentML.

What it does

An autonomous research agent for the KuaiRand-Pure ranking benchmark. It reads the task description, writes a full training and evaluation pipeline, scores itself on validation, proposes the next hypothesis from its own results, and iterates — logging hypothesis, code diff, metrics, and error-recovery events at every step.

It also audits itself. Before trusting any score, it checks whether that score could have been produced by leakage rather than by ranking ability.

How we built it

  • Seeded from a working baseline, not from scratch, so every experiment starts from a proven anchor rather than reinventing it.
  • Isolated re-verification — any candidate claiming a new best result gets re-run from scratch before being trusted, catching false positives from execution bugs.
  • Causal memory — after each experiment, the agent explains why the metric moved (not just what changed), checked against a measured noise floor so it doesn't invent stories for random variation. That reasoning carries forward into every later step.
  • Targeted code edits — improvements are applied as small, surgical patches instead of full rewrites, so working functionality (multi-seed averaging, correct formatting) never gets silently lost.
  • Merge step — when the search plateaus, the agent combines the best complementary ideas from separate branches into one solution, instead of discarding whichever branch wasn't picked.
  • Ensembling — diverse good candidates (within a run and across runs) get blended together, tuned only on validation data.
  • Directed campaigns — rather than repeating the same idea, separate runs are each briefed on a different research direction, with findings from earlier runs feeding into later ones.
  • Robust LLM interface — a local proxy salvages usable answers from truncated model output and tunes timeouts so long reasoning generations aren't cut off mid-thought.

Challenges we ran into

The hardest problem for us wasn't a bug, it was figuring out what was actually limiting our score. When a candidate plateaued, we couldn't immediately tell if we'd hit a real ceiling in the architecture, or if we were just asking too much of the model generating the code. So we tested that directly, running the same search against a whole range of models, cloud and local, before settling on Qwen3 27B. That process itself became one of the most useful things we did: it told us when a plateau was a genuine limit worth redesigning around, versus just a weaker model failing to execute a good idea.

Accomplishments that we're proud of

We're proud that we beat last year's SOTA solution, and did it inside a short hackathon window. But what we're more proud of is what we built to get there: an agent that doesn't just write code, but checks its own work, explains why a result moved, and builds on its own past attempts instead of repeating them. Getting an agent to reliably tackle a real, open-ended ML problem, not a toy task, and trust its own conclusions turned out to be a genuinely interesting problem in its own right, separate from the recommendation task itself.

What we learned

We learned to actually verify results before trusting them. We had a moment where the agent reported a fantastic score, 0.6714, way above anything else we'd seen. We were about to celebrate it, but something felt off, so we reran the exact same code ourselves outside the framework. It crashed immediately. A silent bug meant the model's own "main" function had never actually executed, and the framework had quietly graded an empty run as a success. That single moment changed how we built everything after: every claimed improvement now gets reexecuted from scratch in isolation before we believe it, and every result has to survive multiple random seeds before we call it real, because we also saw plenty of single-seed scores that looked great on validation and fell apart the moment we checked them on held-out data.

What's next for TAISP Baddies Track 2

We are inspired to continue developing this project and we have several suggestions.

We should stop hand-tuning MARS ourselves and let the agent do it. We picked one similarity signal (shared authors), one activity threshold, and trained it exactly once. AIDE is built to sweep exactly this kind of choice automatically -- Donor selection strategy, activity threshold, how much borrowed history to trust -- the same way it already sweeps hyperparameters for everything else. We suspect a properly searched version would beat the one we picked by hand, and it would get there faster than we could by testing configurations one at a time.

Built With

Share this project:

Updates