RecForge: An Autonomous ML Research Agent for Recommender Systems

A research agent should leave behind evidence—not just a score.

Inspiration

Machine learning research often follows the same repetitive loop: inspect the data, engineer features, train a model, evaluate it, study the failures, and try again. Although each iteration requires judgment, much of the process is structured and code-driven.

We built RecForge to explore a simple question: what if an AI agent could manage this entire research cycle while remaining rigorous, resource-aware, and reproducible?

Most automated ML tools focus on hyperparameter tuning or finding the highest leaderboard score. We wanted to build something closer to an autonomous research laboratory—one that proposes diverse hypotheses, writes and executes experiments, learns from failures, and records enough evidence for another researcher to reproduce every decision.

Recommender systems made this especially interesting. Small mistakes such as temporal leakage, misaligned prediction rows, duplicated interactions, or incompatible evaluation metrics can make an improvement look real when it is not. RecForge was inspired by the idea that automation should make ML research more trustworthy, not merely faster.

What We Built

RecForge is a local-first, CPU-first autonomous ML research agent for recommender systems.

Given a benchmark configuration, RecForge can:

Read and lock the benchmark contract.
Profile the permitted training and validation data.
Reproduce baseline models and evaluation metrics.
Propose typed hypotheses across different research families.
Generate experiment configurations and candidate code.
Execute experiments inside resource-limited subprocesses.
Independently audit predictions for leakage, alignment, and metric correctness.
Reflect on the results and choose the next experiment.
Recover safely from invalid code, timeouts, numerical errors, and other failures.
Export the best validated checkpoint, submission file, research report, and complete provenance record.

Instead of repeatedly changing parameters within one model family, RecForge maintains a portfolio of genuinely different approaches. These include collaborative filtering, feature interactions, graph-based methods, sequential models, semantic side information, multi-feedback learning, negative sampling, contrastive learning, reference models, and ensembles.

Every hypothesis becomes a node in an experiment tree. Its configuration, parent experiment, generated code, source diff, metrics, resource consumption, audit result, and failure state are preserved. This makes it possible to understand not only which experiment won, but also how the system arrived there.

How We Built It

RecForge is implemented in Python 3.11+ using NumPy, pandas, SciPy, Pydantic, PyYAML, FastAPI, Jinja2, Uvicorn, and psutil. The main research path does not require PyTorch, a GPU, or an external model service.

The system is divided into several independent layers:

A contract layer records the dataset split, label definition, evaluation profile, and evaluator fingerprint before optimization begins.
A leakage-safe data layer separates training, public validation, and blind features. Blind rows are parsed without the target column.
A planner selects hypotheses from different scientific families and prevents superficial parameter changes from being treated as new ideas.
A compiler converts each typed hypothesis into reviewable experiment artifacts.
A bounded worker executes candidate code with time, memory, and retry limits.
An independent evaluator verifies finite predictions, row alignment, repeated interactions, metric semantics, and data boundaries.
A SQLite/WAL research ledger stores experiment state, while an append-only SHA-256 event chain protects the history of decisions and recoveries.
A read-only FastAPI dashboard displays convergence, family coverage, experiment ancestry, metric changes, resource usage, failures, and reproduction commands.

We also created two scientist modes. The deterministic planner provides a fully offline and reproducible path, while an optional Codex-powered scientist can propose and revise generated model code. Both pass through the same validation, execution, and auditing boundaries.

On the public KuaiRand-Pure validation split, RecForge completed 19 autonomous iterations across 10 research families. Its validation-best ensemble achieved a primary score of 0.6030456, compared with the organizer-published baseline of 0.6016—an improvement of approximately 0.00145. This is strictly a public validation result; we do not claim access to hidden labels or a hidden-test score.

The run used no GPU, recorded zero manual interventions, and selected the validation-best checkpoint rather than simply returning the final experiment.

Challenges We Faced

One of our first challenges was discovering that the written problem statement and the executable Starter Kit described different evaluation profiles. Comparing results across those definitions would have produced misleading conclusions. We solved this by treating the executable evaluator as the scoring authority, fingerprinting it, and keeping incompatible metric profiles separate.

Preventing leakage was another major challenge. KuaiRand contains time-dependent interactions, so an apparently useful feature may contain information from the future. We introduced a strict temporal boundary and physically removed the target column when blind rows were loaded. Features with uncertain temporal provenance were excluded by default.

We also had to define what counts as meaningful autonomous research. Without safeguards, an agent can spend its entire budget making tiny parameter changes and still report many “experiments.” RecForge instead tracks mechanism signatures and requires coverage across distinct model families before concentrating on a promising approach.

Failure recovery was equally important. Generated code can contain syntax errors, invalid outputs, numerical problems, or excessive resource usage. Rather than hiding these failures, RecForge classifies and records them. Only retryable failures receive bounded retries, while invalid candidates remain visible in the research history.

Finally, building immutable provenance exposed a lineage mismatch in one early synthetic scientist run. Because RecForge preserved the generated source, parent pointer, and event history, we could identify the inconsistency, add a regression test, and fix future runs without rewriting the historical evidence. That experience reinforced the core principle behind the project: transparent failures are more valuable than perfect-looking but unverifiable results.

What We Learned

The biggest lesson was that autonomous ML research is not primarily about generating more code. The difficult part is building a trustworthy control plane around that code.

An agent needs clear contracts, isolated evaluators, protected data boundaries, meaningful experiment diversity, resource accounting, recovery policies, and durable provenance. Without those pieces, autonomy can amplify mistakes just as quickly as it discovers improvements.

We also learned that failed experiments are useful scientific results. A rejected hypothesis, an invalid generated model, or a negative ablation can narrow the search space and inform the next decision—as long as the system preserves that evidence.

Most importantly, we learned that reproducibility should be a product feature rather than an afterthought. RecForge’s final output is not only a trained model or submission file. It is a replayable research bundle showing exactly what was attempted, why it was attempted, what happened, and how the final checkpoint was selected.

What’s Next

Our next step is to expand RecForge with additional recommender-model families, richer literature-informed hypothesis generation, and controlled multi-agent collaboration. We also want to evaluate it across more recommendation benchmarks and study how well its research strategies transfer between datasets.

Longer term, we see RecForge as a foundation for autonomous but accountable ML research: a system that helps small teams explore more ideas while maintaining the scientific discipline required to trust the outcome.