Autonomous RecSys Researcher

An autonomous ML agent powered by Gemini that independently architectures, debugs, and tunes PyTorch recommendation models, successfully beating the ByteDance baseline with zero human intervention.

Comment

💡 Inspiration

In industrial Recommender Systems (RecSys), Machine Learning Engineers spend the vast majority of their time trapped in a repetitive, manual cycle: engineering features, tuning architectures, debugging syntax crashes, and fighting overfitting.

I wanted to build an Autonomous ML Agent that could handle this entire cyclical R&D loop completely on its own. The ultimate goal was to see if an LLM could not only write PyTorch code, but mathematically evaluate its own experiments, debug its own tracebacks, and systematically beat a professional baseline on a real-world Click-Through Rate (CTR) dataset.

🚀 What it does

My project is a fully autonomous ML Research pipeline designed for the KuaiRand CTR prediction benchmark. Given the starter code and dataset, the agent acts as a tireless ML Engineer:

  1. Architects & Upgrades: It independently rewrote the basic NumPy Factorization Machine into a highly optimized PyTorch DeepFM neural network.
  2. Evaluates & Reflects: It trains the model, scrapes the standard output for GAUC and nDCG metrics, and writes a "Reflection" note to its future self on how to improve the next iteration.
  3. Manages Convergence: It uses a custom Sliding-Window Convergence Tracker that perfectly adheres to the hackathon's strict mathematical early-stopping rules.

⚙️ How I built it

The project was built entirely in Python using a decoupled architecture:

  • The Brain: The Google Gemini API (gemini-3.5-flash / gemini-3.7-flash) powers the agent's code generation, error interpretation, and strategic reflection.
  • The Environment: A Python subprocess wrapper safely executes the generated PyTorch code in isolation, capturing both standard output (metrics) and standard error (tracebacks).
  • The Math: I implemented a strict sliding-window algorithm that looks exactly \( N = 3 \) successful steps backward to calculate the net improvement delta \( \Delta \ge \epsilon \), where \( \epsilon = 0.002 \).

🚧 Challenges I ran into

1. The "Complexity Trap" & "AI Goldfish" Problem The LLM inherently knew that Attention networks (AutoInt) and Deep & Cross Networks (DCN-v2) are state-of-the-art for RecSys. However, on the small KuaiRand-Pure dataset (1.1M rows), these massive models instantly overfit. Furthermore, the LLM had a "goldfish memory," often overwriting constraints and reverting to these complex models.

  • Solution: I engineered strict "Anti-Goldfish" system prompts to constrain the AI, forcing it to hyper-tune a simpler DeepFM model and retain techniques like AdamW and nn.BatchNorm1d across iterations.

2. Unfair Convergence Penalties Early on, if the LLM wrote a PyTorch syntax error, the script crash would count as a "strike" against the early-stopping rule, unfairly killing the run.

  • Solution: I engineered a crash-resilience protocol. When a traceback occurs, the agent feeds the error to the LLM but freezes the convergence sliding window. This allows the agent to demonstrate true robustness—debugging its own code without mathematically exhausting its ML patience budget.

🏆 Accomplishments that I'm proud of

  • Beating the Baseline: The agent successfully engineered a model that achieved a Hidden Test Primary Score of 0.5966, definitively beating the official ByteDance baseline of 0.5946. This represents a +0.0020 absolute delta, which is 2.5x larger than the dataset's standard deviation.
  • 100% Autonomy: The R&D loop operated with exactly 0 manual interventions during the active R&D phase. It successfully navigated 7 iterations, handled its own errors, and dynamically achieved mathematical convergence completely unattended. (Note: 1 manual intervention was performed post-convergence to append CSV export logic).
  • Resource Efficiency: The agent hit convergence utilizing only 38,340 tokens and 5.65 minutes of wall-clock time on a CPU.

📚 What I learned

I learned that in Autonomous ML Research, data volume dictates architecture, and complexity is a double-edged sword. Teaching an LLM to write PyTorch code is straightforward; however, teaching an LLM to understand the nuanced relationship between dataset size, model complexity, and overfitting requires highly targeted meta-prompting and rigid environmental constraints.

⏭️ What's next for Autonomous RecSys Researcher

The current agent architecture is entirely dataset-agnostic. While currently constrained to DeepFM to prevent overfitting on the KuaiRand-Pure dataset, the true potential of this framework lies in massive data.

Next, I plan to unleash this exact agent on the 322-million-row KuaiRand-27k dataset, removing the architectural constraints so the LLM can freely engineer the massive DCN-v2 and Multi-Head Self-Attention networks it naturally wants to build.

Built With

Share this project:

Updates