AIDE-Based Autonomous Optimization for KuaiRand-Pure Ranking

Motivation

The starting point of this project was a practical observation from the KuaiRand-Pure statement: the benchmark is evaluated as a user-level ranking task, while the official starter-kit baseline is trained as a pointwise Factorization Machine (FM). This does not mean the baseline is weak. In fact, the FM baseline is already a strong and carefully designed reference. But it does suggest that useful improvements are more likely to come from aligning the search process with the ranking nature of the task than from blindly adding features.

We therefore framed the project as a constrained autonomous research problem. This framing led to our main design choice: before allowing open-ended exploration, the agent should first follow a small number of task-informed search stages derived from our analysis of the benchmark.

Project Summary

We adapted AIDE ML into an autonomous, auditable experiment runner for the KuaiRand-Pure benchmark. The agent first reproduces the official FM baseline as iteration 0. It then proposes candidate ranking improvements, generates runnable code, executes that code in isolated workspaces, evaluates successful candidates with the official evaluate.py, and records every hypothesis, code diff, metric result, and error event in a per-iteration log.

The selected final approach combines:

weekday_hour_bucket feature + same-user uniform-negative BPR

The weekday_hour_bucket feature adds coarse temporal context from the exposure date and hour. The BPR objective changes the learning problem from pointwise label prediction to pairwise within-user ranking: for the same user, a positive long-view exposure should receive a higher score than a negative exposure.

How the Solution Addresses the Problem Statement

The problem statement asks for an agentic system that can autonomously explore improvements for the KuaiRand recommendation benchmark while producing per-iteration logs and following the official evaluation rules. Our solution addresses this in three ways.

First, we preserve the official benchmark harness. The official data split, target label long_view, submission schema, GAUC implementation, nDCG@5 implementation, and primary metric definition are kept unchanged. This ensures that every candidate solution is evaluated under the same rules as the starter kit.

Second, we make the search process auditable. Each AIDE iteration records what the agent intended to try, why it believed the idea was relevant, what code it generated, what metric result it obtained, and whether any error occurred. Buggy nodes remain in the log, but they cannot overwrite the last accepted submission artifact.

Third, we connect the search procedure to our task analysis. The agent is not simply asked to "improve the model." Instead, its early search is structured around the main technical hypothesis suggested by the statement: since the metric is based on within-user ranking, the agent should first test features that can affect within-user ordering and objectives that more directly optimize ranking quality.

Why AIDE Fits This Task

AIDE is well suited to this problem because the benchmark naturally forms an iterative research loop. A candidate solution can be proposed, implemented, run, scored, compared, and revised. This is the structure AIDE is designed for: it operates as a coding-and-evaluation agent rather than a one-shot code generator.

For this project, we extended AIDE's behavior in several important ways.

1. Baseline Seeding

The first iteration is forced to reproduce the official FM baseline. This gives the run a trustworthy reference point and verifies that local data loading, scoring, and submission generation are correct.

2. Task-Informed Staged Search

Before free exploration, the agent follows three controlled search stages derived from our analysis of the problem:

1. Feature-domain ablation. The agent first tests compact candidate-aware feature domains, focusing on fields that can change within-user item ordering rather than static user-side constants.

2. Ranking-objective search. The agent then tests pairwise/listwise ranking ideas, with BPR prioritized because GAUC and nDCG@5 evaluate relative order rather than independent classification accuracy.

3. CWM-inspired auxiliary modeling. Finally, the agent considers whether censored watch-time modeling can provide useful auxiliary signal, inspired by the CWM paper and reference implementation.

Only after these stages does the agent enter freer exploration, and even then it remains constrained by the official evaluator, submission schema, and convergence rule.

3. Artifact Protection

When a non-buggy node produces a valid submission.csv, the runner snapshots it as:

aide_runs/workspaces/<run>/working/accepted_submission.csv

Later failed experiments cannot overwrite this accepted artifact.

4. Unchanged Convergence Rule

The project keeps the official stopping rule: a run converges if validation primary does not improve by more than epsilon = 0.002 for N = 3 consecutive iterations, or if it reaches the 50-iteration cap or 6-hour wall-clock limit.

Together, these changes make AIDE a benchmark-aware autonomous research assistant: it can explore, but its exploration is shaped by the problem statement and checked by the official scoring harness.

Design and Implementation

The implementation was developed in VS Code, using PowerShell for local execution and Git/Git LFS for version control and packaging.

The agentic component uses the OpenAI API with gpt-5.6-sol as the main model for planning, code generation, feedback interpretation, and report generation. The local project is built around a modified AIDE ML checkout, which manages task prompts, workspaces, execution, logging, and convergence tracking.

The main libraries and frameworks used include:

• Python for the full experiment pipeline;
• NumPy for FM/BPR training arrays and score computation;
• pandas for reading and validating raw interaction logs;
• PyTorch as part of the broader AIDE ML and experimentation environment;
• Hydra/OmegaConf for AIDE configuration;
• scikit-learn utilities for supporting experimentation workflows;
• igraph for AIDE search-tree visualization.

The dataset used is KuaiRand-Pure. We did not use manually labeled data or external prediction assets.

Search Process and Key Findings

Our first requirement was to reproduce the official FM baseline. This became iteration 0 of the AIDE run and achieved validation primary 0.601469, confirming that the local pipeline matched the starter-kit evaluator.

The next stage tested candidate-aware feature domains. This step was guided by the structure of the metric: because ranking is computed within each user, additional features are useful only if they can change the relative order of items for the same user. The agent therefore focused on features such as author history and temporal context.

The strongest final direction was not simply adding more fields, but changing the training objective. Same-user BPR directly compares positive and negative exposures from the same user, making it better aligned with GAUC and nDCG@5 than pointwise logloss. Under this setup, weekday_hour_bucket produced the best validation result among the successful candidates.

Buggy Nodes and Recovery Behavior

Autonomous code generation can fail, so the runner treats buggy nodes as part of the research process rather than as silent failures.

When a candidate crashes or fails validation, the node is marked as buggy in journal.json. Its code, traceback, and attempted hypothesis are still preserved. However, the node cannot replace the accepted submission, cannot become the final selected solution, and cannot overwrite accepted_submission.csv.

After a buggy node, the agent may take one of two paths depending on the search state and feedback:

1. It may enter a debug/improvement path, using the traceback and previous code to repair the failed idea.
2. It may abandon that branch and start a fresh candidate from a stronger valid parent.

This behavior mattered in our run. One generated feature-engineering attempt assumed the raw training log was strictly chronological. That assumption failed. The system preserved the error, marked the node as buggy, and then produced a recovery iteration that fixed raw-log alignment by validating against the official loaded split order and preserving duplicate exposure rows.

This design allowed the agent to continue learning from failed attempts without letting them corrupt the final artifact.

Built With

• agent
• aide
• ml
• python