Autonomous ML Research Agent for KuaiRand-Pure

What it does

This project is an autonomous machine-learning research agent for the KuaiRand-Pure long-view ranking benchmark. The challenge is not simply to train one recommender: the agent must reproduce a fixed official baseline, propose and evaluate improvements, recover from failures, and select a final model without using hidden test feedback during development.

Our solution turns that workflow into a governed experiment loop. It starts from the organizer-provided factorization-machine baseline, constructs an evidence context from prior experiments and public validation metrics, asks a constrained research agent for one safe hypothesis at a time, validates the proposal against experiment contracts, runs the candidate, records the outcome, and decides whether to refine or change direction. Candidate selection is strictly limited to training data and public validation results. Test access is human-gated and used only once for the approved final candidate.

The final submission is E0016, a three-seed compact sequence ensemble. It captures strictly prior positive-video history for each user, averages three independent members, and was chosen for replication evidence rather than only the highest single validation run. It achieved a validation primary score of 0.604170 versus the official baseline's 0.601600. On the final test evaluation, it achieved a primary score of 0.597778 versus the baseline's 0.594600:

How it addresses the problem

The project addresses the core autonomous-ML problem in four ways:

1. Baseline reproduction. The organizer baseline is protected by hash checks and reproduced on the public validation split before candidate search.
2. Governed autonomous iteration. The agent can choose only catalogued templates, reviewed features, and one scalar change per experiment. Every proposal has a hypothesis, evidence, expected effect, provenance, and an immutable experiment specification.
3. Leakage-safe ranking improvements. Sequence features are built from earlier training dates only; validation and test contexts are frozen from train history. The test split is removed from candidate execution.
4. Robust operation. The supervisor persists leases and state, detects stale runs, validates proposals before reserving IDs, retries transient model service failures, uses deterministic fallbacks for known invalid proposals, and preserves failed runs as audit evidence. Two real encoder failures were fixed, tested, and retried without losing the original records.

Development tools used

• Python 3.14 and a local Python virtual environment
• Terminal/command-line workflow and Git for version control and immutable experiment commits
• NumPy-based CPU training and evaluation
• unittest for deterministic regression tests
• Devpost-ready Markdown documentation and CSV output validation

APIs used

• Google Vertex AI / Gemini through google-genai, used by the autonomous proposal agent to generate structured, evidence-backed experiment proposals.
• No external API is used for training, validation scoring, or final ranking; those operations run locally on KuaiRand-Pure.

Libraries and frameworks used

• NumPy
• Python standard library (argparse, json, csv, pathlib, subprocess, hashlib, and related modules)
• google-genai
• google-adk
• python-dotenv

The ranking models are implemented directly in NumPy rather than relying on PyTorch, pandas, scikit-learn, or a high-level recommender framework. This keeps the experiment runner deterministic, compact, and easy to audit.

Datasets and assets used

• KuaiRand-Pure, the organizer-provided recommendation benchmark.
• The official train/validation/test temporal split supplied with the starter kit. Training and validation were used during search; the hidden test split was evaluated exactly once after approval.
• The official factorization-machine baseline scores and implementation from the starter kit.
• The saved academic reference Self-Attentive Sequential Recommendation (SASRec) by Kang and McAuley, used as evidence for the sequence-attention direction.

Deliverables

The repository includes the final submission CSV, three final checkpoints, approval receipt, immutable specification, validation evidence, final-test result, iteration log, and resource summary under kuairand-starter-kit/deliverables/phase8/.

Built With

• artificial-intelligence
• automation
• autonomous-agents
• data-science
• deep-learning
• experiment-tracking
• generative-ai
• google-gemini
• llm-agents
• machine-learning
• mlops
• model-evaluation
• numpy
• personalisation
• python
• ranking
• recommendation-system
• recommender-systems
• robustness
• sequential-recommendation
• vertex-ai