AutoRec Lab: Auditable Autonomous Recommender Research

Inspiration

Recommender-system research involves a repetitive loop: inspect results, form a hypothesis, modify the model, run an experiment, evaluate it, recover from failures, and decide what to try next. An LLM can accelerate this process, but allowing generated code to access the entire environment creates serious risks. It could accidentally inspect held-out labels, modify the evaluator, expose credentials, or report results that cannot be reproduced.

We built AutoRec Lab to explore a safer approach: allow the agent to be creative while keeping data access, evaluation, and evidence generation under trusted deterministic control.

What it does

AutoRec Lab is an autonomous experimentation framework for recommender systems.

For every experiment, the agent produces a structured hypothesis, deterministic seed, configuration, and complete candidate model. The trusted controller then:

Validates the generated source code.
Records the actual source-code diff.
Runs the candidate inside an isolated Docker worker.
Prevents the worker from accessing the network, credentials, or held-out labels.
Computes validation metrics outside the generated code.
Records metrics, errors, recovery events, token usage, time, and resource consumption.
Stops when validation performance converges or the iteration budget is exhausted.
Selects the validation-best candidate rather than simply using the final attempt.

The repository also contains an AliCCP adapter supporting:

joining AliCCP skeleton and common-feature files;
filtering impossible (click=0, conversion=1) records;
train-only preprocessing to prevent data leakage;
CTR AUC over all impressions;
CVR AUC over clicked impressions only;
separate challenge and public-NISE compatibility evaluation modes; and
label-free CTR/CVR prediction output.

How we built it

The system is divided into three security boundaries:

1. Agent control plane

Provider-neutral adapters support OpenAI-compatible Chat Completions and Anthropic Messages APIs. The agent receives bounded dataset summaries and trusted aggregate feedback, then proposes the next hypothesis and model implementation.

2. Isolated experiment worker

Generated code runs inside a Docker worker with:

no network access;
no API credentials;
no validation or test labels;
a read-only root filesystem;
dropped Linux capabilities;
CPU, memory, process, file-size, and timeout limits; and
an explicit allowlist of numeric operations.

3. Trusted evaluation and evidence plane

Metrics are calculated outside generated code. Each iteration records its hypothesis, source diff, score, errors, recovery events, and resource use. The evidence validator checks score files, hashes, source provenance, convergence, and validation-best selection.

We used Python 3.12, Docker, NumPy, PyTorch, pandas, scikit-learn, and GitHub. The repository contains 147 passing automated tests, with one expected platform-specific skip.

Challenges we faced

The greatest challenge was making autonomy auditable. A final score alone cannot show whether validation labels leaked into training, whether failed experiments were omitted, or whether the reported model was actually selected by the stated convergence process.

We therefore treated the evidence trail as part of the product. Generated code cannot calculate or report its own official metric, and every completed iteration is connected to retained score files and source changes.

AliCCP presented another challenge because CVR must be evaluated only over clicked impressions, while the public NISE implementation evaluates it differently. We preserve public-NISE behavior only as a clearly labelled compatibility mode and use the clicked-impression definition at the trusted challenge boundary.

Results

To provide a completely reviewable run without requiring private credentials, the repository includes a deterministic CPU reference run for the KuaiRand-Pure GAUC/nDCG@5 protocol.

It converged after 6 of the 50 permitted iterations. The validation-best factorization machine achieved:

GAUC: 0.66739545
nDCG@5: 0.53612650
Primary mean: 0.60176097
Mean component delta: +0.00021097
Wall-clock time: 190.372 seconds
GPU-hours: 0
Manual interventions: 0
LLM tokens: 0

These values are explicitly the deterministic KuaiRand-Pure reference, not an AliCCP score or live-provider result. The AliCCP adapter is implemented and tested, but we do not invent an AliCCP or protected-test score without a genuine data-backed run.

What we learned

The main lesson was that autonomous ML research needs a security boundary, not only a good prompt. Evaluators, credentials, held-out labels, resource limits, and model selection must remain under deterministic control.

We also learned that reproducibility requires recording unsuccessful experiments and recovery events, not only the winning configuration.

What’s next

Our next step is to connect the AliCCP adapter to a full data-backed agent run, reproduce the organizer-pinned NISE baseline, and execute a budget-capped live search. After selecting the validation-best model, the protected evaluation should be run exactly once using the pinned audit environment.

Team contributions

Sutolimin Widjaja, Jayden Jurianto, and Andreas Auwyano collaborated across research planning, implementation review, testing, and submission preparation.