Tiktok 2026

An autonomous agent that handles the weekly grind of retraining and tuning your recommender system.

Comment

AutonoRec

Inspiration

Machine learning engineers often spend significant time repeating the same research cycle: propose an idea, modify the pipeline, train, evaluate, analyze the result, and try again.

AutonoRec automates this process for recommender systems.

Instead of creating a single LLM that freely edits code and decides whether its own experiments are successful, we designed a controlled autonomous ML research system where AI agents handle reasoning and implementation while deterministic infrastructure controls execution, evaluation, permissions, provenance, and resource limits.

The goal is to make autonomous ML experimentation both useful and trustworthy.

What It Does

AutonoRec is an autonomous ML research agent designed for the KuaiRand recommendation benchmark.

During a run, the system can:

  • analyze the current experiment state;
  • generate a measurable ML hypothesis;
  • review whether the experiment is valid and worth running;
  • create an isolated Git worktree;
  • implement the proposed change;
  • validate the code diff;
  • execute the experiment in a restricted environment;
  • evaluate predictions using GAUC and nDCG@5;
  • analyze whether the experiment improved the current best result;
  • recover from implementation or execution failures;
  • record the complete experiment history;
  • and automatically decide what experiment to try next.

The loop continues until convergence or the configured experiment/resource limit is reached.

Core Architecture

AutonoRec uses four specialized agent roles.

Orchestrator

The Orchestrator decides what should happen next in the research loop.

It determines whether the system should propose a new experiment, repair a failed implementation, continue evaluation, or stop.

Research Agent

The Research Agent generates new ML hypotheses.

Each proposal includes:

  • the proposed technique;
  • the expected reason for improvement;
  • measurable success criteria;
  • estimated resource requirements;
  • and the code areas expected to change.

The system can explore improvements across feature engineering, model architecture, optimization, sampling, regularization, training strategy, and other parts of the recommender pipeline.

Implementor

The Implementor converts an approved hypothesis into code.

Every experiment runs inside its own Git worktree so the source state is isolated and reproducible.

The Implementor can modify only approved experiment files and cannot overwrite protected benchmark or controller infrastructure.

Validator

The Validator independently reviews experiments and implementations.

It checks whether the code actually implements the intended hypothesis and whether the resulting experiment is scientifically valid.

The Validator is read-only and cannot modify the implementation it reviews.

Deterministic Controller

A key design principle is:

LLMs make research decisions; deterministic code controls authority.

The controller is responsible for:

  • experiment lifecycle management;
  • Git commit identity;
  • protected-file policies;
  • dataset access;
  • execution permissions;
  • resource budgets;
  • evaluation;
  • experiment persistence;
  • failure recovery;
  • convergence;
  • and finalization.

This prevents an agent from changing benchmark rules or declaring its own result valid without independent verification.

Experiment Isolation and Reproducibility

Each experiment is created from an approved parent Git commit.

AutonoRec records:

  • experiment specification;
  • parent commit;
  • resulting source commit;
  • code diff;
  • execution identity;
  • dataset manifest;
  • metrics;
  • artifacts;
  • failures;
  • and recovery events.

This allows every score to be traced back to the exact source code and experiment that produced it.

Secure Execution

Experiments are executed through Docker with configurable restrictions such as:

  • disabled network access;
  • CPU limits;
  • memory limits;
  • GPU allocation;
  • execution timeout;
  • and controlled dataset staging.

Only the authorized training and validation data are available during iterative research.

Hidden-test data is excluded from the normal agent workflow.

Evaluation

AutonoRec evaluates recommendation predictions using:

  • GAUC
  • nDCG@5

The local primary objective is the mean of these two metrics.

The evaluation layer also validates prediction structure and provenance before accepting a result.

A score must correspond to the correct:

  • source commit;
  • dataset;
  • execution;
  • prediction artifact;
  • and experiment identity.

This prevents invalid outputs from entering the research history.

Failure Recovery

Autonomous experimentation frequently produces broken implementations or failed executions.

AutonoRec detects and records failures instead of stopping the entire research process.

Depending on the failure, the system can:

  • retry;
  • repair the implementation;
  • resume from persistent state;
  • or abandon the experiment and propose a different approach.

Scientific non-improvement is treated differently from system failure.

A valid experiment that performs worse is still preserved as useful evidence so the Research Agent can avoid repeatedly testing the same unsuccessful idea.

Resource-Aware Research

The challenge limits the number of iterations and evaluates practical resource usage.

AutonoRec therefore tracks resources as part of the experiment state.

The controller can track:

  • total iterations;
  • wall-clock usage;
  • LLM token usage;
  • GPU usage;
  • memory requirements;
  • disk usage;
  • and experiment runtime.

Research proposals can also estimate their expected computational cost before execution.

This allows the agent to prioritize experiments that are both promising and feasible.

Technology Stack

AutonoRec is implemented primarily in Python.

Core technologies include:

  • LangGraph — autonomous workflow orchestration
  • OpenAI-compatible LLM APIs — research, implementation, validation, and orchestration
  • LiteLLM — optional multi-model routing
  • Pydantic — typed agent and controller contracts
  • PyTorch — recommender model experimentation
  • NumPy / pandas / scikit-learn — data processing and ML utilities
  • Git worktrees — isolated experiments
  • Docker — constrained execution
  • SQLite — persistent experiment state
  • Typer — command-line interface
  • MLflow integration — experiment observability
  • pytest, Ruff, and Pyright — testing and code quality

Repository:

https://github.com/nghiant03/TikTok2026

Dataset

AutonoRec is built for the KuaiRand recommendation dataset family.

The required benchmark is KuaiRand-Pure.

The system follows the organizer-provided train/validation split and Starter Kit conventions. External training data is not added to the research pipeline.

The official Factorization Machine baseline remains protected and is used as the reference for improvement.

Autonomy

AutonoRec automates more than hyperparameter tuning.

The agent system can autonomously:

  • decide what to investigate;
  • generate experiments;
  • implement them;
  • execute them;
  • evaluate results;
  • interpret failures;
  • repair broken experiments;
  • learn from negative results;
  • and select the next research direction.

Human interaction is intended mainly for initial configuration and environment setup rather than manually directing every experiment.

Robustness

The system includes several mechanisms for reliable long-running research:

  • isolated Git worktrees;
  • protected source paths;
  • typed agent outputs;
  • resource limits;
  • execution timeouts;
  • persistent checkpoints;
  • bounded repair attempts;
  • failure classification;
  • source and artifact hashing;
  • recovery validation;
  • and deterministic experiment logs.

These mechanisms allow the system to continue operating even when individual experiments fail.

Results

AutonoRec currently implements the complete infrastructure required to reproduce the baseline, run autonomous experiments, evaluate validation results, track convergence, and preserve experiment provenance.

For the final submission we will report:

  • KuaiRand-Pure GAUC;
  • KuaiRand-Pure nDCG@5;
  • improvement over the official baseline;
  • number of iterations;
  • total agent wall-clock;
  • total LLM token usage;
  • GPU-hours, if used;
  • and number of manual interventions.

Challenges

The main technical challenge was balancing autonomy and control.

Giving an LLM unrestricted repository and execution access makes experimentation flexible but difficult to trust.

Restricting everything through hard-coded rules makes the system safe but removes meaningful research autonomy.

AutonoRec solves this by giving AI agents freedom over scientific decisions while deterministic infrastructure controls execution authority.

What We Learned

Building an autonomous ML researcher requires more than strong language-model reasoning.

A practical system also needs:

  • experiment provenance;
  • secure execution;
  • durable state;
  • reproducibility;
  • failure recovery;
  • resource governance;
  • and independent evaluation.

These components allow the agent to explore more freely without sacrificing trust in the resulting experiments.

Built With

Share this project:

Updates

Submission history