AutoRecAI

Project Story

What Inspired Me

The challenge was not simply to train a higher-scoring recommender model; it was to build an autonomous agent that discovers and validates recommender improvements on its own.

That framing fundamentally shifts the goal: the primary deliverable is not just a static score, but an autonomous scientific process capable of reasoning about domain constraints, implementing minimal code changes, evaluating empirical evidence without bias, and knowing when to stop. Rather than using an LLM as an interactive chatbot, I wanted to deploy DeepSeek V4 Pro as the core reasoning engine of an automated empirical research loop.

The KuaiRand benchmark served as the ideal proving ground. With an official FM baseline variance of σ = 0.0008 across random seeds, meaningful improvements are measured in thousandths of a metric point. This demanded rigorous statistical decision-making, strict control arms, and robust isolation from random seed noise.

The Problem

KuaiRand requires ranking logged video impressions within each user session so that high-engagement content (long view) ranks at top positions. The official benchmark evaluation metric is the mean of two intra-user ranking metrics:

primary = 0.5 × (GAUC + nDCG@5)

Where GAUC is the positive-count weighted average of user-level ROC-AUC:

         Σ_{u ∈ U} (w_u · AUC_u)
GAUC = ─────────────────────────── ,   where w_u = # {positives of user u}
               Σ_{u ∈ U} w_u

And nDCG@5 measures top-5 ranking quality against the ideal discounted cumulative gain:

nDCG@5 = (1 / |U|) × Σ_{u ∈ U} (DCG@5_u / IDCG@5_u)

where:
  DCG@5_u = Σ_{i=1..5} (2^{rel_{u,i}} - 1) / log2(i + 1),  rel_{u,i} ∈ {0, 1}
  IDCG@5_u = ideal DCG@5 achievable for user u

The official Factorization Machine baseline achieves 0.6016 on the validation split. While this might appear far from 1.0, that ceiling is illusory: because 27.1% of test users have zero positive impressions (nDCG ≡ 0) and 9.2% are all-positive, the theoretical oracle ceiling (a model with perfect access to ground-truth labels) is 0.8484 on validation. The FM baseline already consumes roughly one-third of all attainable headroom, leaving an effective competitive window of ≈ 0.2468 where every basis point must be earned through genuine generalization.

How I Built It

AutoRecAI operates as a closed autonomous loop: Context Construction → Hypothesis Generation → Code Diff → Sandboxed Execution → Evaluation → Reflection & State Update.

  ┌──────────────────────────────────────────────────────────┐
  │                 AutoRecAI Autonomous Loop                │
  └──────────────────────────────────────────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │  1. Compact Context  │
                   └──────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │ 2. DeepSeek Hypoth.  │
                   └──────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │ 3. Unified Diff Gen  │
                   └──────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │ 4. Static Leak Gate  │ ──► Leak Detected ──► Re-prompt
                   └──────────────────────┘
                              │ Passed
                              ▼
                   ┌──────────────────────┐
                   │ 5. Sandboxed 3-Seed  │ ──► Runtime Crash ──► Self-heal
                   │    Train & Eval      │
                   └──────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │ 6. Stat. Accept Gate │ ──► Gain < 2σ ──► Reject / Log
                   │    (Δ ≥ +0.0015)     │
                   └──────────────────────┘
                              │ Passed
                              ▼
                   ┌──────────────────────┐
                   │ 7. Reflection & Mem  │
                   └──────────────────────┘
                              │
                              ▼
                   ┌──────────────────────┐
                   │ 8. Convergence Check │ ──► Done (Δ < ε for N=3)
                   └──────────────────────┘

Each iteration:

Context Manager: Synthesizes a token-efficient brief containing problem constraints, current best score, iteration history, and failed hypotheses.
Hypothesis & Code Diff: DeepSeek V4 Pro proposes an ML hypothesis and generates a minimal unified diff against the current best code.
Execution Sandbox: The diff is applied in an isolated iteration directory branched only from verified successful states.
Multi-Seed Evaluation: The model is trained and evaluated across 3 independent seeds on the validation split (the hidden test split is never touched during search).
Gating & Reflection: The outcome is validated against three strict gates before reflection updates the persistent knowledge memory.

Three Robustness Gates:

Multi-Seed Acceptance Bar: A candidate code change is accepted if and only if its 3-seed mean primary score improves by Δ ≥ +0.0015 ≈ 2σ. This prevents the agent from adopting random seed fluctuations.
Static Leakage & Plausibility Gate: Automatically rejects any code referencing current-row engagement outcomes (play_time_ms, is_click, long_view) as input features, and triggers a safety tripwire if scores approach the oracle ceiling (0.8484).
Formal Convergence Protocol: Evaluates whether three consecutive iterations fail to beat the threshold ε = 0.002, stopping execution cleanly when progress plateaus.

Key Architectural Findings:

The primary modeling breakthrough came from engineered sequential dynamics: capturing user intra-day hour, sequential impression index, inter-impression time delta, and lagged historical watch-ratio fed into a specialized sequence MLP head:

ŷ = σ( w_0 + Σ_i w_i x_i + 0.5 × Σ_f [ (Σ_i v_{i,f} x_i)² - Σ_i v_{i,f}² x_i² ] + MLP_seq(x_seq) )

Adding this sequence MLP head produced a verified +0.0024 paired gain across all random seeds over the baseline.

Next, a Mini-DIN target-attention architecture was tested:

          exp( qᵀ k_t / √d )
α_t = ─────────────────────────── ,    h = Σ_{t=1..L} α_t k_t
      Σ_{j=1..L} exp( qᵀ k_j / √d )

The agent evaluated 3 distinct DIN variants across all seeds and determined that target-attention on short 2-week history windows degraded test generalization (deltas −0.0019 to −0.0027). The agent rejected all 3 variants and logged the dead end.

Finally, a 5-seed probability ensemble was generated, boosting the validation primary score to 0.6086 (+0.0070 over baseline) by smoothing pointwise decision boundaries.

Challenges I Faced

Separating Signal from Noise: With baseline seed standard deviation σ = 0.0008, an apparent +0.0003 gain is statistical noise. Enforcing paired control arms and 3-seed mean evaluation prevented deceptive mutations from surviving.
The 99.7% Multi-Task Trap: Auxiliary click prediction initially seemed intuitive. However, dataset inspection revealed that is_click is a superset of long_view for 99.7% of rows, offering virtually zero orthogonal supervision and resulting in a confirmed no-op.
Listwise Ranking Loss Variance: Softmax/ApproxNDCG listwise losses on short user impression lists (average ~4 items) suffered from severe gradient variance, underperforming stable binary cross-entropy by −0.001 to −0.002.
Autonomous Self-Healing: When the LLM produced a syntax runtime error or attempted a feature leakage, automated feedback loops re-prompted the model with error tracebacks, recovering valid executions without human intervention.
Strict Compute Efficiency: Every experiment was engineered to run rapidly on CPU. The entire converged Pure benchmark search consumed only 67,821 tokens and 9 minutes 48 seconds of wall-clock execution (0 GPU-hours).

What I Learned

Autonomous ML research requires statistical discipline encoded directly as automated architecture.

Prompting an LLM to "improve a model" produces unconstrained, noisy ideas; but embedding that LLM within strict multi-seed acceptance gates, static leakage tripwires, and persistent failure memory transforms it into a reliable scientific discovery engine. Documented negative results (e.g., DIN attention, listwise losses, multi-task auxiliary heads) proved just as vital as positive gains by mapping the boundaries of the problem space.

Benchmark Results

1. Required Benchmark: KuaiRand-Pure

Metric	Official FM Baseline (Valid)	Single Model (3-Seed Mean)	5-Seed Ensemble (Valid)	Total Delta vs Baseline
GAUC	0.6674	0.6742	0.6772	+0.0098
nDCG@5	0.5357	0.5386	0.5400	+0.0043
Primary (Mean)	0.6016	0.6064	0.6086	+0.0070

Single Model Valid Primary: 0.6064 (+0.0048 over baseline)
5-Seed Ensemble Valid Primary: 0.6086 (+0.0070 over baseline)
Official Convergence Rule: Reached at iteration 3 with 0 manual interventions.
Canonical from-scratch proof (config_canonical.yaml, official-baseline start, no pre-baked template): baseline reproduced 0.6016 → autonomous experimentation → 0.6038 (+0.0022) at iteration 8 → ε=0.002/N=3 convergence at iteration 13 → 0 manual interventions; submission final_submission_canonical/submission.csv validated.
Agent-efficiency control (config_canonical_gate.yaml, same official start): with persistent evidence-aware research memory + a novelty/redundancy gate over measured verdicts, the agent reached the same model family (0.6031, within 3-seed noise) with 38% fewer iterations, 30% fewer tokens, 54% less wall-clock, and zero repeated measured dead ends — still 0 manual interventions. DeepSeek thinking remains disabled by measured decision (8/12 thinking-on calls starved at the production budget with no quality gain).

2. Bonus Benchmark: KuaiRand-1K

Metric	Official FM Baseline (Valid)	Single Model (3-Seed Mean)	3-Seed Ensemble (Valid)	Total Delta vs Baseline
GAUC	0.6746	0.6983	0.6998	+0.0252
nDCG@5	0.6136	0.6279	0.6346	+0.0210
Primary (Mean)	0.6441	0.6641	0.6672	+0.0231

Single Model Valid Primary: 0.6641 (+0.0200 over baseline)
3-Seed Ensemble Valid Primary: 0.6672 (+0.0231 over baseline)

Resource Usage & Telemetry

Benchmark Run	Total Tokens (In + Out)	Wall-Clock Duration	Iterations Used	GPU Hours
KuaiRand-Pure (Required)	67,821 (60,512 in + 7,309 out)	9 min 48 s	3 / 50	0 (CPU only)
KuaiRand-Pure canonical from-scratch (official-baseline start)	320,777	57 min 1 s	13 / 50	0 (CPU only)
KuaiRand-Pure agent-efficiency control (evidence-aware memory + novelty gate, official-baseline start)	223,292	26 min 9 s	8 / 50	0 (CPU only)
KuaiRand-1K (Bonus)	346,083 (260,084 in + 85,999 out)	4 h 5 min 50 s	10 / 50	0 (CPU only)

Built With

Language & Frameworks: Python 3.9, PyTorch, NumPy, pandas, PyYAML, requests
LLM Reasoning Engine: DeepSeek V4 Pro (deepseek-v4-pro via OpenAI-compatible API)
Evaluation Protocols: Official KuaiRand-Pure within-user ranking (GAUC + nDCG@5), 20-test automated regression suite
Deliverables Package: AutoRecAI_submission.zip (uploaded to Devpost)
GitHub Repository: https://github.com/sasssyboujee/kuairand-agent