Fypothesis Overview
Submitted Metrics vs Baseline
Configure & Run The Agent
Autonomous Decision Branching
In-depth Experiment Breakdown
Agent Observability With Run Logs & Audit Trails

FYPothesis: AI That Doesn't Just Think. It Experiments.

About the Project

What if an AI could do more than write ML code?

What if you gave it a dataset, an objective, a limited budget, and a deadline, and it could research its way toward a better model on its own?

That is FYPothesis.

FYPothesis is an autonomous ML research agent built for the KuaiRand Pure recommendation ranking challenge. It runs the full research loop:

Hypothesize → Code → Execute → Evaluate → Learn → Confirm → Repeat

The latest autonomous run completed with 0 manual interventions, 27 training executions, $0.72 in LLM spend, 0 GPU hours, and 43 minutes of wall clock time.

Most importantly, it produced a measurable improvement on the hidden test set.

The Problem

ML research is inherently iterative. A researcher forms a hypothesis, implements it, evaluates the result, and decides what to try next.

Today's coding agents can increasingly write the experiment, but humans still decide:

What should we try next?

Did the result actually mean anything?

A naive research agent could simply run hundreds of experiments and keep the highest validation score. But that risks lucky seeds, validation overfitting, wasted compute, and misleading ensembles.

We wanted to build an agent that does not just optimize the score.

It optimizes the quality of the research process itself.

How FYPothesis Works

FYPothesis maintains an append only research history containing hypotheses, experiments, failures, metrics, and decisions.

For each iteration, the agent can:

• Form a measurable hypothesis
• Generate a complete training program
• Validate the experiment before execution
• Run it in an isolated, label protected environment
• Evaluate using the official KuaiRand Pure evaluator
• Analyze the result against previous evidence
• Repair failures using tracebacks
• Abandon unproductive approaches
• Confirm promising ideas with additional seeds
• Build a fixed ensemble
• Stop when further research is no longer justified

This turns an LLM from a code generator into an autonomous research loop.

The Result

The official hidden test baseline was 0.5946.

FYPothesis achieved 0.59810.

Metric	Baseline	FYPothesis	Gain
Primary	0.5946	0.59810	+0.0035
GAUC	0.6610	0.6651	+0.0041
nDCG@5	0.5282	0.5311	+0.0029

The hidden test was evaluated once.

The current validation artifact reached 0.60541, improving on the organizer baseline by +0.00381.

The final artifact is a fixed 16 seed rank normalized ensemble. Every seed was retained, with no validation based member selection or blend weight tuning.

What Makes It Different

FYPothesis treats evidence as part of the optimization problem.

A single seed can screen an idea, but cannot promote the final submission.

Promising results require confirmation.

Failures become evidence rather than dead ends.

• Runtime failure → recover and retry
• Malformed artifact → reject and repair
• Timeout → pivot to a cheaper experiment
• Repeatedly unproductive approach → abandon
• Promising result → confirm before promotion

The goal is not simply to find the highest number.

The goal is to find a result the agent has good reason to believe.

Built for Trust

Autonomy is only useful if the result can be trusted.

FYPothesis therefore enforces:

• Hidden test labels are mechanically unavailable during research
• Official evaluation code remains unchanged
• Baseline files are integrity checked with SHA256 hashes
• Experiments run in isolated Git worktrees
• Single seed results cannot directly become the final submission
• Research and competition convergence rules are separated

The agent has freedom to explore, but not freedom to compromise the evidence.

Autonomy and Robustness

The latest competition run required 0 manual interventions.

Our full loop recovery evaluation successfully recovered from 3/3 scenarios:

• Runtime failure
• Malformed artifact
• Timeout

Each recovery used the actual agent loop and executor and reached a later scored action.

Resources

The latest autonomous run used:

27 training executions

203,602 provider reported tokens

$0.718218 model spend

2,567 seconds wall clock

0 GPU hours

The system is CPU capable and all resource limits are defined before execution.

How We Built It

Python handles orchestration and subprocess isolation.

NumPy implements the submitted factorization machine.

PyTorch supports optional neural and sequential branches.

OpenAI gpt 5.4 currently drives the research loop through agent/llm.py, with Anthropic supported through the same provider interface.

Streamlit powers the live research dashboard.

Git worktree provides isolated experiment environments.

We deliberately avoided AutoML and hyperparameter optimization frameworks because the search policy itself is the contribution.

What We Learned

The hardest part was not getting an LLM to write experiments.

It was teaching the system when an experiment should be believed.

We learned that:

• A higher score does not automatically mean a better idea.
• Failed experiments can consume resources without producing evidence.
• Repeated validation can turn research into overfitting.
• Ensemble gains must be measured against the mean member, not the best draw.
• Guardrails should preserve autonomy rather than replace it with human approval.

These lessons shaped the confirmation, recovery, and stopping policies at the core of FYPothesis.

Limitations

The hidden test improvement is real but modest at +0.0035.

The model search space remains relatively narrow, and our feature discovery path did not produce a confirmed improvement.

The final ensemble primarily reduces seed variance rather than introducing substantial model diversity.

We did not attempt the optional KuaiRand 1k or KuaiRand 27k benchmarks.

We therefore do not claim that FYPothesis independently discovered a fundamentally new ML algorithm.

We built something more specific:

An autonomous ML researcher that can form hypotheses, write experiments, evaluate evidence, recover from failures, and improve a model under strict resource and evaluation constraints.

Reproduce

From the repository root: https://github.com/Khantminn290/FYPothesis.git

python3 -m agent.baseline_repro
python3 -m agent.final_ensemble --seeds 16
python3 -m agent.verify_incumbent
python3 tests/test_harness.py
python3 -m agent.make_submission --split valid --out submission_valid.csv --score --ensemble
python3 -m agent.recovery_eval
streamlit run app.py