Workflow-Canvas-UI
UI-Experiments-Table
MCP Architecture Diagram
FlowState Architecture Diagram

Executive Summary

FlowState is an autonomous, self-healing ML research engineer engineered for the TikTok TechJam 2026 KuaiRand-Pure challenge that automates the entire hypothesis to submission lifecycle without human intervention. Orchestrated via LangGraph state machines and powered by Azure AI Foundry agents (gpt-5.6-terra and gpt-5.6-sol), FlowState dynamically retrieves academic literature via an MCP knowledge layer, issues typed contracts, commits candidate architectures inside isolated Git worktrees, and gates them through multi-stage proxy and evaluation suites. By coupling an append-only, tamper-proof SQLite ledger with strict data leakage controls and failure self-healing loops, the system systematically beat the official Factorization Machine baseline, reaching a primary score of 0.6043 with an MMoE architecture, while ensuring complete auditability, budget enforcement, and zero leakage reproducibility across every research cycle.

Inspiration

An ML engineer today spends hours inspecting feature distributions, writing pipeline modifications, parsing validation outputs, and debugging runtime crashes. While code-generating LLMs can draft individual snippets, true breakthrough requires a system that can orchestrate this entire cycle fully autonomously with scientific discipline.

That vision inspired FlowState: a fully autonomous, self-healing ML research agent purpose-built for the TikTok TechJam 2026 Track #2 KuaiRand-Pure challenge. Rather than functioning as a lightweight wrapper or simple hyperparameter tuner, FlowState drives the complete machine learning engineering loop, moving from exploratory data analysis to baseline-beating pipeline architectures without requiring human intervention.

Why FlowState Stands Out

Uncompromising Autonomy: Automates the complete ML stack by drawing on academic literature and industry methods to propose hypotheses, modify source code, train candidate models, and reflect on outcomes independently.
Resilient Self-Healing: Engineered with robust error-handling loops that catch code exceptions, handle unexpected data inputs, and repair broken iterations to keep long research runs executing reliably without crashing or diverging.
Transparent Provenance: Preserves full inspectability across the entire evolutionary tree by logging every hypothesis, code diff, metric shift, and recovery action.

Autonomy is only as valuable as its reliability: FlowState transforms open-ended LLM capabilities into a disciplined, self-correcting research engine where every decision is backed by evidence and every result is reproducible.

What it does

FlowState acts as an autonomous research engineer for short-video recommendation experiments. It coordinates a multi-stage workflow:

A. The DATA Engine

Locks the dataset and official challenge contract.
Profiles the data and checks split integrity.
Reproduces the official Factorization Machine baseline.
Runs evaluator and label-shuffle sanity checks.

B. The RESEARCHER (Agent)

Retrieves supporting and contradicting research evidence.
Asks a research agent to preregister one bounded hypothesis.

C. The CODER (Agent + Engine)

Implements the bounded hypothesis's code change inside an isolated Git worktree.
Runs static, smoke, proxy, and full-training gates.
Evaluates predictions with the organizer's official evaluator.
Accepts, rejects, retries, or rolls back the experiment.
Repeats until convergence or resource-budget exhaustion.
Freezes the validation-best run and creates a checked submission package.

D. The CHATBOT (Agent)

Learns from your entire session history and codebase to answer your doubts at any time of the session.

The target is the binary long_view signal. FlowState evaluates candidates using GAUC and nDCG@5, with the primary score defined as:

$$\text{Primary} = \frac{\text{GAUC} + \text{nDCG@5}}{2}$$

Because evaluation happens within each user, the system focuses on whether positive videos are ranked above negative ones, not on retrieving videos from the entire catalog.

FlowState also includes a live web interface called the Live Workflow. It presents the end-to-end process as an interactive pipeline and provides dedicated views for:

Live workflow execution
Data profiles
Experiment results
Retrieved research
Compute and token consumption
The autonomy and decision log
Final submission artifacts

The interface consumes real workflow events through Server-Sent Events, so users can follow running, successful, rejected, failed, skipped, waiting, and recovery states without relying on external datasets.

How we built it

We implemented the backend in Python 3.12 using LangGraph to model the autonomous engineering loop as a persistent, fault-tolerant state graph. The architecture coordinates specialized modules: a Polars-driven data engine that profiles splits and reproduces official baselines, an MCP knowledge layer retrieving curated and dynamic literature from Hugging Face and GitHub with reciprocal-rank fusion, and Azure AI Foundry agents functioning as dedicated researchers and coders. Every iteration is bound by a strict typed Pydantic contract declaring explicit hypotheses, falsification criteria, permitted file scopes, and compute budgets before any code generation begins.

The Coding Agent translates approved hypotheses into unified Git diff patches (PatchProposal), executing modifications strictly within isolated Git worktrees against hash-locked starter-kit files. To prevent silent regressions and resource exhaustion, patches must pass an execution funnel comprising AST syntax verification, inert-patch detection, smoke checks, and a filter-only proxy gate before reaching full GPU training. Candidate recommendation architectures, including DeepFM, DCNv2, DIN, and multi-task MMoE, are evaluated directly by KuaiRand's official within-user GAUC and nDCG@5 harness, where negative controls (like label-shuffling) ensure ranking validity.

All workflow transitions, metric receipts, unified diffs, GPU memory spikes, and self-healing recovery loops are recorded to an append-only, hash-linked SQLite ledger. A FastAPI backend streams live execution states, token burn rates, and artifact manifests to a React 18 / TypeScript frontend via Server-Sent Events (SSE). The session culminates in an atomic, one-way submission finalizer that exports verified prediction files and cryptographic run manifests.

Tools, APIs, Libraries, Frameworks, and Data

Development Tools

Developed with Python 3.12 and uv package management alongside Node.js, Vite, and TypeScript, backed by isolated Git worktrees for run isolation, SQLite for the local append-only ledger, OpenSSL for .env.enc secret encryption, and tested via pytest, pytest-asyncio, Vitest, and React Testing Library.

APIs and Services

Uses Microsoft Azure AI Foundry (gpt-5.6-terra for research/chat, gpt-5.6-sol for coding/recovery) via the Responses API, integrates Hugging Face Papers and GitHub REST APIs through an stdio Model Context Protocol (MCP) server, and streams live telemetry to the UI using a custom FastAPI service with Server-Sent Events (SSE).

Libraries and Frameworks

Orchestrated via LangGraph, LangChain Core, and langchain-azure-ai, configured with Pydantic and PyYAML, driven by PyTorch for deep recommendation models, optimized with Polars, NumPy, and PyArrow for tabular processing, and embedded via FastEmbed with sentence-transformers/all-MiniLM-L6-v2.

Datasets and Assets

Evaluates on the KuaiRand-Pure benchmark using official starter kit assets (data.py, evaluate.py, baseline.py), official temporal splits targeting the binary long_view label, a curated 25-paper offline rec-sys knowledge bank, and reproducible run manifests.

Summary of Services Used

Category	Tools, Frameworks & APIs	Role / Primary Purpose
Development Tools & Infra	Python 3.12, Node.js, Vite, TypeScript, `uv`	Backend runtime, package management, and frontend build tooling.
	Git & Isolated Git Worktrees	Version control and reproducible code branch isolation for autonomous experiments.
	SQLite	Local append-only ledger for workflow state, experiment logs, and research cards.
	OpenSSL	Encryption utility for secure `.env.enc` password-protected secrets management.
	`pytest`, `pytest-asyncio`, Vitest, React Testing Library, JSDOM	Unit testing and async validation for backend services and React components.
LLMs & External APIs	Azure AI Foundry (`gpt-5.6-terra`, `gpt-5.6-sol`)	Agentic reasoning engines for Research, Coding/Recovery, and Chat via Responses API.
	Hugging Face Papers API & GitHub REST API	Automated retrieval of external research papers, implementation repos, and commit histories.
	Model Context Protocol (MCP)	Local stdio server exposing bounded tools (evidence retrieval, paper search, research cards).
	FastAPI & Server-Sent Events (SSE)	Backend web service exposing workflow state and real-time event streams to the UI.
Libraries & Frameworks	LangGraph, LangChain Core, `langchain-azure-ai`	Stateful agent orchestration graph and LLM integration framework.
	Pydantic, PyYAML, `python-dotenv`, `httpx`	Typed contracts, configuration management, environment loading, and async HTTP.
	PyTorch, NumPy, Polars, PyArrow	Model architectures (FM, DeepFM, DCNv2, DIN), numerical processing, and tabular profiling.
	FastEmbed (`sentence-transformers/all-MiniLM-L6-v2`)	Local vector embeddings for semantic search over cached research literature.
	`psutil`, `nvidia-ml-py`	Real-time system monitoring, GPU tracking, and experiment resource logging.
	React 18, React Router, Zustand	Single-page application UI for real-time visual tracking of autonomous agent runs.
Datasets & Assets	KuaiRand-Pure Benchmark Dataset	Short-video recommendation dataset (1.4M rows, temporal split, `long_view` target).
	KuaiRand Starter Kit Assets	Official reference baseline, data loaders, and model-agnostic `evaluate.py` harness.
	Offline Knowledge Bank & Ledger	Curated 25-paper rec-sys literature bank, dynamic HF/GitHub cards, and run manifests.

Resources

Cumulative usage since the first agent action, tracked alongside the metric score:

Metric	Value
LLM tokens	460,914 in / 305,651 out (766,565 total)
GPU / CPU wall time	0h 20m
GPU-hours	0.2085
Peak RSS memory	4019 MB
Recovery retries	1
Manual interventions	0

Challenges we ran into

The primary obstacle was ensuring autonomous safety and resilience across unpredictable LLM failures. Models frequently produced syntactically correct code that violated experiment scopes, missed training-path connections, or introduced inert adjustments. To mitigate this, we enforced strict patch-path constraints, isolated Git worktrees, and added proxy comparisons to filter inert or regressive changes before committing to expensive full runs. Failures required granular attribution rather than naive retries, allowing FlowState to diagnose syntax errors, malformed diffs, or missing citations and dynamically route actions between targeted code fixes, hypothesis resets, or fallback restorations.

Token consumption and evidence routing also posed significant operational bottlenecks. Early prototypes on AWS Bedrock incurred escalating token overhead across multi-agent loops. Moving to Microsoft Azure AI Foundry gave us tighter model governance, complemented by bounded prompts, structured Pydantic outputs, and strict per-stage token and wall-time budgets. Concurrently, dynamic MCP retrieval initially over-indexed on the offline paper collection. We balanced discovery by enforcing a 40/60 weighted split between curated literature and live GitHub/Hugging Face findings, ranking them via reciprocal-rank fusion to maintain novelty without sacrificing grounding.

Safeguarding data integrity and establishing valid evaluation thresholds were critical throughout the temporal benchmark. Repeated single-model loss tuning quickly hit a performance plateau, pushing us to enable strict multi-model rank ensembling only after candidates passed diverse qualification gates. Because KuaiRand relies on temporal splits, we strictly ring-fenced chronological histories, split dates, and starter-kit evaluation hashes to prevent data leakage. Finally, since metric nuances meant GAUC omitted single-class users while nDCG@5 penalized all-negative sets, we implemented metamorphic tests and label-shuffled controls to confirm model improvements reflected genuine ranking power rather than baseline noise, verifying that shuffled labels performed no better than random guessing.

Accomplishments that we're proud of

We were able to pass the Official FM validation baseline! Details of our best experiment that passed the baseline:

Metric / Attribute	Value / Details
Model / Architecture	MMoE (Multi-gate Mixture-of-Experts) • Two shared 32-dimensional experts with task-specific soft gates • Two separate BCE heads for `long_view` and `is_click` • Categorical inputs: `user_id`, `video_id`, `author_id`, `tab`, `dur_bucket` • Inference: Emits only the `long_view`-head score from permitted features
GAUC	0.6711
nDCG@5	0.5376
Primary Score	0.6043
Research Agent Hypothesis	The accepted parent is B0. The only completed alternative, FM user-group BPR, reached primary 0.6031105 (+0.0015384 versus B0), below the required 0.002 improvement, and regressed after epoch 6; its mechanism is forbidden. The attempted SASRec mechanism failed and is also forbidden. A non-FM family is mandatory. The profile provides a materially different, bounded signal: `is_click` has 0.7596456 correlation with `long_view`, and MMoE is directly supported by `mmoe-kdd2018` for learning task-specific use of shared experts. A shallow two-expert static-input MMoE is feasible on the 100,000-row CUDA proxy and requires no chronological-history implementation. Change: Replace the static single-task scorer with a compact two-task MMoE using the fixed categorical inputs `user_id`, `video_id`, `author_id`, `tab`, and `dur_bucket`. Train two shared 32-dimensional experts with task-specific soft gates and separate `long_view` and `is_click` BCE heads, optimizing BCE(long_view) + 0.25 * BCE(is_click). Emit only the `long_view`-head score from the five permitted inference features.

We are especially proud that FlowState treats unsuccessful experiments as useful research evidence rather than disposable output. Rejected and failed candidates remain visible in the append-only ledger alongside their hypothesis, patch, failure stage, resource use, and recovery action.

Other accomplishments include:

Reproducing the official five-seed FM validation reference of approximately 0.6016
Building an official-metric evaluation path for GAUC and nDCG@5
Adding evaluator metamorphic tests and an in-workflow label-shuffle control
Implementing strict train, validation, and sealed-test boundaries
Creating evidence-grounded experiment contracts
Detecting research citations that were not supplied by the knowledge layer
Isolating generated changes in Git worktrees
Supporting FM, DeepFM, DCNv2, and DIN model families
Tracking separate validation and innovation frontiers
Measuring wall time, GPU usage, and agent token consumption
Implementing pause, resume, cancellation, recovery, and reconnectable live events
Creating a one-way, explicitly confirmed finalization process
Producing schema-checked predictions.csv and reproducibility manifests
Building a multi-page observer interface backed by real workflow data
Developing a repository test suite containing more than 140 automated test cases

We are also proud of what the system learned not to repeat. Experiments based on small changes to the same FM loss remained close to the baseline, while increasing latent capacity and adding coarse static user features showed little benefit. FlowState records those outcomes so future iterations can move toward genuinely different mechanisms instead of endlessly retuning the same idea.

What we learned

We learned that a recommendation metric must shape the entire research process, not only the final evaluator. Since the challenge ranks candidates within each user, features that are constant for a user cannot change that user's ordering. Likewise, optimizing independent binary predictions is not necessarily the best match for groupwise GAUC and top-heavy nDCG@5.

We also learned that experiment diversity matters more than generating a large number of minor variations. Several loss adjustments such as focal loss, label smoothing, simple recency weighting, and auxiliary labels attached to the same FM score produced nearly identical outcomes. This indicated that the bottleneck was more likely the representation of user history and candidate relationships than another small loss-function adjustment.

The project reinforced several engineering lessons:

An agent's output should be treated as untrusted input.
A successful process exit does not prove that a code change affected the model.
Cheap proxy runs are excellent rejection tools but poor selection tools.
Research evidence should include contradicting findings, not only supporting papers.
Failures need structured attribution to make retries productive.
Reproducibility requires preserving code, data, metrics, and decisions together.
Observability is part of autonomous-system safety, not merely a UI feature.
A final submission should be a one-way, auditable transition rather than another ordinary workflow step.

What's next for FlowState

The next modeling milestone is to move beyond small FM loss modifications and test mechanisms that better match the structure of the benchmark.

Our planned sequence is:

Evaluate a rank-averaged ensemble of the five reproducible FM seeds.
Add leakage-safe features derived from each user's past behavior.
Train with pairwise or grouped ranking objectives aligned with within-user ordering.
Expand the DIN implementation for candidate-aware user-history modeling.
Add separate heads for click, long view, and watch completion.
Compare DeepFM and DCNv2 after richer history features are available.
Ensemble structurally different models using within-user ranks.
Explore the randomized exposure log for off-policy bias correction.

On the platform side, we plan to strengthen experiment replay, expand the research evidence bank, improve hardware-aware scheduling, and add richer comparisons between accepted, rejected, and failed runs.

Ultimately, we want FlowState to generalize beyond KuaiRand. The long-term vision is an autonomous research environment where challenge contracts, official metrics, integrity rules, budgets, and finalization policies can be swapped for a new dataset while preserving the same evidence-grounded and reproducible scientific workflow.

Important Notes

In order to run the full workflow on the FlowState interface, follow the steps in README.md in the GitHub repo to open it locally.
The FlowState Demo Simulation is a data-free simulation showing the completed metrics and audit trail from the best FlowState session on the KuaiRand-Pure dataset.
Please refer to the uploaded Results.zip to access all the results for our best session.