ByteRush

Inspiration

Imagine five machine learning researchers sharing one goal, one GPU, and one clock. One studies recent user interests. Another tests a ranking loss. A third explores a different model architecture. They start from the same baseline, see the same validation results, and build on what the others discover.

Now imagine that the researchers are AI agents. They write the code, run the experiments, compare the evidence, and decide which direction deserves the next hour of compute. At the end, they do not hand us a list of suggestions. They hand us a frozen model and a checked submission.

That is the idea behind ByteRush. We wanted to turn the repetitive machine learning loop of hypothesis, code, training, evaluation, and reflection into an autonomous research process. KuaiRand-Pure gave us a clear test: improve short-video ranking from the official Factorization Machine baseline while keeping the hidden test set outside the search.

What it does

ByteRush acts like a small recommendation research team. It reproduces the official baseline, then explores features, architectures, losses, and training settings. Our campaign covered FM, MLP, Wide & Deep, DCN, recent-interest encoders, and pairwise ranking.

ByteRush turns each idea into code, checks it, trains it, reads GAUC and nDCG@5, and records the outcome. Strong candidates earn more tuning and seed evaluations. Ablation studies test their new components. The best verified model becomes the incumbent until another candidate earns its place.

The workflow continues all the way to delivery. ByteRush confirms the winner across five seeds, freezes the model and feature state, generates predictions for every hidden-test row, and validates the submission format. The agent uses validation feedback for research but never sees hidden-test labels or scores.

How we built it

We built ByteRush on AI Scientist-v2 and reshaped its tree search into a four-stage recommendation campaign.

Stage 1 reproduces the official FM baseline. Stage 1B opens the field with MLP, Wide & Deep, and DCN. Stage 2 gives each root the same budget and standardized seeds, creating a fair race between model families.

In Stage 3, five agents receive the same incumbent and choose different directions across features, models, and training objectives. Promising ideas receive targeted refinement. Stage 4 tests the strongest candidate's components and runs final multi-seed confirmation.

Every hypothesis stays connected to its code change, metric, and experiment record. A dashboard shows the search tree and current leader. Duplicate detection saves work, file hashes protect the official evaluator, and stopping rules preserve time to package the winner. Configuration keeps the workflow independent of one LLM provider.

Challenges we ran into

The main challenge was giving the agents freedom without losing a fair comparison. A creative idea still had to use the correct data, reach the actual prediction path, and run under the same evaluation rules. We added candidate contracts and smoke tests so the system could check those requirements before spending a full training budget.

Recommendation improvements are also small enough that one run can be misleading. We separated early screening from promotion, checked both ranking metrics, and repeated the finalists across multiple seeds. Long campaigns had to share limited GPU time, so ByteRush learned to stop weak directions and reserve time for final confirmation and submission.

Accomplishments that we're proud of

ByteRush completed the entire journey without handing the research loop back to a human. It reproduced the official FM validation primary score at 0.601655, explored several model families, and selected Wide & Deep as the winner. Across five seeds, the final model reached 0.671028 GAUC, 0.537670 nDCG@5, and 0.604349 primary. Both overall ordering and top-five ranking improved, for an absolute primary gain of 0.002694.

The selected campaign finished in about 89 minutes, including 25 counted iterations and 17 seed evaluations. ByteRush explored several directions, spent more effort on promising candidates, and delivered a frozen checkpoint plus a validated submission for all 170,588 hidden-test rows.

The winning source, seed results, checkpoint, code changes, and resource record can all be inspected. ByteRush also ran on KuaiRand-1K and KuaiRand-27K, where it kept the incumbent when no candidate earned promotion. A trustworthy decision mattered more than forcing a new winner.

What we learned

An autonomous researcher needs imagination and restraint. ByteRush became stronger when it could explore several directions, remember the outcomes, and give more attention to evidence that held up.

We also learned that a simple model can be the right answer. Wide & Deep won because it performed well under the same rules, survived repeated seeds, and produced a complete submission. We could follow the whole decision: an AI system explored, tested, compared, and delivered a result with a clear history.

What's next for Untitled

Next, we want ByteRush to carry its memory across longer campaigns. Durable checkpoints will let an interrupted run resume with its candidates, code versions, data hashes, and seed results intact. Clearer feature lineage will show how each proposed idea travels from the dataset into the final prediction.

We also plan to add an independent confirmation slice for finalists and make compute allocation more adaptive. With those foundations, ByteRush can enter new recommendation tasks, establish a baseline, organize its own research campaign, and leave behind a model and experiment story that another engineer can reproduce.