Hackathon 1

Inspiration

The organizers of the KuaiRand-Pure task published a baseline, a list of things to try. There were two dead ends with numbers. More features made it worse. A bigger model changed nothing. Someone had already spent the obvious ideas and written down that they failed.

So the interesting question wasn't "can a model do better." It was can an agent do research? Which is form a hypothesis, say what would refute it, run it, and believe the answer even when the answer is no.

What it does

How we built it

An agent that improves a ranking pipeline with no human in the loop. Each iteration it proposes three falsifiable hypotheses, picks one, writes the code, runs it on matched seeds, and reverts anything that doesn't clear a noise-calibrated bar.

Every step lands in a hash-chained journal. verify.py proves the final choice was sealed before any test row was read so "we only tuned on validation" is checkable, not a promise.

Challenges we ran into

Nine independently written implementations of the same idea — a within-user listwise softmax — scored anywhere from −0.0032 to +0.0016. Seven lost. One won and became our submission.

The spread across implementations of one label was 5× wider than the spread across random seeds. Our agent's memory recorded one verdict per idea, so two early failures closed the direction three runs before the version that actually worked turned up.

Accomplishments that we're proud of

What's next for Hackathon 1

The same ablation showed the batching those losses require costs −0.0024 on its own. Our gain is a big positive minus a big negative, and the negative is still there to claim.

Built With

• anthropic
• autonomous-agents
• claude
• claude-opus-4.5
• factorization-machines
• information-retrieval
• learning-to-rank
• listwise-ranking
• llm-agents
• machine-learning
• numpy
• prompt-caching
• pytest
• python
• recommender-systems