HeatCheck
Inspiration. For thirty years the basketball "hot hand" was the textbook example of humans seeing patterns in noise, until 2018, when two economists found a counting bias in the original 1985 study and the effect came back. If careful people with one hypothesis get it wrong over decades, an AI testing a thousand ideas an hour with nobody checking will manufacture discoveries on demand. Brevan Howard's brief asked for an AI researcher that can tell a real discovery from a lucky one. We decided the missing piece is not a smarter pattern-finder but a referee the pattern-finder cannot argue with.
What it does. Three characters. Heater, a loud AI pundit (Meta muse-spark-1.3), takes your question by voice or text and turns it into a claim that is locked before any data is opened. Frost, a calm challenger, asks up to three questions a sceptical quant would ask, each anchored to a check that actually ran: would shuffled data show the same, is the sample big enough, did you pick the shots that suit you. The Ref is plain code: it applies the rule registered in advance, corrects for every idea tried (Holm), and stamps one of five verdicts. Nothing Heater says can change a verdict. Every ruling goes into an append-only, hash-chained evidence ledger; flip one byte and the next read reports the break. A read-only board compares the model's forecast to Polymarket prices, labelled "Edge unverified" because nothing has been graded yet.
How we built it. Python 3.12/3.13, pure statistical kernels with no I/O, a custodian that seals seasons and binds every run to data hashes, strict-grammar JSON for the model's proposals, a number guard that rejects any figure in a model line not present in the record, and browser speech synthesis for three voices. NBA play-by-play 2023–26 pinned at one commit; current season via balldontlie; Polymarket and Kalshi read-only. 860+ tests, CI on every push. Four Claude sessions on separate machines with fixed roles (integration, research, experience, review), and an independent reviewer who wrote no code.
The result. Three claims were written and locked on 2023–24 data. The whole 2024–25 season (1,224 games, 219,274 shots) was sealed, then opened once, on written authorisation, by a session that wrote none of the code. Home advantage: SURVIVED (54.5%, at least 1.6 points above even). Momentum and the hot hand: INSUFFICIENT EVIDENCE (47.27% after three makes, 47.27% after three misses). The Ref refused to upgrade a null to "disproved" because no zero-effect test was registered. Earlier, on the development split, a rest effect came back at $p = 0.049$ and the Ref refused it: the pre-registered rule required the confidence interval's lower bound above zero, and it was $-0.0043$. Every one of us would have kept that result.
Challenges. Keeping an LLM from leaking numbers or verdicts into its banter (guard plus template fallback). Making Frost's questions honest: each must map to a check that ran, not to the model's opinion. The reviewer forged 40 results and found the custodian accepted them as a "retry"; fixed and re-verified on a separate build before the sealed run. Testing the tester on synthetic leagues with planted truth: 4.3% false positives at the 5% level, and the hot-hand test also fires on within-game form drift, so a real rejection shows association, not cause. Four parallel sessions meant schema drift, port clashes and one laptop failure at 3am.
What we learned. The scarce thing is not finding patterns, it is a process that can say no and cannot be edited afterwards. Pre-registration, sealed data, multiplicity correction and an immutable record are cheap to build and almost never built. Our own forecaster audit (it overrates home teams by 6.9 points on a held-out season) is printed on every market card where it applies.
Next. ElevenLabs voices, WNBA and NCAA with the same claims and a new seal, football and factor claims through the same registry, graded forecasts so "Edge unverified" becomes a number with a history, and a memorisation audit of the model against the sealed season.
