House Edge - datajohn

Inspiration

Slot machines look random, but the House Edge challenge hinted that each machine hides a rhythm. We wanted to see how much of that rhythm we could recover from nothing but spin logs, and whether we could forecast honest probabilities instead of just guessing outcomes.

What it does

For every machine, our model predicts the probability of each payout (0×, 1×, 2×, 5×, 20×) on the next spin. It learns from the machine's own recent spins, from machines of the same type, and from what is happening across the whole episode, and it keeps adapting as machines drift or change together.

How we built it

The score compares our squared probability error to a frozen reference model across four conditions (stable, short history, drift, shared change):

$$R = \frac{1}{4}\sum_{c} \frac{E_c}{B_c}, \qquad \text{points} = \frac{\text{max}}{1 + R}$$

So every condition counts equally, and beating the reference everywhere matters more than winning big in one place.

Our model is an online mixture of experts, written in pure standard-library Python:

Frequency experts with different memories (short, medium, all-time), smoothed toward a prior learned from the training machines of the same type and signal.
Pattern experts keyed on pairs of earlier spins. Mining the training data showed that machines follow one of three lag pairs, $ (1,3) $, $ (2,5) $ or $ (4,7) $, and that what matters is whether those two spins match (both lost or both paid). That interaction moves $ P(0) $ by about 0.40, versus about 0.10 for either spin alone.
An episode expert that notices when all machines in an episode shift together.
Hedge weighting: each expert's weight is $ w_i \propto e^{-\eta L_i} $, where $ L_i $ is its recency-weighted Brier loss on "zero vs. non-zero", so the model trusts whichever expert has been right lately.
A win-size split: the non-zero probability is divided across 1×/2×/5×/20× using a blend of the machine's long-run mix and its type prior, followed by a light calibration of $ P(0) $.

We went from 49.30 to 53.72 / 60 on official validation through five versions, keeping only changes that improved both halves of a condition-stratified split of the validation episodes.

Challenges we ran into

Overfitting to validation. With many ideas to try, it's easy to fool yourself. We chose settings on one half of the validation episodes, confirmed them on the other half, and used paired bootstrap confidence intervals. Many ideas that looked good on the first half died on the second, including a learned stacker, Bayesian log-loss weighting, trend experts and joint patterns.
Win sizes. Error analysis showed about half of our remaining error comes from how wins split across sizes. Each machine has its own split, and recent spins barely predict it.
Reproducibility. The organizers re-run the code offline and require byte-for-byte matching output. We stress-tested shuffled rows, different hash seeds, injected "future" spins and a read-only code folder (14/14 passed). ## Accomplishments that we're proud of
53.72 / 60 on official validation with a model that runs in a couple of minutes on one CPU core, with no external libraries.
An adaptation runner for the 90-minute second round. It immediately packages a safe baseline, then tests about 25 predeclared variants and switches only if a variant wins on both validation halves. We rehearsed it on synthetic worlds unlike the Main data (no pattern, streaks, three-spin patterns, machines influencing each other), and it recovered up to +12 points over the baseline. ## Round two (Adaptation) The new kit changed the hidden patterns: our model learned from the new training data that machines now follow lag pairs $ (1,4) $, $ (3,8) $ and $ (2,6) $ instead of the Main round's pairs. Our unchanged Main model already scored 18.00 / 20 on the new validation set. We then:
averaged three settings of our model in log-odds space (18.08), and
built a two-stage model that predicts whether a spin wins separately from how big the win is, averaging over 65 candidate lag rules weighted by each machine's own history. Its win-size split beat ours, so the final entry combines our ensemble's win chance with the two-stage size split: 18.16 / 20 on adaptation validation, better on both validation halves. ## What we learned
Probability forecasting rewards calibration and humility. Shrinking toward a sensible prior often beats a clever model.
Validation discipline (split halves, bootstrap CIs, predeclared candidates) is what separates a real gain from noise.
Structure in data is worth finding: one discovered interaction gave us more than dozens of hyperparameter tweaks. ## What's next for Head Edge Better per-machine win-size estimates, and a more principled way to detect which kind of structure a new dataset has before choosing a model.