Inspiration
The House Edge challenge asks for a probability for each payout a machine might give on its next spin, scored against a pooled reference. We wanted to see how far careful probability modeling could get beyond "every machine is a coin with a fixed win rate."
What it does
House Edge predicts a full probability distribution over a machine's next payout (0, 1, 2, 5 or 20), using only that machine's own history before the cutoff.
- Win chance: the model looks at whether two of the machine's earlier spins, at a specific pair of lags, agreed or disagreed. That pattern moves the win chance between about 3% and 97%.
- Pattern flips: under drift or a shared change, the pattern can reverse. A two-state hidden Markov model tracks those flips as they happen.
- Payout size: each machine type has two payout-size profiles. The model estimates which profile is active from recent winning payouts, giving more weight to recent ticks.
It scores 55.60 out of 60 on validation, against 30 for the pooled reference.
How we built it
- We started from a blend of a neighbor model and a pooled model, which scored 30.91. Then we added pieces one at a time and kept each one only if validation improved:
- best lag pair, static: 32.20
- recency weighting: 46.94
- change-point detection: 53.64
- two payout-size profiles: 54.16
- a flip HMM in place of change points: 54.92
- a rate prior for each machine type: 55.40
- a full 2×2 model averaged 50/50 with the 2-cell model: 55.60
- Within each model family, Bayesian model averaging combines the candidate lag pairs with a no-lag baseline, weighted by each machine's own history.
- It's pure Python and NumPy, runs on a single CPU core in about a minute, and makes no random choices. Refitting gives a byte-identical model file.
Challenges we ran into
- Change versus noise: short runs of losses look like a regime change, so early change-point models either reacted too slowly or flipped too often. Replacing them with an HMM fixed most of that.
- Short histories: some machines have very little data. Empirical-Bayes priors for each machine type let them borrow strength from similar machines.
- Missing ticks: gaps in a machine's history have to stay unobserved rather than count as losses, which complicates the lag features.
- Not overfitting the leaderboard: with this many design choices, it was easy to tune to validation, so we audited the model before submitting.
Accomplishments that we're proud of
- Taking the score from 30.91 to 55.60 out of 60, with error about 93% lower than the pooled reference in every scenario: stable, short history, drift and shared change.
- An overfitting audit we trust:
- fake future ticks and scrambled neighbor machines change predictions by exactly 0, so there's no leakage
- the model is keyed only by machine type, so it can't memorize machine IDs
- 4-fold cross-validation inside train selected the same lag pairs every time
- A fully reproducible entry. A fresh rebuild from the ZIP matches the submitted CSV exactly.
What we learned
- Look at the data before reaching for bigger models. The largest jumps came from understanding how the machines behave, not from adding model capacity.
- Probabilistic structure (hidden states, priors and model averaging) beats ad hoc rules when the behavior can change.
- Calibration and reproducibility deserve as much attention as raw accuracy.
What's next for House Edge
- Round two: a single-command pipeline refits on the new data across several flip rates and model families. It keeps the main-round model if that still scores best, then rebuilds and verifies the submission automatically.
- Payout sizes: guessing which size profile is active is now the largest remaining source of error, so better ways to tell the profiles apart are the next target.
- More scenarios: testing against more synthetic setups to make sure the model holds up beyond the scenarios it was tuned on.
