Inspiration
House Edge looked like a standard forecasting problem: predict the odds of a slot machine's next payout (0×, 1×, 2×, 5× or 20×). But the organizers hinted that the machines followed hidden rules: delayed effects, drift over time, and changes shared across machines. So instead of just fitting a model, we set out to reverse-engineer the simulator.
How we built it
Predictions are scored against the simulator's true odds:
$$\text{error} = \sum_{k \in {0,1,2,5,20}} \left(\hat{p}_k - p_k\right)^2$$
normalized against a reference model separately for each of four conditions: stable, short history, drift and shared change.
- Find where the error is. A diagnostic showed almost all of our error came from predicting whether a machine pays, not how much. With the true win chance plugged in, our score would nearly max out.
- Discover the hidden rules. Analyzing machines one by one, we found that each follows its own rules linking its earlier spins to its win chance. Payout sizes, by contrast, are much simpler.
- Model it. We turned each machine's discovered patterns into features for a gradient-boosted tree model, plus a small second model for payout size.
- Ship it reproducibly. The final predictor runs in pure Python with no outside libraries, is deterministic, and reproduces byte-for-byte.
We worked with an AI coding agent that ran experiments overnight under strict acceptance rules we set.
Challenges we faced
- Not fooling ourselves. We held back a hidden slice of validation data and accepted a change only if it improved both our working set and that hidden slice, in every condition.
- Staying within the rules. We caught and threw out an early, high-scoring version that trained on data the rules didn't clearly allow.
- Dropping ideas that didn't hold up. A full simulator of the machines, model blends and calibration all looked promising but didn't survive validation.
- A 90-minute adaptation round with changed rules. We prepared overnight: we built a one-command pipeline and tested it against a fake dataset with deliberately altered rules.
Accomplishments
- Raised our validation score from about 31 to about 54.6 out of 60, beating the reference model in every condition.
- In the adaptation round, the pipeline detected within seconds that the hidden rules had changed, discovered the new ones, retrained automatically, and submitted a safe entry in under a minute, with improved entries as they were verified.
What we learned
- In a problem with hidden state, discovering how the data is generated beats adding model capacity.
- Careful validation matters as much as the model itself.
- Preparing for change ahead of time is what makes a short adaptation round manageable.
Built With
- gradient-boosting
- lightgbm
- machine-learning
- numpy
- python
- time-series
