Inspiration
Heads-up Short Deck PLO5 is hard for computers: five hole cards, a 36-card deck and pot-limit betting. Strong human players beat each other with judgment, not lookup tables. We wanted to see how far a small team could get in 24 hours by stacking cheap, explainable rules with learned models, while staying inside tight limits: 1 CPU, 8 s of thinking per 1000 hands, and 1 s per decision.
What it does
Patrick Jane is a poker bot that makes each decision in four layers:
- Rules: Monte Carlo hand equity plus pot odds give an expected value for each action. An opponent model adjusts for how often the opponent raises and folds.
- Q-net: a neural network trained on 23.7M self-play decisions predicts the long-run value of each action. Its weight fades to zero against hyper-aggressive opponents it never saw in training.
- Learned correction: a ridge + 5-MLP ensemble trained on Databricks fixes specific mistakes, such as folding too often to turn and river raises.
- Mixed strategy: a softmax over the scores picks the action at random, so opponent-modelling bots can't read us.
If any model fails, the bot falls back to the layer below, so it never errors or forfeits.
How we built it
- Engine: Python and numpy only. Equity is vectorised with numpy, and every setting lives in
params.json. - Tuning: we tuned the parameters with self-play against a pool of 15 opponents: the baselines, 4 stronger hand-written bots, past versions and style variants. A change was accepted only if it beat the current bot on held-out seeds, judged by its 95% confidence interval.
- Databricks:
- Spark generated 205k decisions. Each decision was replayed with every legal action forced and then played out 8 times, which gives a label for each action.
- We trained the ensemble on these labels, tracked runs in MLflow and stored the data in Delta tables.
- We exported the model to numpy with bit-exact parity.
- Evaluation: a local copy of the tournament judge: 1000 duplicate deals × 2 halves on fresh servers, with the official time and error limits. The final image passed a Docker check at 1 CPU, 4 GB, with no network.
Challenges we ran into
- Noise: equally strong bots chop about 90% of matches, so telling real gains from luck took thousands of deals and strict statistics.
- Training data vs live play: equity sampling at match time differed from sampling during training until we fixed the sample count.
- Generalisation: a model that beat our pool sometimes lost to unseen styles, especially passive ones.
- Not every model helped: a Databricks equity model was as accurate as Monte Carlo but gained nothing. Round-2 retraining lost to round 1 head-to-head, so we rejected it.
- Time limits: three models plus Monte Carlo had to fit in milliseconds per decision on 1 CPU.
Accomplishments that we're proud of
- The final bot beat every earlier version. It won 25 of 25 matches against both the rules-only bot and the ensemble bot. It went unbeaten in 50 matches against a teammate's Q-net bot (2 wins, 48 chops, +0.25 bb/deal).
- It is reliable: 0 errors in hundreds of matches, and at most 3.4 s of the 8 s thinking budget.
- It is evidence-driven: every change passed a held-out accept test, and we rejected even our own Databricks work when it didn't win.
- Four teammates' work combined into one bot: rules, Q-net and ensemble, with each layer's gain measured.
What we learned
- Decision quality mattered more than hand-strength accuracy.
- Ensembles of simple models, layered on safe rules, beat any single model.
- Consistency between training data and live play matters as much as the model itself.
- Duplicate deals and confidence intervals are what make poker results trustworthy.
What's next for Heads Up
- Switch by opponent type: use separate exploit settings for stations, maniacs and nits, chosen live from the opponent's stats.
- League training: co-train the Q-net and the correction layer against each other and against a growing pool of past versions.
- Lower variance: turn more chops into wins.
- Wider testing: test against more varied unseen bots before the knockout.
