Inspiration

The maximum time bank for 1,000 hands is limited to 8 seconds, so there is a limitation against using solver lookups and deep searches. Knowing this, we wanted to see how well a bot could perform teaching itself from its own hands.

What it does

Plays a head-on short-deck poker Evaluates Equity: Uses a Monte Carlo simulation to estimate chances of winning against a random hand and a range weighted towards better hands. Builds Environment Description: Constructs 50 features including pot odds, stack-to-pot ratio, betting history, board info, hand shape, and a profile of the opponent _ (how often they raise, fold, and what they bet on) _. Evaluates Legal Actions: Using a learned action-value model, the bot predicts how many big blinds each action wins from its current point onward, choosing the best one. Manages Time: Keeps track of how much time it is using, and trims down sampling sizes if it is using too much time to prevent running out of time.

How we built it

Fast Hand Evalutator: Each hand is mapped to keys that can be added together to index precomputed lookup tables. All 100 two-hole card and three-board card combinations are broadcasting using numpy. Preflop Equity is computed for all 19,998 starting hands differing by more than just suit. Deterministic Starting Bot: Plays from manually-tuned thresholds with simplified opponent tracking to act as a baseline and teacher for future models Data Generation using Self-Play: The most current bot is put into a simulation for millions of hands against the baseline bots, some rule variant bots, specific style bots, and then all previous versions of itself. Constant random action 15% of the time to make sure every alternative is checked and none are phased out by earlier weights. Each decision is then labelled by the amount of big blinds won from the decision to the end of the hand. If a hand goes all-in, instead of using random runout we use the exact equity, which helps cut label noise. **Learning with XGBoost:* Uses a two-stage residual model, where the first values the situation and the second small per-action correlation is trained on top. The actual action signal is small compared to the value of the situation, so a split was necessary to make it learnable. Ran through about 14 million hands of training data in total. Numpy Inference: The trees trained are exported to flattened arrays and then walked level by level in numpy, using only about 0.1 ms per decision. This means that the actual submitted bot does not need to use any ML libraries, only numpy and FastAPI. Automated policy iteration: Each round the winner generates fresh data and a new model gets trained, but its only promoted if: It beats the current winner by at least 6,000 duplicates, along with a confidence interval completely above 0. Its performance doesn't get worse against more held-out opponent styles that aren't included in training data. Its thinking time stays within a thinking budget Averaging: The final bot uses the average of two successive models merged into a single forest so there is only the cost of one prediction.

Challenges we ran into

Time Bank: The bot is only given 8 seconds for around 4,000 decisions, so it only has about 1-2 ms per decision. We built an estimate of the charged time, including the server overhead. Testing showed a model with only 60 samples beats a full-strength bot, so if a bot is near running out of time it will keep the model and reduce the number of samples. Measuring Progress Against Noise: Even when both bots are given equivalent hands, it takes around 6,000 deals to measure a difference, so each decision to promote a model needed matches of that size. Subtle Data Bugs: Some opponent bots were slowing themselves down with their time-bank logic during offline-data-generation which made the data non-reproducible. Inference Cost: One model's value part grew to 1,366 trees and pushed past the thinking budget We split the model so value is only evaluated when folding is legal. We rewrote the tree walk to be over twice the speed with identical output.

Accomplishments that we're proud of

The bot plays every test match with zero errors and functions without ever reaching the time bank overhead under normal conditions. Starts within a second and passes all build conditions. The loop of generating, training, and promoting was ran overnight.

What we learned

Model Speed Improvements: Vectorizing the evaluator heavily assisted making decisions within a time bank. Usage of Plain Tools: Using fitted-action learning with Monte-Carlo simulations gave gradient-boosted trees gave a good bot without resorting to deep reinforcement-learning. A key part of this was separating the value of the situation from the value of the action. Evaluation Design: Paired duplicate matches and confidence intervals prevented models from being promoted purely based on luck. Using opponents outside of a training pool ensured that the bots were learning a real strategy instead of just what the training pool was rewarding. Minimizing Samples: Using only 120 Monte Carlo samples produced plays as good as 400 samples

What's next for XGBoost Poker Bot

In-Match Adaptations: Adjusting to opponents within a single match could show improvements over adjustments to average opponent stats. Targeted Searches: Limiting lookahead or solving for highest-stake spots. More Diverse Opponents: Overnight training prevented the bot from playing against other bots at the tournament, so training would likely benefit from having more poker-bots to train against.

Built With

Share this project:

Updates

Submission history