Inspiration
honestly we just wanted to see if we could build a poker bot that actually plays well and not just one that folds bad hands and calls good ones. short deck 5 card PLO is a weird game, flush beats full house, trips beat a straight, and you have to use exactly 2 of your 5 hole cards. none of the normal poker intuition really works here, so we thought ok lets not hand write rules, lets make the bot learn the game by playing against itself.
How we built it
the core of the bot is CFR (counterfactual regret minimization), the same family of algorithms behind the poker AIs that beat pros. the idea is simple: play the game against yourself millions of times, and for every decision track how much you regret not taking each action. over time the average strategy converges to something close to a Nash equilibrium, i.e. a strategy that is very hard to exploit.
for an action $a$ at an information set $I$, the regret after $T$ iterations is
$$R^T(I,a) = \sum_{t=1}^{T} \left( v^t(I,a) - v^t(I) \right)$$
and the next strategy is just regret matching:
$$\sigma^{T+1}(I,a) = \frac{\max(R^T(I,a),0)}{\sum_{b}\max(R^T(I,b),0)}$$
some things we did on top of that:
- exact betting tree built straight from the game rules, 678 decision nodes, so the bot always knows the exact pot and sequence
- card abstraction: you cant store a strategy for every possible hand, so we group hands into buckets using monte carlo equity plus features like made hand strength, draws, blockers, board texture and what happened on the previous street
- external sampling MCCFR in numba so training is fast, and we trained on TAMU's Grace HPRC cluster with 48 cores
- hardened training: we co-trained against a stronger adversary bot so our strategy doesnt have obvious holes, then ran a separate exploiter bot against it to measure how exploitable it still is
- opponent adaptation: the bot tracks how the opponent plays (how often they fold to bets on each street, how often they bluff the river, etc) and only switches to an exploit when its ~99% confident the opponent is outside the normal range. so against balanced bots it stays balanced, against over folders it starts applying pressure
- time bank safety net: we get 8 seconds of thinking per 1000 hands, so if our main model ever uses more than 5s it falls back to a cheaper model. we never want to forfeit a match because of time
we went through a lot of versions (alpha, beta, gamma ... zeta, hardened, theta) and the final one is theta, the one with the richer card features and hardened training. it beat or tied every older version head to head.
How we tested it
this was honestly the part we spent the most time on. we built our own harness that runs matches exactly like the tournament: official judge, 1000 duplicate deals × 2 halves, fresh server for every half, 95% confidence interval for the verdict.
then we ran it against every public bot from other teams we could find on github, plus 25 practice bots with different styles (maniacs, calling stations, nits...). on grace we ran hundreds of live matches and full round robin tournaments. every change had to pass a paired A/B test on the same seeds before we kept it, and a lot of "smart" ideas got rejected because the numbers said no.
Challenges
- the variance is brutal. two good bots almost always chop because the confidence interval touches 0. you can be +0.2 bb/deal better and still not get the win, so we had to learn to trust large samples instead of one match
- bugs that look like strategy. at one point our river fold rate stats were reading ~100% because we never counted river calls, so our exploit was firing against the wrong bot and losing. caught it only because we tested against real bots
- compute limits. 1 CPU, 8s per half, no internet. startup was ~5s on one core at first so we precomputed tables to get it to ~1s
- time. we always wanted to retrain one more round but at some point you have to stop and submit what you tested
What we learned
- equilibrium strategies are great for not losing, but to score more points you also need to exploit weaker opponents harder
- test everything against real opponents, your own bots will lie to you
- statistics matter as much as poker, paired tests and confidence intervals saved us from shipping a lot of bad ideas
- a bot that never crashes is worth more than a clever bot that sometimes does. we finished with 0 errors and 0 forfeits in every test
Whats next
we want to train specific counter strategies for different player types, learn opponent ranges from showdowns, and use more exact hand strength tables instead of sampling, so our buckets are less noisy.
