-
-
Bots find the broken hero, AI proposes the fix, fresh seeds prove it.
-
Scan the QR: your phone becomes the controller.
-
Every patch is a version: play it, test it, continue from it.
-
The dashboard on a phone.
-
Gemini patch re-tested on fresh seeds: 80% to 56%, accepted.
-
Same pipeline, card game: 36% to 55.5% after one card-cost change.
-
Real Armor vs Bari, live on the big screen.
-
First fall ends the race: each phone shows only YOU WIN or YOU LOSE.
-
Race mode: two players, one seeded course; the big screen shows both runs.
-
Third genre, built tonight: a side-scrolling wire runner (jump, slide, hook and swing).
Inspiration
Broken balance kills games: one hero wins too often, players quit, and small teams have no QA department to catch it. Playtesting by hand takes weeks, and "it feels unfair" is not evidence. I wanted a tool that finds the overpowered hero, suggests a fix, and proves the fix worked instead of guessing.
What it does
- Play anywhere. Scan a QR code and your phone becomes the controller, or switch the lobby to PC mode and play on the laptop with the keyboard. Same rooms, same server, same measured balance data.
- Arena: arrows/WASD move, Space attack, Z skill, X guard
- Cards: ←/→ pick a card, Space play, E end turn
- Runner: Space jump, Space again in the air to hook, ↓ slide
- Measure. Bots play the same matchup hundreds of times in seconds, swapping seats every pair. In a live run on one laptop, 200 matches of Bari vs Real Armor (100 seat-swapped pairs) took about 6 seconds (5.5 s measured), and Bari won 79.5% (Wilson 95% CI 73.4–84.5%, n=200), far above the designer's intended 60% (fair band 55–65%).
- Diagnose. The report explains why: an evidence block and a winner/loser hero profile for each outlier, with a suggested remedy.
- Auto-balance. Rounds of diagnose → up to three candidate patches → fresh-seed re-test, each accepted patch saved as a version, pending designer approval.
- Patch. An LLM (Gemini) reads only aggregate rows and proposes a bounded numeric patch: allowlisted fields, a few changes, a reason for each. A validator rejects anything out of range.
- Prove. The patched data is re-measured on fresh holdout seeds that never overlap the first run. In our recorded live run, Gemini cut Bari's health 540→378 and speed 120→84 and raised Real Armor's health 700→770 and speed 85→93; the holdout rate moved 79.5% → 56.5%, inside the intended band, and the verdict was accepted. The whole measure → patch → re-measure loop took about 21 seconds. The verdict comes from the numbers, not from the AI. Matchups are judged against the designer's intended counter chart (for example, Bari is meant to beat Real Armor 60% of the time), not a flat 50%; the designer can change the intent with a tap and the verdict is re-judged instantly from the stored results. Fair matchups end with "no patch needed".
- Version it. Every patch is a version with a parent hash, like a commit. You can play any version, run bots on it, continue auto-balancing from it, or make an older one current again.
The same game contract runs three genres:
- Hero Arena (real-time 1v1, above).
- Hero Cards (turn-based duel). Gemini raised one card's cost from 1 to 2, and a weak deck went from 36.0% to 55.5% on fresh seeds (accepted).
- Sepia Tentacle Run (new tonight): a 2D side-scrolling wire runner. Jump the gaps, slide under pipes, and hook a tentacle onto pillar tops to swing across chasms. The analysis is per section: the clear rate of each of six sections against the designer's target band. Section 4 was built to be nearly impossible, with its pillars too far apart. In a live run on the final build (100 runs, about 38 s), the bots cleared it 0% of the time against a 65% target. Gemini moved the pillars closer (anchor spacing 9,500 → 4,000 units), and on fresh seeds section 4 came back at 74.0% (95% CI 63.3–82.5%, the 77 runs that reached it), within reach of the band, accepted.
- Race mode: two players, or one player against a bot, run the same seeded course, each on their own phone. The first fall ends the race, and each phone shows only YOU WIN or YOU LOSE, with the cause in small text underneath.
By the numbers
- Arena throughput: 200 matches (100 seat-swapped pairs) in about 6 s on one laptop
- Card throughput: 200 games in 0.9 s
- Full roster run: 3,000 games, 5 heroes, 10 non-mirror matchups, 0 invalid matches
- Measure → patch → re-test loop: about 21 s in the arena (about 66 s with the full hero diagnosis), about 22 s for cards; Gemini answers in ~1 s
- Arena fix: Bari vs Real Armor 79.5% (CI 73.4–84.5, n=200) → 56.5% on fresh seeds, accepted
- Card fix: weak deck 36.0% (CI 29.7–42.9) → 55.5% (CI 48.6–62.2) after one cost change 1→2
- A plausible fix we rejected: arrow power 92→72: 79.5% → 76.0%, inside the noise, rejected
- Auto-balance, 3 rounds: 8 candidate patches: 2 accepted, 6 rejected by the numbers; Bari vs Daewi 99% → 93%; Daewi (19.8% over 800 games) and Bari were handed to the designer with evidence when number changes stopped helping
- Self-play training: 6,720 games per hero; Daewi's training fitness rose 9% → 29%; against an opponent held out of training (Nightcrow), Sepia's trained bot did worse (85.5% → 71.5%), exposing overfitting
- Skill sensitivity: the same matchup ranges 54%–90% across three bot policies, 97% with trained bots
- Human play: gentle bot needs ~16.6 s (median) to KO a standing player vs 5.7 s for the measurement bot
- Runner fix: section 4 clear rate 0% → 74.0% on fresh seeds after one Gemini spacing change (9,500 → 4,000), accepted
- Lighter assets: game images cut from 23.8 MB to 1.6 MB (arena 3.5 → 0.44, shared 1.7 → 0.36, runner 18.6 → 0.78 MB), so the screens stay smooth on a hot laptop
- Quality: 516 automated tests passing (0 failing); same seed + same data → same final state hash
How we built it
- One TypeScript codebase: React + Phaser + Vite front end; engine, analysis and server in TypeScript on Node, with runtime JSON-schema validation.
- A deterministic simulation (integer math, seeded PRNG, fixed 60 Hz ticks, canonical state hashes) behind an authoritative WebSocket room server.
- Run records stored in MongoDB Atlas; per-match time series in Tiger Data (TimescaleDB) power a win-rate-by-version trend.
- A bot runner on worker threads; the bot policy draws reaction delay, guard chance and aim from its own seeded PRNG, so every seed plays a different match but any match can be replayed exactly.
- Analysis with Wilson intervals, seat-swapped pairs, holdout seeds and a patch validator; a Gemini adapter with timeout, one retry and a labelled offline fallback.
- Phone controller with a floating stick and haptics; a gentler bot for human play (measurement bots are unchanged).
- Self-play trained bots (an evolution strategy over bot habits, one opponent held out) showed that skilled play makes the imbalance worse (Bari vs Real Armor 79.5% → 97%), so balance has to be checked at several skill levels. The training loop extends ideas from a self-improving agent I built for a previous competition, rebuilt from scratch for game balance.
- Built solo; AI coding assistants helped write and review code; design and integration decisions are mine.
Challenges we ran into
- Bots that never guard hide the bug. The designed flaw (an arrow that ignores guard) didn't matter because the first bots never guarded, so every run said 50%. Making bots guard like people exposed it.
- Too-perfect numbers. The next bots ignored the match seed, so results came out at exactly 0%, 50% or 100%. Seeding each bot from (match seed, seat) gave realistic, continuous rates.
- Dropped button presses. The server only accepted inputs stamped for exactly the next tick, so phone presses that crossed the internet a few ticks late were silently dropped and the game felt unresponsive. Inputs up to half a second late are now applied on the next tick, and real-time play got much better. A press and its release that reach the server in the same tick were also dropping the release (a jump stuck down); the second input now waits one tick.
- "Better" is not always good. One AI patch took a level from 15% to 97.9%. A check that only asked "did it improve?" would have accepted it, but the level became trivial. Each target now has an intended band (target ±5 points). A result counts as on target when its 95% confidence interval overlaps the band, and a patch is rejected when the whole interval sits above the band (too easy) or a level that wasn't flagged gets clearly worse.
- Statistics vs. demo time. A verdict needs at least 200 games; we sized the live run to a single matchup (seconds) and show the all-matchup table from a recorded run.
Accomplishments that we're proud of
- A judge can play, watch the AI fix the game, and replay the fixed version within two minutes.
- It also found an imbalance I didn't design: Daewi loses to almost everyone. Raising his attack or speed by 30% changed nothing, raising his health did, and replays showed the root cause (a 10-tick heavy swing interrupted by 3-tick jabs), which is a designer's decision, so the platform hands it over with that evidence.
- A third game with real art, built and tuned in one evening on the same contract. The engine, bot, analysis and dashboard reuse the same pipeline.
What we learned
Trustworthy AI changes need a referee that isn't the AI: fixed seeds, holdout data and a hard validator made the LLM's suggestion something you can check, not something you have to believe. And bot-fair is not always human-fair: a patch that fixes bot-vs-bot balance can still feel one-sided to a human, which is why we record human matches next to bot data.
What's next
- Grow it as a cross-platform playtest service: PC and phone players in the same rooms, feeding the same balance data.
- Up to 4 players per match (2v2 or free-for-all), on phones and laptops alike. The room server already seats four; the games are 1v1 or solo today.
- 2v2 team races for the runner, building on tonight's 1v1 race mode.
- Runner levels with layered platforms, tuned with the same bot → patch → holdout loop.
- More genres on the same contract and a plug-in for existing engines.
- Human-style bots from recorded matches (they already share the bot data format).
- Bot personas for different skill levels, since our measurements show balance shifts with play style (Bari vs Real Armor: 54% to 90% across three bot policies).
Rights
Characters and art are original, all rights reserved.
Built With
- gemini
- mongodb
- node.js
- phaser.js
- react
- timescaledb
- typescript
- vite
- websockets