Pikachu

N/A

Comment

Pokemon Showdown is an application that simulates Pokemon battles between the 1025 currently existing Pokemon under various rulesets, including Pokemon Champions, doubles, VGC formats and Smogon single battles. Competitive Pokemon allows for a significant amount of variance and strategy, and we, being fans of competitive Pokemon, thought it would be cool if we simulates Pokemon Showdown and tracks the performance of Showdown as well as the various moves that CPU trainers make. This also has a real world application in that one can compare it to the AI in the games and potentially optimise the AI that the actual games developed by Nintendo uses.

Effectively, what our benchmark does is that it plays Pokemon Showdown through a locally hosted server, or rather, an AI plays Showdown through a locally hosted server. It simulates random battles with random Pokemon under a Generation 9 singles ruleset (which is the latest ruleset), and tracks which moves that the AI uses. Pokemon is a turn-based game, where each turn a unique move, among 940 moves, is used, and these moves are what are tracked.

The main challenge we ran into was that the environment would continuously fail to build, specifically it would not submit correctly.

We learned about optimising benchmarks effectively, and we hope to be able to continue this project so that we can understand how to best optimise Showdown movesets.

Project Pikachu — Build Summary

What We Built

A Pokémon Showdown benchmark environment for the SWECC Mesocosm platform. The environment pits AI models against a random opponent in Gen 9 Random Battles, scoring them on win rate. The goal is to test how well different AI models understand game strategy, type matchups, and decision-making under uncertainty.

Architecture

Mesocosm CLI (benchmark runner)
        ↓
  adapter.py  (HTTP server on port 8765)
        ↓
   env.py  (game logic)
        ↓
  poke-env  (Python library)
        ↓
  Pokémon Showdown server (Node.js, port 8000)
  • Mesocosm sends the LLM's move decisions to the adapter
  • adapter.py translates HTTP requests into env calls
  • env.py manages the battle using poke-env
  • poke-env communicates with the Showdown server via websockets
  • Showdown server runs the actual battle simulation

Key Files

  • env.py — battle logic, threading, poke-env integration
  • adapter.py — HTTP server exposing /reset, /step, /health, /close
  • benchanything.json — defines the domain, scoring, and the AI's instructions
  • requirements.txt — pins poke-env==0.15.0

Challenges We Hit

1. Node.js not installed

npm wasn't recognized because Node.js wasn't on the machine. Had to install it from nodejs.org first before the Showdown server could be set up.

2. Showdown server build

node pokemon-showdown start failed because the TypeScript source hadn't been compiled yet. Had to run npm run build first, then use node dist/server/index.js instead of the documented command.

3. mesocosm init put files in auxiliary/

The scaffold put env.py, adapter.py, and benchanything.json inside an auxiliary subfolder instead of the root, causing the CLI to not find them. Had to move them manually.

4. poke-env API changes

The ShowdownServerConfiguration class changed in newer versions of poke-env — it's no longer callable as a constructor. Had to switch to the pre-built LocalhostServerConfiguration constant instead.

5. Async event loop conflict

poke-env runs its own async event loop internally. Our original env.py tried to run asyncio.get_event_loop().run_until_complete() inside it, causing RuntimeError: This event loop is already running. Fixed by running poke-env in a separate thread with its own event loop using asyncio.new_event_loop() and threading.Thread.

6. benchanything.json schema

The manifest format was more complex than documented — required deeply nested objects for binding_vow, observation_space, action_space, reward, and episode. Had to read the source code of bench_common directly to figure out the exact Pydantic schema.

7. adapter.py missing last line

The HTTPServer(...).serve_forever() line got cut off when pasting into the file, causing the adapter to exit immediately with no error. Fixed with Add-Content.

8. Threading synchronization

Getting the HTTP adapter (synchronous) to talk to poke-env (async) required threading Events (_state_event, _action_event) to block and wait for battle state updates without deadlocking.

9. Platform build failures

The Mesocosm platform repeatedly failed to build the environment even with minimal code. The CLI has no build log access so the root cause is unknown — likely a platform-side issue that requires SWECC Discord support to resolve.

Current Status

  • Local runs work — 5/5 episodes scoreable, 35 avg steps per battle
  • Model makes decisions every turn using the battle state
  • Scoring works — win/loss tracked correctly
  • Code pushed to github.com/careycc6/Project-Pikachu
  • Platform submission keeps failing — needs SWECC Discord support
  • Prompt includes full type chart, ability knowledge, and strategy rules for stronger models
Share this project:

Updates