Pokémon JS: One Trainer, Many Agents

Tools exposed in Chrome WebMCP
Game Screen

Inspiration

In 2014, tens of thousands of strangers played a single game of Pokémon Red together by typing commands into a Twitch chat. One avatar, one party, one save file, and a crowd all pressing buttons at once. It took over two weeks to finish, it produced its own mythology — the Helix Fossil, Bird Jesus — and it was gloriously, famously chaotic. The trainer spent whole hours walking into walls.

The chaos was not a bug in the crowd, it was a missing capability. Those players had no way to coordinate and no way to see the game as anything but a video stream. They could not check whose turn it was, could not read the party's HP as data, could not agree who was driving. All they could do was mash inputs and hope. It got so bad they had to invent "Democracy mode" — a voting system — just to get through a maze.

So we asked: what if the crowd were agents, and what if they could actually read the game?

That question runs straight into the reason WebMCP exists. A Game Boy is the hardest possible surface for an agent: 160×144 pixels of sprite work, no DOM, no ARIA labels, no semantics. A battle menu is four words drawn onto a tilemap. A wall is a pixel that happens to be dark. Tall grass and a decorative bush look identical and behave completely differently. A crowd of agents watching a video stream would be exactly as lost as the crowd in 2014.

So we took an existing open-source Pokémon Red/Blue recreation and made it expose what it knows instead of what it draws — then let many agents into the same save file at once. Twitch Plays Pokémon, if the crowd could read the game and talk to each other.

What it does

Gen-1 Pokémon in the browser, playable by a person, an agent, or both at the same time.

Thirteen tools on document.modelContext cover the whole game, shaped for a reader rather than a renderer:

The map arrives as an ASCII grid with a legend — @ you, ~ tall grass, D door, T trainer — plus the notable tiles in view with absolute coordinates you can pass straight back to walk_to.
The battle's 52-step animation state machine becomes a named phase: choose-move, opponent-fainted, animating. Agents never see a stage number.
Menus report their entries and where the cursor sits, so choosing is select_menu_item("BULBASAUR") instead of three arrow presses and a hope.
Every screen that holds input reports waitingFor — dialogue, menu-open, wild-encounter — so a frozen screen is never a dead end.

And it is one world, not one session per player — the Twitch Plays part. Every tab that joins a room drives the same trainer. Game and battle actions are broadcast over Supabase Realtime and applied everywhere, so a person on the D-pad and an agent calling tools are in the same fight, on the same party, and either can take the turn.

The difference from 2014 is that this crowd can coordinate. get_party_agents shows who else is connected and what they have been doing; tell_agents leaves a note to claim a task before starting something long, so two agents do not both walk into the same trainer. One tab is elected driver — oldest peer, tie-broken on id, derived from presence so it needs no negotiation and hands over on its own — which is what stops every tab rolling its own wild Pokémon. No Democracy mode required.

The room is the save file. A world survives everyone disconnecting, and handing a playthrough to a friend, or to another agent, is sending a URL.

How we built it

The registration is the easy part: webmcp-react, a WebMCPProvider, and a useMcpTool call per tool. The real work was making the game legible.

src/webmcp/snapshot.ts — one read model for everything: position, ASCII map, notable tiles, party with types and PP, bag, on-screen state, battle phase.
src/webmcp/pathfinding.ts — BFS over the reducers' own walkability rules, so a path we return is a path the game will actually accept. No separate model of the world to drift out of sync.
src/app/use-menu-registration.ts — mirrors on-screen lists into Redux so a tool can read the cursor and pick by label.
src/state/session.ts — the shared world: action broadcast, presence-based driver election, room persistence.
src/state/battleSlice.ts — battle state lifted out of component state, which is what lets every agent watch one fight.

Writes go through the same event bus as the on-screen buttons, so an agent cannot reach a state a player couldn't.

Challenges we ran into

We found most of these by actually playing it with an agent, not by reading the code.

A battle read cost 37,000 tokens. The battle payload included the trainer object, which carries a portrait and twelve walk sprites — inlined by the bundler as base64 data URIs. Every single read of a trainer fight shipped all of it. Replacing it with {name, canBattle} took the payload to ~600 tokens, a 60× cut.

The party screen was a blind spot, and it cost us a run. PokemonList kept its cursor in component state and never mirrored it anywhere, so get_game_state reported no menu at all while the party was on screen. The agent picked by arrow presses it could not verify — and sent out a Pokémon that had already fainted, because the voluntary switch had no guard on it. The party wiped shortly after. That one bug taught us more than anything else on this list.

Presses were landing on deaf frames. Battle choreography ignores input while animating, so a burst of A presses mostly hit nothing. Tools now wait for a phase that actually wants input before spending a press.

Pathfinding refused to walk onto items. The game's canWalk treats an uncollected item as solid but stepping onto it is exactly how you pick it up. Item tiles are now valid destinations, though never intermediate steps, so paths stay ones the reducers accept.

Battles rewound themselves. The intro effect depended on the driver flag, which is recomputed on every presence sync, so a peer joining or leaving mid-fight replayed the choreography from step 0, rewinding the battle under everyone.

Then battles froze instead. Fixing the rewind exposed the deeper problem: the choreography runs on setTimeout chains that do not survive a reload, so a fight restored from the save had no timer left to advance it. It now resumes at the last stage that accepts input.

Deploys did not reach anyone. The service worker precached the app shell cache-first, so a browser that had loaded the game once kept booting the old bundle — the stale build even re-registered the worker after we removed it. The worker is gone entirely now; a shared world is worse off stale than offline.

Accomplishments that we're proud of

We are equally proud of the boring parts: a 60× reduction in the cost of reading a battle, and a test suite that pins the agent-facing behaviour, that pathfinding only routes through tiles the reducers accept, that no sprite data can leak back into a battle payload, that every driver election is unanimous without negotiation, that a frozen screen always says what it is waiting for.

What we learned

Agent-facing state is a product surface, and it needs the same care as a UI. Every place we had kept state in a component was a blind spot : invisible to a tool, and therefore a place an agent would guess and get it wrong. Lifting it into the store was consistently the fix.

A frozen screen with no explanation is the cardinal sin. A human sees a text box and presses A. An agent sees frozen: true and nothing else, and has no move. Every refusal we return now names its cause, and the snapshot and the refusal are computed by the same function so they can never disagree.

Cheap reads change how an agent plays. When reading the game was expensive, the agent guessed to avoid looking. When it got cheap, it looked before acting, and played far better.

And the tools should encode the game's own rules, not a copy of them. Pathfinding over the actual reducers, phases named from the actual state machine: every place we duplicated a rule instead of reusing it, the two drifted and the agent paid.

What's next for Pokémon JS: One Trainer, Many Agents

Collaboration, and it turns out to be a stranger problem than we expected.

There is only one avatar. You cannot parallelise a single body walking through a single world — two agents cannot grind in two places at once, and a third cannot go shopping while the other two fight. So "many agents" can never mean dividing the labour, which is what we assumed at the start and had to throw away.

What's left, once you drop division of labour, is everything that happens around the one person holding the thing. Three shapes we want to try:

The second player who isn't holding the controller. Your little brother on the couch, watching you play and telling you where to go. The friend who read the strategy guide and knows the next gym is north and its leader uses rock types, so bring water. The one who remembers this route has better odds for the Pokémon you actually want. None of that is in the game, a Game Boy never tells you what beats a rock type, and no amount of reading pixels will either. It comes from outside, which is exactly what an agent is good for and a tilemap is not.

The same thing pointed the other way. A person who can see the whole screen at a glance steering the agents: go and grind for a bit, don't waste the Max Potion, stop trying to catch that. Humans are better than agents at noticing when a plan has quietly stopped working, and right now we give them no way to say so.

Passing the Game Boy. The oldest multiplayer protocol there is — you hit a bit you can't do, so you hand it over, and you keep watching. An agent should be able to take the console for the boring stretch and give it back, or a person should be able to take over mid-battle when it matters. Half of this already exists by accident: the room is the save file, so handing a playthrough to someone else is sending them a URL. What's missing is the handover itself, knowing whose turn it is, and being able to give it up or ask for it.

We have not designed any of it properly yet. Collaboration mode is genuinely undefined: we do not know whether this belongs in the notes board we already have, or whether the people and agents in a world need an actual chat to talk while they play. Probably a chat. We would rather ship the honest version of that question than pretend we have the answer.