GIF
GIF
GIF

tensormew

tensormew plays Pokémon Crystal and beats Falkner for the first gym badge. We use reinforcement learning to teach an agent to play Pokémon Crystal, and call an LLM only when the game actually needs a decision.

Inspiration

Pokémon Crystal is a long, stateful game: maps, one-way ledges, scripts, menus, and a gym fight at the end of the first stretch. Calling a language model on every frame is slow and expensive, and a screenshot still does not show collision bytes or event flags.

Most steps do not need language. Walking a tile, closing a text box, and picking a damaging move are mechanical. Language is for the moment the agent is stuck and has to choose a goal. The question we measured is how many steps actually needed an LLM on the way to the first badge.

Crystal is a hard test of that split:

PyBoy has no collision grid for this ROM.
The map is a pair (group, number).
Cut is face the tree, press A, confirm.
Ledges are one-way.
Dialogue is an open script (wScriptMode / wScriptRunning), not a single textbox bit.

What it does

An agent plays Pokémon Crystal (US 1.1) inside PyBoy and clears the opening of Johto: leave New Bark, reach Cherrygrove, finish the Mystery Egg quest, walk to Violet City, and beat Falkner.

Three actors share one loop:

Actor	When it runs	What it outputs
RL policy	Overworld, no trigger	One of `up`, `down`, `left`, `right`, `A`
Macros	Open dialogue or a routine battle	Mash A, or pick a move slot
LLM director	Cut tree, stuck tile, hard battle, open script	One tool call plus a one-sentence reason

Tools the model may call: go_to, run_path, use_cut, mash_text, choose_move, switch_pokemon. Every call requires a reason. That sentence is logged for the judges. It is never sent to the emulator as input.

There is no town-to-town route table. The policy is paid for new tiles, new maps, story flags, catches, levels, and the maps on the way to Violet Gym. Falkner is the terminal reward on that chain: EVENT_BEAT_FALKNER pays +50, and wJohtoBadges flips to the Zephyr Badge.

Outcome

The run starts in New Bark, map (24, 4), with no badges. It crosses Route 29, Cherrygrove, Route 30, Mr. Pokémon's house, Route 31, and Violet City, map (10, 5), then enters Violet Gym, map (10, 7).

Dialogue with Elm, the rival, and Mr. Pokémon is finished by macros.
Falkner's Flying-types are fought with a type chart, not move slot 1.
The episode ends with one Johto badge and EVENT_BEAT_FALKNER set.

Demo surfaces

PyBoy window of the Game Boy screen, from New Bark through the gym.
Live HUD: return, entropy, tiles and maps discovered, badge count.
Streamlit thought log (app.py) over logs/decisions.jsonl: map, position, actor, tool, reason, tokens, and estimated dollars. Plain walks show actor=rl and $0. The LLM rows are the stuck, dialogue, and gym moments.

How we built it

Stack: Python 3.13, PyBoy 2.8.1, the pret symbol file rom/pokemon_crystal.sym, NumPy PPO, PyTorch for the vision policy, Gemini Tool Calling, Pydantic tool schemas, Streamlit, SQLite run log.

ROM identity is pinned: SHA1 f2f52230b536214ef7c9924f483392993e226cfb. RAM is read through symbol names and banked WRAM (pyboy.memory[bank, addr]).

Position is wXCoord / wYCoord.
Badges are wJohtoBadges.
Story state is the 2,048-bit wEventFlags block. That's how we know the egg was delivered and Falkner is down before the badge sprite even updates.

Layer 1: emulator and game state

src/emulator.py: loads the ROM, ticks, and saves.
src/ram_parser.py: turns RAM into a GameState and a text snapshot for the model.
src/collision.py: classifies pret tile IDs: walls, Cut trees, warps, and one-way COLL_HOP_* ledges.
src/step.py: moves exactly one tile.
src/states.py: a labeled checkpoint manifest, so a gym attempt can resume from Violet instead of the title screen.

Layer 2: the RL agent

Routine overworld steps are a policy, not the LLM. That's what carried the agent from New Bark to the gym door.

Small CPU policy (src/ppo.py, models/ppo_policy.npz, ~14.5 KB)

Linear or 2-layer MLP actor-critic. Five actions.
17 observations: four neighbor tiles, x/y, badges, battle, dialogue, four visited-neighbor bits, party size, max level, caught count, and a story-stage scalar.
Story stage is 0 in New Bark, ~0.45 with the Mystery Egg, 0.75 in Violet City, 0.85 inside the gym, and 1.0 once Falkner is beaten.

Vision policy (src/deep_policy.py, models/deep_policy.pt, ~50.5 MB, 13,220,678 parameters)

A ResNet-style trunk on the raw 144×160 frame, an MLP on the 17 RAM features, a 512-wide GRU, then actor and critic heads.
Trained with PPO and curriculum resets (tools/train_deep_ppo.py) on parallel headless PyBoy workers.
Play it with ./run_live.sh.

Reward (src/rl_env.py) is potential-based shaping (Ng, Harada & Russell) plus one-time bonuses that spell out the Falkner route:

Signal	Reward
Step cost	−0.01
New tile	+1 (+2 on the current objective maps)
New map	+10
Waypoints	Route 29 +3, Cherrygrove +5, Mr. Pokémon's house +8, Violet City +12, Violet Gym +20
Story flags	Starter from Elm +15, Mystery Egg +20, egg returned +20, rival in Cherrygrove +15, beat Falkner +50
Catches	10 / 5 / 2.5 (floor 0.5)
Level-ups	scaled by 5, so the party is strong enough for the gym before the badge reward dominates

Route 29's potential pulls the agent up the one-tile gap at x=31 and away from the south ledge.

Layer 3: the LLM director

src/triggers.py fires on CUT_BLOCKED, STUCK (same tile for 8 steps), BATTLE, and DIALOGUE.
src/llm_supervisor.py sends the state text plus the trigger and forces a tool call.
src/tool_dispatch.py runs it.
src/agent_loop.py logs map, position, badges, actor, trigger, tool, reason, tokens, and estimated USD.

The system prompt holds the facts the pixels omit: no Cut before the badge, ledges don't climb, and the northern highway on Route 29 is the road to Cherrygrove.

Layer 4: macros and the gym fight

src/macros.py holds dialogue open until wScriptMode returns to 0. That's how Elm, Mom, and the rival actually finish.
src/battle.py scores moves by power, accuracy, and the Gen 2 type chart. Ground into Flying is 0×, so Mud-Slap is not what we throw at Pidgey. Electric, Rock, and Ice are 2×.
src/saycan.py blocks the gym while the party is empty or under level 10, then lets the fight start.

Challenges we ran into

The screen is not the state. Map ID, event flags, and collision live in banked WRAM. The vision net still needed the 17 RAM features to tell "egg delivered" from "standing on a wall."
One-way ledges. Hopping south on Route 29 dead-ends the run before Cherrygrove. Collision became directional, and the reward potential was written around the x=31 gap so the policy takes the northern highway.
Dialogue looks like a stall. A long NPC script used to trip the stuck detector and abort the quest. We raised the dialogue and battle streak cap and made mash_text wait until the script closed. That's what got us through Elm and the Mystery Egg handoff.
A tiny policy can fake an exit. Early linear PPO showed a high New Bark exit rate in a batch, then failed a single visual run. The 13.2M-parameter GRU is what we trusted for the full walk to Violet.
Falkner is not a walking problem. Pressing move slot 1 loses to Flying-types. The fight only landed after move choice used the type chart and the party was gated at level 10.
Tokens versus frames. An always-on LLM on this route would have been a call per tile from New Bark to the gym. Triggers-only kept the dollar counter on the decisions that mattered.

Accomplishments we are proud of !!!

Zephyr Badge. The agent leaves New Bark, finishes the Mystery Egg, reaches Violet Gym, and beats Falkner. wJohtoBadges is 1. EVENT_BEAT_FALKNER is set. That's the run.
A full agent loop where the language model is a tool-calling director. Button presses stay in the policy and the macros, including the gym fight.
The early Johto plot encoded as reward: Mystery Egg +20, egg returned +20, rival +15, Violet Gym +20, Falkner +50, plus a Route 29 potential that avoids the ledge trap.
Two policies in-tree: a CPU MLP for fast segment tests, and a 13.2M-parameter screen-plus-RAM GRU trained on vectorized PyBoy workers, which is the policy that plays the badge run.
A decision log and Streamlit view of the whole route: $0 on walking steps, with a reason and a token count only when the director is invoked.
Directional collision, warps, and indoor transitions, so the agent enters Elm's lab, Mr. Pokémon's house, and Violet Gym instead of treating doors as walls.
Banked RAM, pret collision, one-tile stepping, a labeled checkpoint chain, and 34 test modules around that stack.

Cool stuff we learned !!!

The first badge is a control problem with a few language moments. Wiring the model to buttons made the route unreproducible. Wiring it to six validated tools made every intervention on the way to Falkner inspectable.
**Realized that the Pokemon were too low leveled so it had to go back and continue to level up - showcasing long term horizon planning.
Reward has to match the map. A generic new-tile bonus hops the south ledge and never sees Cherrygrove. The Route 29 potential is why the policy arrives at the gym.
Event flags are the progress bar. The badge bit stays zero until the last fight. EVENT_GOT_A_POKEMON_FROM_ELM through EVENT_BEAT_FALKNER is the actual curve, and the +50 at the end is the only reward that should dominate it.
Batch eval is the honest one. A single window can miss an exit the batch finds. We log entropy beside return so a collapsed policy is visible before we claim the gym.

What's next for tensormew

The Zephyr Badge is the first checkpoint, not the goal.

Bugsy in Azalea Town. Teach the same loop HM01 Cut, stand on a real Cut tree, fire CUT_BLOCKED, and clear Ilex Forest with use_cut() on the way to the Hive Badge.
Whitney in Goldenrod. The Miltank fight is the next battle-policy test. Replace "best damaging move" with a search battler (Metamon) once the party can reach her.
Publish the cost table for the Falkner run: calls, tokens, and dollars for triggers-only vs always-on, taken from the saved episode rather than a short sample trace.
Keep the curriculum moving: resume from post_falkner and pay the same style of waypoint and event rewards through Union Cave, Slowpoke Well, and Goldenrod.
Longer term, the research stack in SPEC.md Appendix A: a larger simulator fleet, ghost scouts, and a minimax battler, on top of the mover, triggers, supervisor, and PPO that already earned badge one.