PokeClanker 9b

A Big Problem

In 2023 frontier LLMs struggled to maintain coherence between multi-step tasks. Now, the largest of models are able to navigate dense long horizon tasks spanning several hours (think end-to-end software dev, SEO work, long-form video understanding, etc). These tasks are costly, time consuming, and still difficult to get working adequately. One such task which is commonly used to benchmark a model's long horizon reasoning is... Pokémon Fire Red, a 2004 Game Boy Advance game that takes ~30 hours to complete end-to-end. Pokémon Fire Red is a great candidate for this evaluation because the sequencing required to beat the game necessitates a careful blend of long-term planning and short-term action. In fact, it is so difficult for LLMs to complete, that it was only recently that Claude Fable 5 became the first LLM to successfully complete the game end-to-end.

A Small Solution

So what if I told you a model ~600 times smaller than Claude Fable 5 could complete Pokémon Fire Red in half the time and for 1/1000th of the cost?

PokéClanker 9b & Fire Red Harness

Introducing PokéClanker 9b, a fine tune of Qwen 3.5 9b that surpasses frontier performance on domain restricted long-horizon reasoning tasks. Utilizing supervised fine tuning over ~49 hours of gameplay, 6 text walkthroughs, and distillation from Gemini-3.5-flash, PokéClanker excels at long term planning. Under the hood, PokéClanker has been fine tuned to utilize Fire Red Harness, a custom harness I developed to reduce model-environment friction and increase accuracy of short-term actions. Fire Red Harness was built by decompiling Pokémon Fire Red to identify the underlying location of game state in system memory.

Methodology

Data Collection

~49 hours of gameplay was sourced from three Pokémon Fire Red walkthroughs found on Youtube. This raw footage was parsed into (state, action) pairs by a video understanding framework using Gemma4 via Cerebras and informational frame detection by reverse engineering H.265 video decoding. This allowed for all ~49 hours to be preprocessed in less than three hours (~16x realtime speed) - a monumental achievement in data engineering which would not have been possible even just a year ago.
6 walkthroughs were sourced and preprocessed into (question, answer) pairs based on general game knowledge and strategy.
Gemini-3.5-flash was hooked up to Fire Red Harness and used to generated training samples matching (state, action) in the exact format that Fire Red Harness uses. This was used as a method of distillation.

Post Training

I used Freesolo with an SFT setup on 90% training samples 10% eval samples, three epochs, a random shuffle of (state, action) and (question, anwer) pairs.

Results

See the video attached to this submission for a demo of the system end-to-end (includes the start of the game to beating the first gym).

General game knowledge evaluation:

39.3% Gemini-3.5-flash
25.2% PokéClanker 9b
11.6% Qwen3.5-9b

State <-> Action evaluation:

92.4% PokéClanker 9b (100.0% valid tool calls)
47.3% Qwen3.5-9b (69.5% valid tool calls)'
40.0% Gemini-3.5-flash (72.6% valid tool calls)

Start -> First Gym evaluation:

100.0% PokéClanker 9b
~85.0% Gemini-3.5-flash
~10.0% Qwen3.5-9b

Analysis

PokéClanker 9b demonstrates the immense capability of a hyperspecialized small language model with a strong harness. SFT was able to allow PokéClanker 9b to correctly answer over double of the general game knowledge evaluation questions. The state-action pair training allowed the model to understand how to transform temporal game state into long-term planning.

Limitations

Due to time constraints, PokéClanker 9b was only validated end-to-end on it's gameplay from game start to the end of the first gym. Given the complex nature of Pokemon Fire Red, it is likely that changes would have to be made to the harness to complete a full playthrough (track key items, where to find which Pokémon, which NPCs to interact with, etc).
Due to time constraints, the harness does not support adding, removing, or swapping Pokémon from the active team. It is technically possible for the model to do this manually on it's own, but no training samples of this behavior were present in the training data.