Pokemon-Agent

Inspiration

I've been playing Pokemon since the original, my son loves Pokemon, everyone loves Pokemon. I've seen a few videos on an older ML version of this playing the game and a newer version of what looks to be a very expensive AI attempting to play the game, I wanted something a little more simple that can be played on local compute with a more simplistic model.

What it does

A combination of Hermes using llama.cpp with Gemma-4-12B (text + visual) and Laya (the open weights version of Jev) work together. Gemma helps when Laya is stuck and Laya helps to make decisions and move quick. This went under some revisions where it was changed to Von only, trying to train Qwen2-VL-Instruct then even a Nemotron equivalent.

How we built it

Hermes agent using the Nemotron Ultra 550b model with High/Max reasoning building on top of a fork of the NousResearch codebase.

Challenges we ran into

As we add more, it gets more complex and makes it harder for the Gemma and Laya models to move forward sometimes, finding the balance between adding more features that should theoretically make it easier for the LLM to move through the world can make it harder. More work to be done.

Accomplishments that we're proud of

It actually moves and the models co-operate all on local compute.

What we learned

So much about AI and ML, what to feed and what not to feed, I was hoping initially for Gemma to compute screenshots and make decisions but the game is way too slow at that point so learning how to find that sweet spot with the two models was a good experience.

What's next for Pokemon-Agent

Keep on building and running it, at least till it defeats Brock and then we keep going :D

Built With

• discord
• hermes
• jetson
• nemotron
• nvidia