Inspiration
Currently bringing AIs into the most of games is unavailable to people. Mainly due to the latency it takes AI to take screenshots, process them, come up with an action, verify that action. Ais are just too slow for the game and they would get mogged by a fast thinking human 10/10 times. Using APIs reduces the latency, but not enough + produces another problem — huge costs.
I just wanted to create not a system, that would allow AIs to play old (pre-AI) games. But a system that would allow AIs to play new (built for AIs) games natively. Also so that this system would be affordable to people.
What it does
My stack divides the "brain" of AIs in games into 3 layers: instincts, tactical, and strategic.
Instincts - basic game actions that are available to a character. Go to, attack, defend, etc. Tactical - Luna in fast mode to handle the majority of decisions. Strategic - Sol as a strategic mind setting goals for Luna.
Both Luna and Sol can "program" the body. They can set triggers for its instincts, set policies, bundles of instincts. As a result real-time action - like fighting is handled by programmed approach created by slow brains. The body executes them. After a fight is finished - AIs Sol and Luna can reflect on it and update its "skills". The thinking layers are also asynchronous from the body actions. While the Ais are thinking, the body carries on - fighting, healing, cooking, mining. etc. There is no unproductive idle time.
Kinda like a human brain works. It creates "routines" for the body. It also teaches the body how to handle various situations.
As a result we get an embodies AI character in a game. Which can set strategic goals, find tactical solutions, and handle real time fight. Basically - a competent AI character that will not get easily destroyed by human players.
I also created a short game - Blackglass Crossing, to showcase the stack. The character there is tasked with a delivery of package, and will fast an unexpected combat along the way, will have to replan a failed plan, and think through its approach toward a boss.
Why it is a developer tool
Agent Brain is a reusable runtime. Developers expose character-visible information and legal actions; the Brain returns intents that the authoritative game validates and executes.
How we built it
We built Agent Brain as portable TypeScript packages for its game-neutral protocol, asynchronous scheduler, shared memory, secure local pairing, and Codex runner. GPT-5.6 Luna provides Tactics, GPT-5.6 Sol provides Strategy, and deterministic game-approved Instinct rules execute at body speed. The stack uses the player's local Codex login, with no OpenAI API key.
How we used Codex
I have built this with a combination of models. Codex with gpt 5.5 played a key part. It had much better technical understanding of my vision and became a trusted, competent partner.
Apart from that, the usage limits of Codex are amazing a helped me a lot with playtesting the stack.
GPT 5.6 accelerated that even further. Amazing usage limits made it my core gaming agent + Luna model allowed to add one more "fast" layer due to being faster and cheaper then Sol. All that impacted the gameplay experience to a large degree.
How to test it
Blackglass Crossing is the playable demo. On macOS, Windows, or Linux, install Node.js 22+ and the Codex CLI, then run:
npm ci
npm run judge
Open http://127.0.0.1:8787, sign in with codex login, and run the pairing command shown in the game. No OpenAI API key is required.
Hard parts
Conceptualizing that there should be a programmable fast-automatic response layer - instincts, like we, humans have. Direct AI controls of game characters is easy, tried many times and doesn't properly work. The instinct layer - is the core innovation here.
Making a game "native" for AI agents. Building a channel for Ai - how they perceive the game world. instead of the now standard "control as humans" via screenshots.
Hammering all 3 layers together into one system, bound by a shared memory.
What we are proud of
Giving people a way to make games, where they are able to bring their AI friends. And to do that without using expensive APIs. Basically now codex can become your gaming buddy.
Also, making SOL, Luna and instincts async allows to utilize thinking time of SOL and Luna. The body doesn't need to stand still while the brain is thinking. It can continue on the planned route, it can harvest resources, make campy, heal itself.
What we learned
That in traditional pre AI games there were NPC, that had some decent scripted in-game behavior. Nowadays, we can just bash an AI on top of old NPCs and allow it to create new scripts (skills).
What is next
- optional memory across later sessions and different games — your agent learns with you.
- some optimization to the current stack.
- I'm making my game - Simulation — multiplayer skyrim with a real economy and NATIVE support for Ais to play along with humans
Build Week scope and provenance
I have worked on the agent brain before the build week. During the built week I expanded the Agent Brain from one thinking layer (gpt 5.5) to a strategic - Sol and tactical - Luna fast. I also made the gameplay demo using Sol Ultra as an orchestrator and coder and Grok for sprites and animation generation
Current proof and honest limit
I have recorded a playthough of my gameplay demo. In it, judges can see how strategic and tactical layers plan and act together. How the instinct layer automatically protects the body when under attack and how it automatically utilizes the waiting idle time while Sol and Luna thinking about next steps.
The submission video is a time-compressed edit of that playthough video. It fits 3 minutes. It adds the architecture introduction, shows the real starting prompt.
The honest limit is that my agent brain cannot be used in old games. There needs to be a perception built natively for the AI players in each game. When there is a sequence of actions that require thinking, it can feel quite long for humans. Even tho the Ai character's body will be doing some useful work in the meanwhile.
Built With
- gpt-5.6
- node.js
- openai-codex
- pixijs
- playwright
- typescript
- vite
- vitest