GameSpeak

Inspiration

Games are often designed around precise keyboard, mouse, and controller inputs, which can make them difficult or inaccessible for players with limited mobility or dexterity.

We wanted to explore a different way to play: instead of requiring a player to perform every physical action themselves, what if they could simply describe what they want to do and let an AI agent handle the mechanical inputs?

That idea became GameSpeak.

What it does

GameSpeak is an AI accessibility layer that allows a player to control a game through real-time natural-language voice commands.

Instead of pressing specific buttons or performing precise combinations of movements, the player communicates their intent, such as where they want to move or what they want to accomplish, and the AI interprets the environment and performs the corresponding in-game actions.

Our goal is to shift game interaction from physical execution toward player intent.

How we built it

We used Godot as our game engine to create puzzles and environments for an AI-controlled character to navigate.

To allow the agent to interact with the game, we developed an MCP server that exposes functions for character movement and other in-game actions. Godot preprocesses the world and emits a structured snapshot of the game state, so the agent reasons over meaningful information rather than raw pixels.

Decision-making runs on Jev, TypeSafe AI's System One model, which we access through Backboard.io. Jev is not a text generator. We hand it the current game state and a set of typed questions, and it returns structured answers directly: a Choice between the actions available right now, a Score on an ordered rubric, or a Boolean. Every answer comes back with a probability distribution across the options and a confidence value.

That turned out to be the right shape for our problem. "What should the character do next?" is a classification question, not a writing question. With an ordinary LLM we would have been generating prose, parsing it, validating it, and handling the cases where the model wrote something that didn't map to a real action. Jev removes that step, and removes the latency that comes with it, which was the same problem that pushed us off visual processing earlier.

Backboard sits in front of the stack as a single API layer, so the speech-understanding call and the Jev decision call go through one key, and we get per-request latency and cost visibility while tuning the loop.

This separates the system into three layers: player intent, world understanding, and action execution.

Why the decision model choice mattered

Our control loop calls the decision layer constantly. Every time the world changes, the agent has to decide what to do next, which means the cost and latency of a single decision get multiplied by the length of the play session. Frontier LLMs are priced and paced for occasional, heavyweight calls. We needed the opposite: a small decision, many times a second, all game long.

Jev fit because it doesn't generate tokens to arrive at an answer. It evaluates the declared questions in parallel and returns the typed result, so a decision costs a fraction of what an equivalent LLM call would and returns fast enough to sit inside a real-time loop. TypeSafe reports Jev running up to 193.6x faster and 444.6x cheaper than LLMs on its own workflow evaluations, and in practice it was the difference between an agent that reacts and an agent you wait on. Our earlier Gemini-based vision approach was taking around 8 seconds per decision. Moving the world model into Godot and the decision into Jev collapsed that to something a player can actually play against.

The cost side matters beyond the hackathon budget. An accessibility layer that costs dollars per hour of play is not an accessibility layer most people can use. Cheap decisions are what make continuous, all-session AI assistance realistic rather than a demo.

Confidence-gated execution

Because Jev returns a calibrated confidence score with every decision, we didn't have to treat all decisions as equally trustworthy. Execution is gated on certainty:

High confidence: the action is sent straight to the MCP server and executed.
Moderate confidence: the agent takes the more conservative or reversible option, or re-queries with additional state from Godot before committing.
Low confidence: the agent stops and asks the player to clarify rather than guessing.

This matters specifically because of who the system is for. A player using GameSpeak may not be able to quickly undo a wrong move with a controller, so an agent that acts confidently on a bad interpretation is worse than one that pauses and asks. Confidence scores gave us a principled place to draw that line instead of hardcoding heuristics about which actions are "risky."

It is also only affordable because Jev is cheap. Re-querying on uncertainty means extra calls, and a design that asks the model twice when it isn't sure would be hard to justify at frontier-model pricing.

Designing for any game

GameSpeak is built as three layers with a deliberate contract between them, which is what makes it portable off our own game.

1. Intent layer. Voice is transcribed and resolved into a goal. Nothing in this layer knows anything about Godot, our puzzles, or our character. It only produces goals like "get to the other side," "grab that," "stop."

2. World layer. An adapter produces a state snapshot in a fixed schema: where the player is, what's nearby, what's reachable, what changed. Our Godot build emits this natively, which is what let us drop visual processing. For a game we didn't build, the same snapshot could come from an existing modding or scripting API, an accessibility API, or vision as a fallback where nothing else is exposed.

3. Action layer. The MCP server exposes the set of actions currently available. In our game those are engine calls. In an arbitrary game they would be synthetic keyboard, mouse, or controller events.

The useful consequence is that Jev's questions don't change when the game does. The action set is passed in as data, so porting GameSpeak to a new title means writing one adapter that answers two questions: what can the player perceive right now, and what inputs exist. The intent layer and the decision layer stay as they are.

Challenges we ran into

We initially attempted to use Gemini for visual processing. We expected image analysis to take around 0.5-3 seconds, but in practice it frequently took around 8 seconds and sometimes reached token limits. That level of latency made real-time interaction impractical, so we moved preprocessing into Godot and provided Jev with more useful information about the game state directly.
We originally stored an instruction queue inside the MCP server, but coordinating several asynchronous systems made the implementation unreliable. We ultimately removed it and switched to a simpler sequential action loop.
Getting the decision layer to fail gracefully took real thought. An agent that always acts produces confident nonsense on ambiguous input, so we had to design around uncertainty rather than assume the model would be right.

Accomplishments that we're proud of

Making the game playable entirely through an AI agent
Allowing a player to express high-level intent instead of performing individual mechanical inputs
Giving the AI enough understanding of the game world to interpret situations and confidently perform corresponding actions
Getting decision latency and cost low enough that the agent can run continuously through a play session instead of firing occasionally
Using Jev's confidence scores to decide when to act and when to ask, rather than guessing on every command
Building a working pipeline from real-time voice input to AI reasoning to in-game control

What we learned

Visual understanding is not just an accuracy problem; latency determines whether an agent feels responsive.
Game-engine data can provide useful context that is difficult to recover from pixels alone.
Separating user intent, world observations, and action execution makes the system easier to debug.
Simpler control loops are easier to reason about when several asynchronous systems must work together.
Typed decisions with confidence scores are easier to build a safe control loop around than free-form text.
Per-decision cost is a design constraint, not an afterthought. It decides whether an agent can run continuously or only occasionally.
A capable model still needs reliable tools and timely information to act effectively.
Accessibility does not necessarily require simplifying what a player can do, it can mean providing a different interface for expressing the same intent.

What's next for GameSpeak

Our next step is making GameSpeak faster, more responsive, and easier to use continuously. We want players to be able to speak naturally while playing, interrupt the agent at any moment with commands like "Stop," and immediately give it a new goal.

Ultimately, we want to move GameSpeak beyond a game built specifically for our system. The long-term goal is to create a general-purpose accessibility layer that can understand a game's state and translate natural-language intent into its existing keyboard, mouse, or controller inputs, allowing GameSpeak to work with virtually any game.