GrimDM

GrimDM

## Inspiration My first game was Zork. Text adventures have always been the undercurrent of my life — that feeling of typing into a void and having a world respond. GrimDM is what happens when you replace the keyboard with your voice and the parser with Gemini.

## What it does GrimDM is a real-time voice AI Dungeon Master for a gothic fantasy RPG. Players speak naturally — no menus, no buttons, no typing. Gemini 2.5 Flash listens through native audio streaming, narrates the world, rolls dice, tracks quests and inventory, runs combat, and responds to whatever the player says. NPCs speak in their own distinct AI-generated voices. Imagen 4 paints the scenes as you explore. Music shifts with the mood.

## How we built it

Google ADK for bidirectional audio streaming with Gemini 2.5 Flash Native Audio
28 registered tools that Gemini calls mid-conversation — dice rolls, combat, movement, quests, NPC relationships, inventory, XP
3-channel browser audio mixing — DM voice (PCM WebSocket), NPC voices (async TTS), and ambient music/SFX all play simultaneously
Imagen 4 generates dark fantasy scene art on the fly via tool calls
Qwen3-TTS gives each of the 10 NPCs a unique voice from text descriptions
SQLite session persistence so your character survives disconnects and server restarts
FastAPI + uvicorn backend with vanilla HTML/CSS/JS frontend — zero framework dependencies

## Challenges we ran into

Gemini drops connections when too many tools fire at once. We learned to cap tool calls at 2-3 per turn — any more and the native audio bidi stream crashes with 1011 errors.
Audio resampling in the browser. Web Audio API ignores sampleRate: 16000 — we had to manually resample from 48kHz to 16kHz in an AudioWorklet.
Gemini hallucinates tool parameters. It invents session IDs instead of using real ones. We built a server-side session context module that overrides hallucinated values.
Large payloads crash the WebSocket. Scene images sent inline would kill the connection. We moved to async generation with HTTP polling delivery.
ADK sends thinking text twice — once marked as thought, once without. Server-side dedup filter catches both.

## Accomplishments that we're proud of

Sub-second voice latency — it genuinely feels like talking to a person, not waiting for a response
The Prologue — a guided 3-minute cold open that showcases every feature: voice, combat, NPC dialogue, music, scene art, and quests
NPCs with their own voices — Bramble the forest guardian sounds completely different from Marta the innkeeper
It just works in a browser — no app install, no plugins, open the URL and start talking

## What we learned Gemini's native audio mode is incredibly powerful but fragile at the edges. The sweet spot is keeping tool calls tight (2-3 per turn), responses short, and asset delivery async. The ADK abstracts a lot of complexity but you still need to understand what's happening underneath — especially around session management and connection lifecycle.

## What's next for GrimDM

Multiplayer — players interact through the DM (Gemini is the proxy). No direct player-to-player chat — everything is narrated in-world.
Gesture battles — rock-paper-scissors via camera during combat. The vision model sees your hand and resolves the outcome.
Voice cloning — capture the player's voice and let NPCs reference it ("You sound like someone I once knew...")
Vertex AI integration — blend the player's real environment into scene art using Imagen's image editing API
Mobile-first UI — the game is playable on phones today but the interface needs polish for touch