About the project
Inspiration Traditional visual novel development is gated by extensive resource requirements. To build a compelling interactive story, you typically need a team of writers, character artists, background illustrators, voice actors, and engine programmers. Commissioning custom assets costs thousands of dollars, and programming branching narratives takes months of intense labor. We wanted to shatter this "Creative Barrier." Inspired by the "Creative Storyteller" category of the Gemini Live Agent Challenge, we set out to build a personal AI game studio - an agent that acts as a creative director, weaving text, images, and audio into a cohesive, interactive experience.
What it does AIVN transforms a simple text prompt (a logline or synopsis) into a fully playable, interactive visual novel. You provide the premise and choose an art style, and AIVN automatically generates:
- A structured narrative outline with branching choices.
- Consistent character sprites featuring varied emotional expressions and poses.
- High-quality, stylistic background environments.
- Expressive, unique voice acting for every line of dialogue.
- A playable web client where you can instantly playtest the generated game.
How we built it The system is divided into a React/TypeScript frontend and a Python/FastAPI backend. The core of the application relies on the Google GenAI SDK to orchestrate multiple Gemini models:
- Story & Logic: We use Gemini 2.5 Flash with strict Pydantic schemas to generate the JSON structure of the game, including scenes, character definitions, dialogue lines, and branching choices.
- Art Direction: We utilize Gemini 3.0 and 3.1 Image Preview models to generate character sprites on solid backgrounds (which we dynamically make transparent using a background removal utility) and immersive background locations.
- Voice Acting: We use Gemini 2.5 Pro TTS to synthesize expressive audio for every generated dialogue line.
- Playback Engine: We built a custom "headless" visual novel engine in Python that maintains the game state and streams dialogue, audio, and visual assets to the React frontend in real-time via WebSockets.
Challenges we ran into
- Cost and Quota Bottlenecks: This was our biggest hurdle. We had Google Cloud credits to use via Vertex AI, but the rate limits and quotas for image generation on Vertex AI were prohibitively low for generating dozens of character poses and backgrounds simultaneously. To solve this, we implemented a custom dual-API routing strategy. We routed the high-volume text and audio requests through the Vertex AI API (utilizing our Cloud credits), and exclusively routed the heavy image generation tasks through the Gemini Developer API (paying out-of-pocket). This hybrid approach perfectly balanced our budget with the performance needed for the application.
- Character Consistency: Getting the image models to generate the exact same character in different poses (e.g., "angry," "blushing," "pointing") was difficult. We solved this by forcing the LLM to generate a highly rigid "base appearance" prompt for each character, which was prepended to every single pose generation request.
- Multimodal Synchronization: Chaining text, image, and audio generation required strict data contracts. If the text model generated a speaker name that didn't match the image generation pipeline, the game would break. We had to enforce strict JSON schemas and use fuzzy string matching to ensure assets linked correctly in the final screenplay.
Log in or sign up for Devpost to join the conversation.