wevr

The AI production studio where creators direct living, playable cinema.

Comment

🎬 Inspiration

Every memorable interactive story relies on a complete production crew: a writer crafting dialogue, a set designer establishing the scene, a composer scoring the mood, and a director making sure choices made in Act I aren't forgotten by Act III.

For solo creators, tabletop storytellers, and indie writers, that crew simply doesn't exist. Traditional game engines demand heavy coding and complex tooling, while generic AI chatbots sit at the other extreme: amnesiac text windows with no durable memory, no persistent state, and no real rules.

We built wevr to turn that solo experience into a full AI-native production studio. Our goal was to give any writer an autonomous production crew powered by agentic AI:

  • Studio: Creators design living worlds, factions, characters, deterministic win/lose conditions, generated scene art, and an original score composed with Google Lyria, without writing a single line of code.
  • Play: Audiences experience that world turn by turn. Google Gemini takes the director's chair, streaming prose, generating per-turn scene illustrations, adjusting the musical mood live, and honoring the author's world rules through validated tool calling.

🛠️ How We Built It

We built wevr as a decoupled, multi-service architecture designed to separate standard application state from heavy AI orchestration and streaming.

1. System Architecture

Frontend (apps/frontend)

Built with React 18, Vite, TypeScript, Tailwind CSS, Zustand, and TanStack React Query.

It delivers two main surfaces:

Studio

A no-code authoring workspace for:

  • Entities
  • Factions
  • Facts
  • Conditions
  • End conditions
  • Cover art and music generation

It also features AERO, an AI co-designer sidebar that inspects scenario state and proposes atomic, reviewable changes such as:

  • Entities
  • Conditions
  • Plot hooks

Creators can apply suggestions with a single click.

Play

An atmospheric turn-by-turn console featuring:

  • Real-time Server-Sent Events (SSE) prose streaming
  • Inline scene illustrations
  • Dice rolls
  • Dynamic soundtrack crossfading
  • Spectator support

Core API (apps/core-api)

Built with Python, FastAPI, async SQLAlchemy, and PostgreSQL.

Responsible for:

  • Authentication
  • Scenario CRUD
  • Published game discovery
  • Reviews
  • Bookmarks
  • Background jobs for scenario asset generation

Turn Resolution Service (apps/turn-resolution-service)

A dedicated async Python/FastAPI service responsible solely for turn orchestration.

For every turn it:

  1. Loads the player's current state
  2. Retrieves relevant memory context
  3. Prompts Gemini
  4. Validates tool calls
  5. Triggers scene image generation
  6. Streams narrative output via SSE

2. Google Cloud & Partner Integrations

Google GenAI SDK (gemini-3.5-flash-lite)

Powers the live story narrator in the Turn Resolution Service by:

  • Streaming prose token-by-token
  • Driving gameplay logic through structured tool calls

Google GenAI SDK (gemini-3.1-flash-image)

Automatically generates a fresh visual scene illustration for every turn and renders it directly inside the player's narrative feed.

Vertex AI (Google Lyria lyria-002)

Generates original thematic musical scores for authored scenarios directly inside Studio.

Replit (Partner Integration)

Creators can build custom mini-games using our Replit starter kit template.

At runtime, wevr embeds the Replit game in a sandboxed iframe and executes a two-way handshake over the browser's postMessage API:

  • ready()
  • reportResult()

This allows minigame outcomes to directly update game state.

3. Persistent Memory Layer (HydraDB)

To eliminate LLM amnesia across long playthroughs, we developed a dedicated long-term memory layer combining:

  • PostgreSQL
  • pgvector
  • Graph storage

Bitemporal Fact Tracking

Tracks both:

  • Valid Time: When an event occurred in the narrative
  • Transaction Time: When the system recorded the event

This prevents narrative contradictions and enables consistent world state.

Template Memory Cloning

Scenarios are ingested into a base memory template once during authoring.

When a player starts a new playthrough:

  • The template space is cloned instantly
  • Individual playthrough memories remain isolated
  • Scaling remains efficient

⚡ Challenges We Faced

1. Eliminating State Hallucinations (Validate-Before-Apply)

LLMs prompted to simply "track health and inventory" inevitably hallucinate. They create non-existent items, ignore statutory limits, or overlook critical quest requirements.

How We Solved It

We established a strict Validate-Before-Apply boundary (ADR-4).

Gemini cannot mutate game state directly in prose.

Instead, it proposes structured tool calls such as:

  • update_entity_stat
  • inflict_condition
  • trigger_ending

The Turn Resolution Service validates these calls against author-defined Pydantic schemas and rules before writing changes to PostgreSQL.

If a tool call violates scenario rules, it is rejected safely without breaking the narrative.

2. Turn Latency and Multi-Model Coordination

A single player turn involves:

  • Memory retrieval
  • LLM reasoning
  • State updates
  • Scene image generation
  • Audio mood cues

Running these sequentially created unacceptable delays.

How We Solved It

We split the pipeline into prioritized synchronous and asynchronous paths.

  • Narrative begins streaming immediately via gemini-3.5-flash-lite
  • Scene image generation runs concurrently via gemini-3.1-flash-image
  • Generated images appear as soon as rendering completes
  • Memory indexing writes are batched rather than blocking active gameplay

3. Graceful Multimodal Degradation

Cutting-edge models and cloud quotas occasionally fail or encounter rate limits.

If music or image generation fails, gameplay should continue uninterrupted.

How We Solved It

We built safe fallback boundaries into generation clients.

  • Image generation failures return None
  • Narrative completion proceeds normally
  • Audio generation failures automatically fall back to curated ambient stems

The player's experience remains intact even when auxiliary services fail.

4. Dynamic Audio Crossfading in the Browser

Static loops quickly become repetitive, while abrupt transitions break immersion.

How We Solved It

Gemini tags narrative chunks with ambient mood cues such as:

[MOOD: calm]
[MOOD: melancholy]

The frontend Web Audio manager listens for these SSE events and performs smooth gain-curve crossfades between mood-matched tracks.

This keeps the soundtrack dynamically aligned with the story.

💡 What We Learned

1. Immersion Requires Multimodal Synergy, Not Just More Parameters

Fast, responsive text combined with contextual scene art and a synchronized soundtrack creates a far more convincing cinematic experience than a larger model generating text alone.

2. AI Works Best as a Co-Designer, Not an Autocomplete Engine

When building authoring tools like AERO, creators didn't want an AI that hijacked their story.

They wanted an assistant that:

  • Inspects the current world
  • Identifies gaps
  • Proposes concrete, reviewable cards

Creators remain in control while benefiting from AI-assisted design.

3. Hard System Boundaries Enable Creative Freedom

Allowing an LLM to improvise text freely while restricting its authority over game state through strict validation provides the best of both worlds:

  • Uninhibited roleplay and storytelling
  • Real, enforceable stakes

🚀 What's Next for wevr

Speech-to-Speech Character Dialogue

Add real-time voice input and character-specific dialogue generation so players can converse naturally with NPCs.

Playthrough Branching ("Forking")

Allow spectators to jump into any shared playthrough at critical decision points and explore alternate timelines.

Expanded Creator Ecosystem

Support:

  • Multi-act scenario campaigns
  • Collaborative co-authoring
  • A public community library for sharing custom Replit minigames

Built With

Share this project:

Updates

Submission history