OmniWeave

An immersive AI cinematic storytelling engine weaving realtime voice interaction, generated scripts, art directed illustrations, multicharacter narration, and dynamic atmospheric music.

Comment

Inspiration

I got tired of AI storytelling being a text box. You type something, you get paragraphs back, and that's it. No pictures, no voices, no music. I wanted to build something where you say "a samurai cat defending a moonlit temple" and actually get a cinematic experience illustrated scenes, distinct character voices, background music, the whole thing. Not as a gimmick, but as something you'd actually want to watch and listen to.

What it does

OmniWeave has two modes. In Compose Mode, you type a prompt and get back a 5-scene cinematic story in about 30 seconds each scene has a full illustration, multi-voice narration (an old spirit fox sounds different from a young warrior), and mood-reactive ambient music. The whole thing plays in a full-screen player that feels more like watching a short film than reading a chatbot response.

In Live Mode, you just talk. The AI responds with expressive voice acting while generating illustrations in the background. You can interrupt it, redirect the story, and it adapts naturally. Sessions don't time out because we compress the context window automatically.

Every story feeds a memory bank characters and locations persist, so if you mention the samurai cat in a new story next week, it remembers what he looks like.

How we built it

The backend runs on Cloud Run using Google ADK for TypeScript. There are 9 Gemini models doing different jobs one writes the script, another reviews it for consistency, a third generates images, another handles voice, another composes music, and so on. Cloud TTS Chirp 3 HD provides 28 distinct voices that get automatically assigned based on character names and gender detection.

The trickiest part was image consistency. The image model has no memory between calls, so we built a CHARACTER SHEET system where the story writer outputs exact physical descriptions, and every image prompt copies those descriptions verbatim. We also use multi-turn chat sessions so the model maintains visual context across scenes.

The frontend is React 19 with a cinema-themed design dark frame for controls, warm canvas for story content, Ken Burns animations on the images, particle effects that react to the story mood.

We use 8 Google Cloud services total: Cloud Run, Firestore, Firebase Storage, Firebase Auth, Firebase Hosting, Cloud TTS, Artifact Registry, and Cloud Build. Deployment is automated through a single shell script.

Challenges we ran into

Real-time audio in the browser is harder than it sounds. The native audio model won't call tools mid-conversation, so we had to build client-side orchestration that generates images without interrupting the voice stream. Cloud TTS only supports 2 speakers per API call, but our stories have 3-4 characters so we chunk the text into 2-speaker segments and stream them sequentially through a custom audio player for gapless playback.

The stale closure bug took hours. The audio auto-advance system calls itself recursively, but React closures capture state at render time. Audio played fine for scene one, then silently failed because the callback was reading an empty array from the initial render. The fix was embarrassingly simple once we found it mirror state into a ref.

What we learned

Using 9 specialized models instead of one general-purpose model produces dramatically better results. The write-then-review pipeline (StoryWriter + StoryReviewer) catches about 15% of stories with formatting issues that would break the voice system downstream. And the Live API with native audio changes the whole interaction it stops feeling like a chat and starts feeling like a collaboration.

What's next for OmniWeave

Camera input during Live Mode so you can hold up a sketch and say "make my story look like this." Interleaved generation where text and images stream together in a single call. Collaborative multi-user sessions. EPUB export.

Built With

Share this project:

Updates

Submission history