VibeStory

processing_page
orchestration
video_page
landing_page
recent_masterpieces
CI_CD

Inspiration

Storytelling has always been a living, breathing act — two voices weaving a world together in real-time. Yet every AI creative tool we've built reduces it to a vending machine: type a prompt, wait, receive a static output, repeat. The magic of collaborative imagination — the kind that happens when a child says "what if the dragon was blue?" and the story shifts instantly — is nowhere to be found.

Vibe Story was born from a simple conviction: the future of content creation isn't better prompts — it's better conversations. We set out to build an AI agent that doesn't just execute instructions, but sits across from you like a creative partner — listening, challenging, improvising — and together, through natural voice dialogue, co-creates a fully animated cinematic story. No keyboard. No timeline. No editing suite. Just your voice and a world that renders itself around your words.

How We Built It — ReAct Orchestration with ADK and Gemini

The core technical insight behind Vibe Story is agentic orchestration: rather than a monolithic model call, the system decomposes the creative pipeline into a network of specialized AI agents, each with a single responsibility, coordinated by the Agent Development Kit (ADK) using the ReAct (Reason and Act) cognitive loop.

The flow begins with Gemini Live (gemini-live-2.5-flash-native-audio), which opens a full-duplex, real-time audio channel between the user and the AI. This isn't turn-based — the user can interrupt, redirect, and riff mid-sentence. The accumulated transcript becomes the raw material of the story.

Once the conversation ends, the transcript enters a five-phase agent pipeline:

Story Agent (Gemini 3 Flash) decomposes the transcript into a structured breakdown — scenes, characters, emotional arcs, and visual cues — using structured output parsing.
Phase 1 runs in parallel: the Narration Agent generates professional voice-over with word-level timestamps, while the Character Agent (Nano Banana 2) produces multi-angle reference sheets for visual consistency.
Scene Prompt Agent (Gemini 3 Flash) synthesizes detailed image and video prompts per sub-scene, informed by character references and narrative context.
Scene Image Agent (Nano Banana 2) renders high-resolution scene illustrations; Scene Video Agent (Veo) transforms each image into a cinematic video clip with reference-guided motion.
Compile Agent assembles narration, video, subtitles, and score into a polished final cut via FFmpeg.

Each agent operates within the ADK's ReAct framework: it reasons about the current pipeline state, decides its next action, executes, observes the result, and adapts. If Veo returns a safety block, the agent reasons about the failure and falls back to an alternative generation path — no human intervention required.

The killer feature is the voice-driven edit loop. After watching the generated story, the user enters a second Gemini Live conversation to describe changes. An Edit Agent analyzes the edit transcript, produces a structured EditPlan, and propagates "dirtiness" through a dependency graph — surgically regenerating only the affected assets. Say "make the castle darker in scene 2" and only that scene's image, video, and the final compilation are redone. Everything else stays untouched.

What We Learned — Low-Latency Multimodal Synchronization

Building a system where voice, text, images, and video must converge into a coherent output under time pressure taught us hard lessons:

Parallelism is survival. Narration and character generation run concurrently. Scene images fan out across all scenes simultaneously. Without aggressive parallelism, the pipeline would take 10x longer.
Dependency gating is non-trivial. Video generation depends on images, which depend on prompts, which depend on the story breakdown. The orchestrator must track a live DAG (directed acyclic graph) and only release downstream agents when their inputs are ready.
Retry semantics differ per modality. Image generation failures are fast and cheap to retry; video generation failures are slow and expensive. The retry strategy must be modality-aware, with exponential backoff for heavy assets and immediate retry for lightweight ones.
Streaming progress is a UX requirement, not a nice-to-have. Users need to see the pipeline advancing in real-time via WebSocket — without it, a 3-minute generation feels like an eternity.

Challenges — The Latency-Quality Tradeoff

The central engineering tension in any real-time multimodal system is the Latency-Quality Tradeoff. Every additional modality adds generation time, but skipping any modality degrades the experience. We modeled this as:

$$L \approx \sum_{i=1}^{n} \frac{M_i}{B} + \delta$$

where:

( L ) is the total end-to-end latency from conversation end to final video delivery,
( M_i ) represents the computational cost of the ( i )-th multimodal asset (image, video clip, audio segment),
( B ) is the effective throughput (parallelism factor × per-model speed),
( n ) is the total number of assets to generate, and
( \delta ) is the reasoning overhead — the time the ADK agents spend in the ReAct loop deciding what to generate and how to recover from failures.

Minimizing ( L ) without sacrificing quality required three strategies:

Maximizing ( B ): Running independent agents in parallel and using concurrent API calls within each agent.
Minimizing ( \delta ): Keeping agent reasoning chains tight with well-structured prompts and pre-validated output schemas.
Reducing ( n ) during edits: The dependency graph ensures that only "dirty" assets are regenerated — turning a full ( n )-asset pipeline into a sparse subset.

The result: a full story generates in 2–4 minutes; an edit regenerates in under 60 seconds.