TaleSpark

Inspiration

The idea for TaleSpark came from a simple question: what if bedtime stories could create themselves? Not just text on a screen, but a full cinematic experience — narration, illustrations, and voice — all woven together in real time. When we saw the Gemini Live Agent Challenge focused on multimodal interleaved output, we knew this was the perfect opportunity to build an AI that thinks like a creative director, orchestrating text, images, and audio into a single cohesive storytelling flow.

We were inspired by the magic of children's storybooks and the emerging capabilities of generative AI. Traditional AI chatbots produce text. We wanted to build something that performs — generating a living, breathing story that unfolds before your eyes and ears simultaneously.

What it does

TaleSpark is an AI-powered interactive storybook generator. Users pick a genre (Fantasy, Sci-Fi, Mystery, Fairy Tale, or Adventure), provide a story prompt, and watch as the AI crafts a multi-scene narrative in real time. Each scene includes:

AI-generated illustrations via Imagen 3.0
Streaming narrative text with a typewriter effect
Voice narration synthesized with Google Cloud TTS

The experience is delivered through a polished, scroll-snap interface with Three.js particle effects, GSAP animations, genre-specific color theming, and a custom audio player — all designed to feel like flipping through a magical book.

How we built it

Architecture: The backend is a FastAPI server running two parallel async tasks coordinated via queues:

LLM Producer — Streams from Gemini 2.5 Pro with tool-calling. When Gemini decides a scene needs an illustration, it calls the generate_image tool, which triggers Imagen 3.0 to create a scene image. Text continues streaming after the image is generated.
TTS Worker — Consumes sentences as they're produced, synthesizes MP3 audio via Google Cloud Text-to-Speech, and pushes audio events back to the client.

Challenges we ran into

Synchronizing three output modalities in real time was the hardest problem. Text, images, and audio are generated at different speeds — Gemini streams text almost instantly, Imagen takes a few seconds per image, and TTS runs sentence-by-sentence. We needed to ensure scenes display in the correct order and that audio aligns with the visible text. Our solution was a scene-indexed event system with completion tracking: every event carries a sceneIndex, and a scene is only marked complete when both text generation ends and all pending TTS tasks finish.

Sentence boundary detection for TTS was trickier than expected. Gemini streams text in arbitrary chunks that don't respect sentence boundaries. We built a regex-based sentence splitter ([.?!。？！\n]) with a buffer to accumulate partial sentences across chunks, supporting both English and CJK punctuation.

Interleaved tool calling required careful state management. When Gemini pauses to call generate_image, we needed to handle the tool response, resume generation, and correctly increment the scene index — all while the TTS worker continues processing sentences from the previous scene.

Accomplishments that we're proud of

True real-time multimodal orchestration — Text, images, and audio generate in parallel and stream to the user as they're ready, rather than waiting for everything to finish. The dual-task async architecture with queue-based coordination makes the story feel alive as it unfolds.
The AI is the creative director — We don't hardcode when images appear. Gemini 2.5 Pro decides on its own when a scene needs an illustration by calling the generate_image tool mid-stream. The model genuinely directs the storytelling flow.
Audio-text synchronization — Getting sentence-level TTS to align with streamed text chunks was a gnarly problem. Our scene-indexed completion tracking system ensures narration and visible text stay perfectly in sync, even though they're produced at completely different speeds.
The UI feels magical, not technical — Three.js particles that react to your mouse, 3D-tilt genre cards, genre-specific color theming, typewriter text, horizontal page-flip navigation — it feels like opening a storybook, not using an AI tool. We're proud that the experience hides all the complexity behind something genuinely delightful.
Single SSE stream carries everything — One connection, four event types (text, image, audio, scene_complete), full progressive rendering. No WebSocket complexity, no polling, no waiting screens. The simplicity of this approach surprised us.
It actually works end-to-end — Three different Google Cloud APIs (Gemini, Imagen, TTS), two async workers, a queue system, SSE streaming, and a reactive Vue frontend — all coordinating flawlessly to produce a complete multimodal story from a single prompt.

What we learned

Gemini's tool-calling with streaming is powerful for orchestrating multimodal pipelines — the model itself decides when to generate images, acting as a true creative director
Async queue-based architectures are ideal for coordinating producers and consumers running at different speeds
SSE is underrated — for unidirectional real-time streaming, it's simpler and more reliable than WebSockets
Designing for progressive disclosure (showing content as it becomes available rather than waiting for everything) dramatically improves perceived performance