Canvas Tales

Inspiration

Kids want to create stories but can't draw. Adults have ideas but lack the tools to bring them to life visually. We asked: what if you could just talk and an AI would illustrate your story in real-time? No typing, no drawing skills, no complex tools — just your voice and imagination.

What it does

Canvas Tales is a fully voice-driven creative storytelling agent. An AI named Tali listens to you speak, responds with natural voice in real-time, and builds illustrated storybooks through a guided 4-phase workflow called StoryForge:

1. Concept — Describe your story idea by voice or tap a starter card. Tali helps pick genre, art style, and panel count through natural conversation.
2. Outline — Gemini generates a structured story outline with acts, scenes, and a character visual bible with detailed physical descriptions for consistency.
3. Build — Panels generate using Gemini's interleaved text + image output. Each API call returns narration text alongside the illustration in a single response. The first panel is passed as a reference image to maintain character consistency across the entire story.
4. Review — Swipe through panels while Tali narrates each scene aloud. Export the complete story as an image.

Beyond storytelling, Tali can see through your camera and describe what she sees, remember facts across sessions via Firestore, answer questions grounded by Google Search, generate standalone images on demand, and be controlled entirely by voice — 17 tools total.

How we built it

Frontend: Vanilla HTML/CSS/JS PWA — no frameworks. Four screens: StoryForge, Voice, Camera, Settings. Audio capture and playback via AudioWorklet for low-latency voice.

Backend: Node.js + Express + Socket.IO on Cloud Run. The session manager handles bidirectional audio streaming with Gemini Live API, routes 17 tools (sync and async), and manages StoryForge panel generation with reference image passing.

Key technical decisions:

• Interleaved output for StoryForge panels: each call to gemini-2.5-flash-image with responseModalities: ['TEXT', 'IMAGE'] returns narration + illustration together
• Character consistency via reference images: panel #1 is passed as inlineData to all subsequent panel generations
• Barge-in works natively through Gemini Live API — interrupt Tali mid-sentence and she stops immediately
• Google Search grounding on the Live API prevents hallucinations
• Kid safety guardrails in system prompt and generation prompts ensure all content is appropriate for ages 6+

Challenges we ran into

• Character consistency across panels was the hardest problem. We solved it by generating a detailed character visual bible in the outline phase and passing the first panel as a reference image to all subsequent generations.
• Voice narration flooding — initially, narrating every panel during build queued 4+ messages to the Live API, making Tali unresponsive. We moved narration to Review-only so the user stays in control.
• Rate limits on Gemini image generation required adding delays between panel builds and retry logic with exponential backoff.

Accomplishments that we're proud of

• True interleaved output: narration text and illustrations from a single Gemini response
• A kid can tap one starter card and get a complete illustrated storybook by just talking
• 17 voice-controlled tools — the entire app is usable without touching the screen
• 50 automated tests, CI/CD via Cloud Build, deployed on Cloud Run
• Barge-in that actually works — interrupt Tali anytime, she stops and listens

What we learned

• Gemini's native image generation with responseModalities: ['TEXT', 'IMAGE'] is powerful for creative applications — getting narration and artwork in one call changes the UX paradigm
• Reference image passing is essential for visual storytelling — without it, characters look different in every panel
• The Live API's bidirectional streaming makes voice interaction feel natural in a way that turn-based APIs cannot match
• Grounding via Google Search is critical for a kid-facing app — prevents the AI from making up facts

What's next for Canvas Tales

• Audio export — generate a narrated video of the complete story
• Collaborative stories — multiple users contribute scenes by voice
• Story continuation — "what happens next?" interactive branching
• More art styles and panel layouts
• Classroom mode for teachers to guide group storytelling

Built With

• cloud-firestore
• express.js
• gemini-2.5-flash-(vision-+-reasoning)
• gemini-2.5-flash-image-(interleaved-text-+-image)
• gemini-2.5-flash-native-audio-(live-api)
• google-artifact-registry
• google-cloud-build
• google-cloud-run
• google-genai-js-sdk
• google-search-grounding
• imagen-4.0-fast
• node.js