🧞 FableGenie: The Multimodal Interactive Storyteller

Category: Creative Storyteller

💡 Inspiration

We wanted to bring classic moral fables (like the Panchatantra) into the AI era. Instead of a text-to-speech wrapper reading static scripts, FableGenie is a fully generative engine where every story is unique, shaped in real-time by your voice, environment, and physical gestures.

⚙️ What It Does

FableGenie creates a seamless 6-phase multimodal experience:

1. Voice Setup: Tell Gemini Live what story you want.
2. The Hook: The agent "reads the room" via webcam and personalizes the greeting.
3. Generative Narrative: A fresh story streams with inline AI illustrations and ambient music. You can interrupt with voice questions anytime.
4. Gesture Branch: The story pauses at a climax. Your physical gesture (e.g., thumbs up) decides the outcome.
5. Resolution: A cinematic Veo clip plays while the unique ending is narrated.
6. Moral Reflection: The final lesson ties back to what Gemini noticed in your room.

🛠️ How We Built It

We orchestrated 6 Google AI models through Vertex AI (us-central1):

• Gemini 2.5 Flash Live: Story setup, room-aware greeting, and gesture detection.
• Gemini 2.5 Pro: The core narrative engine.
• Imagen 4 & Lyria: Inline scene illustrations and ambient music loops.
• Gemini 2.5 Flash TTS: Overlapping spoken audio synthesis.
• Veo 3.1: Pre-generated cinematic resolution clips.

Infrastructure: Pure Vanilla JS frontend (WebRTC/Web Audio) communicating with a Node.js 20 backend. Hosted on Google Cloud Run using WebSockets, with heavy media cached in Cloud Storage (GCS), and fully automated via Terraform.

🧗 Challenges & Accomplishments

Our biggest hurdle was parsing the Gemini 2.5 Pro text stream. Because routing tags like [IMAGE: fox] could be sliced across WebSocket chunks, we built a custom buffer to parse and route tags flawlessly without interrupting the TTS audio. We also had to engineer seamless hot-swaps of Gemini Live's system prompt to switch from "storyteller" to strict "JSON gesture-detection" mode on the fly.

We are incredibly proud that nothing is pre-written. FableGenie successfully blends WebRTC, complex stream manipulation, and 6 generative models into a single, uninterrupted, production-ready experience.

Built With

• canvas-api
• cloud-run
• cloud-storage-(gcs)
• cloud-storage-sdk
• container-registry
• css
• docker
• dotenv
• express.js
• gcloud-cli
• gemini-2.5-flash-live
• gemini-2.5-flash-tts
• gemini-2.5-pro
• genaisdk
• google-cloud-platform-(gcp)
• google-fonts-(outfit
• googlefonts
• hcl-(terraform)
• html
• imagen-4
• javascript
• lyria
• node.js
• npm
• powershell
• shell-(bash)
• terraform
• veo-3.1
• vertex-ai
• vertex-ai-sdk
• web-audio-api
• webrtc-api
• websocket-api-(ws)