Gemini Ai

• 

  Storytelling

Inspiration

We wanted to move beyond static, text-based stories. With the release of the Gemini Multimodal Live API, we saw an opportunity to create a "living" storyteller—an agent that doesn't just respond to prompts, but listens to your voice, reacts to your emotions, and weaves a narrative in real-time. We wanted to build an experience that feels like sitting around a digital campfire.

What it does

​Storytelling is an interactive AI agent that crafts dynamic adventures on the fly. Users talk to the agent in real-time using natural speech. The agent uses Gemini’s low-latency voice capabilities to: ​Narrate stories with emotional depth and "affective dialog. ​Listen for user "barge-ins" (interruptions) to change the plot instantly. ​Incorporate real-world context into the story if the user shares their camera or screen.

How we built it

The project is built on a high-performance stack designed for low latency: Gemini 2.5 Flash / 3 Pro: Utilizing the gemini-live models for native audio streaming. WebSockets (WSS): To handle the bidirectional, stateful connection between the user and the AI. React & Tailwind CSS: For a sleek, atmospheric frontend that visualizes the mood of the story. Google Cloud / Vertex AI: Hosting the backend logic to manage session resumption and context window compression.

Challenges we ran into

The biggest hurdle was latency and interruption handling. In a real conversation, people talk over each other. We had to fine-tune the Voice Activity Detection (VAD) to ensure the agent didn't stop narrating at every small background noise, but did stop when the user actually wanted to change the story's direction. We also worked hard on context management.ensuring the agent remembered the World State even after 20 minutes of live conversation.

Accomplishments that we're proud of

Native Audio Integration: Getting the bidirectional 24kHz audio stream to feel truly "live" without the usual 2-3 second delay seen in standard LLMs. Adaptive Tone: We successfully implemented "Affective Dialog," where the AI's voice shifts from spooky and whispered during a mystery to excited and fast-paced during an action scene.

What we learned

We learned the intricacies of Stateful WebSockets and how different it is from standard REST API development. We also discovered how much "human-ness" is added when an AI can handle interruptions naturally—it completely changes the user's emotional connection to the story.

What's next for Gemini Ai

chatbots to Agentic Autonomy—meaning Gemini won't just tell stories; it will be able to take actions for you

Built With

• gemini-multimodal-live-api
• node.js
• react
• tailwind-css
• vertex-ai
• websockets