EchoStory

architecture
user interface

Inspiration

The inspiration for EchoStory came from a profound challenge faced by millions of families worldwide: the communication gap for children with autism. Many autistic children experience a rich internal world but struggle to express their daily experiences verbally. Questions like "How was your day?" can feel overwhelming and high-pressure, leading to frustration for both children and parents.

We were inspired by the proven success of visual communication systems like PECS (Picture Exchange Communication System), which shows that combining visual and verbal channels dramatically improves communication for autistic children. However, existing tools are static, manual, and lack the spontaneity needed for natural storytelling.

With Google's Gemini 2.0 Flash Live API and its multimodal capabilities, we saw an opportunity to create something transformative: a real-time, low-pressure AI companion that turns fragmented words and gestures into beautiful, interactive stories. EchoStory doesn't demand performance—it celebrates any input, making communication joyful rather than stressful.

What it does

EchoStory is a multimodal AI storytelling companion that helps autistic children share their daily experiences through real-time voice, vision, and visual generation.

Core Features:

1. Real-Time Conversational Co-Creation (Hear & Speak)

Uses Gemini Live API for fluid, low-latency voice interaction
The AI acts as a "gentle playmate" with scaffolded prompts: offering choices instead of open-ended questions
Supports interruption mid-conversation, allowing children to change story direction naturally

2. Interleaved Multimodal Output

As the child speaks, the system streams text, voice narration, and AI-generated illustrations simultaneously
Text appears instantly, followed by images within seconds—not a final "submit and wait" experience
This creates an engaging, real-time creative flow

3. Live Vision Integration (See)

Uses Gemini's Live Vision capabilities to detect physical objects through the camera
When a child shows a toy or drawing, the AI seamlessly weaves it into the story: "I see your teddy bear! Should he join the adventure?"
Reduces the pressure to verbalize by accepting visual input

4. Emotion-Aware Design

Analyzes tone and semantic cues to suggest emotional labels (happy, worried, calm)
Provides empathetic reflection: "That sounds like it was very loud. It's okay to feel worried."
Background colors adapt to detected emotions for sensory-friendly feedback

5. Daily Story Journal

Each session creates a visual storybook saved to Google Cloud
Parents can review illustrated stories with their child, transforming "How was school?" into "Tell me about this picture of you and the puppy!"
Stored in Firestore for persistence and future recall

How we built it

Architecture Overview

Frontend: Next.js 14 + React + TypeScript

Built a sensory-friendly UI with gentle animations using Framer Motion
Implemented WebSocket client for real-time bidirectional communication
Used Web Audio API for microphone input and Web Speech API for text-to-speech
Camera integration via getUserMedia for live vision features

Backend: FastAPI + Python 3.11

WebSocket server handling real-time multimodal streams
Asynchronous processing for low-latency responses

Google Cloud Infrastructure

Gemini 2.0 Flash Live API: Core multimodal AI for voice, vision, and text processing
Vertex AI:
- Gemini for conversational intelligence with custom system instructions ("Gentle Playmate" persona)
- Imagen 3 for generating child-friendly story illustrations
Cloud Storage: Hosting generated images with public URLs
Firestore: Storing story sessions and segments for the daily journal
Cloud Run: Containerized deployment for scalable backend hosting

Key Technical Innovations

1. Interleaved Output Pipeline

When Gemini responds with text, we immediately display it and start voice synthesis
Simultaneously, if Gemini calls the generate_illustration function, we trigger Imagen 3 in parallel
Frontend updates the story segment with the image when ready (2-5 seconds later)
This creates a "living canvas" effect rather than turn-based interaction

2. Function Calling for Scene Generation

Defined custom tool: generate_illustration(description, scene_type)
Gemini decides when to trigger image generation based on story context
System instruction guides AI to call illustrations at key narrative moments

3. Low-Demand Interaction Design

System instruction emphasizes validation over correction
Accepts any input: single words, sounds, gestures (via vision)
Never pressures the child to produce specific outputs

4. Real-Time Vision Processing

Captures video frames every 3 seconds via canvas
Sends frames to Gemini Vision API for object detection
Detected objects naturally incorporated into ongoing story

Challenges we ran into

1. WebSocket Connection Stability

Challenge: React strict mode caused duplicate WebSocket connections, leading to message duplication
Solution: Disabled strict mode and implemented connection state tracking with connectingRef to prevent race conditions

2. Synchronizing Multimodal Output

Challenge: Text, voice, and images arriving at different times created a disjointed experience
Solution: Implemented a staged rendering pipeline:
- Text displays immediately (300ms animation)
- Voice starts 200ms later (synchronized with text)
- Image fades in when ready (600ms animation)

3. Camera Permission and Stream Management

Challenge: Video element not found when trying to attach MediaStream
Solution: Used streamRef to store stream before element renders, then connected in useEffect after DOM update

4. Preventing UI Flashing

Challenge: Initial implementation sent audio chunks every 500ms, causing constant UI updates
Solution: Changed to send only final audio chunk when recording stops, preventing intermediate flickering

5. Regional API Availability

Challenge: Gemini API not available in all regions during development
Solution: Designed system with clear separation of concerns, allowing easy switching between real API and testing configurations

Accomplishments that we're proud of

1. True Multimodal Real-Time Experience

Successfully implemented Gemini Live API's full See-Hear-Speak capabilities
Achieved genuine interleaved output where text, voice, and images stream naturally
Not just a chatbot with images—a living, breathing storytelling canvas

2. Mission-Driven Design with Real Impact

Built a tool that directly addresses a genuine need in the autism community
Implemented evidence-based principles (low-demand interaction, visual scaffolding)
Created an experience that reduces communication anxiety rather than adding pressure

What we learned

1. The Power of Multimodal AI

Gemini Live API's ability to seamlessly switch between voice, vision, and text opens entirely new interaction paradigms
Multimodal input reduces barriers: children can speak, show objects, or gesture
Real-time processing feels fundamentally different from turn-based systems

2. Latency Matters More Than We Thought

Even 500ms delays significantly impact perceived naturalness
Interleaved output (showing text immediately while images generate) maintains engagement
Strategic use of loading indicators ("Creating magic...") turns wait time into anticipation

What's next for EchoStory

1. Enhanced Visual Generation with Veo

Integrate Google's Veo model to generate short 3-5 second animated video clips at key story moments
Transform static illustrations into living scenes with subtle motion
Example: A generated puppy could wag its tail, grass could sway in the wind

2. Parent Dashboard & Analytics

Build a parent portal showing:
- Daily story journal archives
- Emotional pattern tracking over time (identifying triggers or growth)
- Communication milestones (vocabulary expansion, story complexity)
Privacy-first: all data encrypted, parental control over storage

3. Therapeutic Integration