DoodleSoul

Transforming children's drawings into interactive imaginary friends via Multimodal Orchestration.

Comment

🎨 DoodleSoul: Animating the Silence of Imaginative Worlds

💡 Inspiration

The inspiration for DoodleSoul came from a staggering reality in pediatric mental health: the "Clinical Blockade." For many children aged 3 to 12, especially those with Autism Spectrum Disorder (ASD), ADHD, or Selective Mutism, traditional "talk therapy" is perceived as a threat. In Brazil, the ratio of psychologists to students is only 0.05%, leaving millions of children without consistent support.

We asked: What if the child didn't have to talk to an adult? What if they could talk to their own imagination?

This sparked the concept of Technological Externalization. By using Google’s frontier AI to animate a child's drawing and grant it a voice, we transform a passive patient into an active storyteller. DoodleSoul isn't just a toy; it's a bridge between a child's inner world and clinical care.

🚀 What it does

DoodleSoul is a Dual-Audience Multimodal Platform that orchestrates complex AI pipelines to create a seamless therapeutic experience:

  • 📸 Single-Shot Drawing Intake: Captures physical drawings via camera.
  • 🧠 Persona Derivation: Uses "Gemini 3.1 Flash Lite" to analyze visual cues (colors, shapes, characters) and derive a unique voice and personality traits.
  • 🗣️ Real-Time Live Adventure: Powered by Gemini Live API (via ADK), providing a bidirectional, low-latency voice conversation that never breaks character.
  • 🎬 Living Scenery (Imagen-4 & Veo-3): Generates dynamic visual scenes. To solve the 45s latency of video generation, we use a cascading strategy: an immediate Imagen still with a Ken Burns CSS effect serves as an "illusion of life" while the Veo video renders in the background.
  • 🛡️ Silent Clinical Alarm: While the child plays, the AI silently monitors for crisis markers (anxiety, bullying). If detected, it triggers a report_clinical_alert to a therapist-only dashboard without interrupting the child's immersion.

🏗️ How we built it

We built a heterogeneous AI stack designed for zero-latency perception and high clinical safety:

🛠️ Technology Stack

  • Google Agent Development Kit (ADK): For real-time Live API orchestration.
  • "Gemini 3.1 Flash Lite": The brain handling voice, vision, and tool-calling.
  • Imagen-4 & Veo-3: For high-fidelity image and cinematic video generation.
  • FastAPI (Python 3.11): A custom full-duplex WebSocket bridge using asyncio.wait(FIRST_COMPLETED) for deterministic resource management.
  • React + TypeScript: Featuring AudioWorklet for raw PCM16 streaming and a custom CSS-only Ken Burns animation engine.
  • Google Cloud Run: Scalable, serverless deployment.
  • Google Cloud Logging: For immutable, PII-redacted audit evidence.

🎯 Challenges we ran into

1. Masking the "Veo Gap"

Generating high-quality video with Veo takes ~45 seconds. In a live voice session, this is an eternity. We solved this by implementing an Architectural Illusion: generating an Imagen still instantly, applying CSS animations for movement, and instructing the Gemini agent to keep the child verbally engaged with playful questions while the "magic" (Veo) was being prepared.

2. VAD Tuning for Pediatric Speech

Standard Voice Activity Detection is too aggressive for children who stutter or hesitate. we tuned the realtime_input_config to LOW sensitivity with a 500ms silence duration, ensuring the agent waits for the child to finish their thought, even with speech delays.

3. Dual-Audience Data Isolation

Ensuring that clinical data reaches the therapist without leaking into the child's audio channel required a custom Media Interceptor layer. This layer filters tool_call events, routing clinical signals to the store while keeping the voice channel 100% lúdico.

🏆 Accomplishments that we're proud of

🌟 Technical Achievements

  • Successfully implemented a Full-Duplex Bridge that handles binary audio and JSON events simultaneously.
  • Created a Cascading Rendering Pipeline that makes high-latency video generation feel instantaneous to the user.
  • Built a DLP Gatekeeper that automatically redacts PII before any therapeutic data is persisted.

💖 Clinical Impact

  • Validated the "Externalization" mechanism, allowing children to project emotions onto their drawings safely.
  • Provided therapists with a Silent Alarm system that captures real-time emotional shifts during play.

🧠 The Engineering Soul: What We Learned

This hackathon was more than a technical challenge; it was a deep dive into the Action Era of AI. Journey through hundreds of micro-decisions taught us that:

  1. Iteration is for Production, One-Shot is for Demos: Good software is the result of continuous refinement while the system is running, not a single macro-decision made before the first line is written. I embraced Extreme Programming principles—TDD, small releases, and continuous refactoring—as my brakes and steering.

  2. The Developer is the Bottleneck, not the AI: AI is a powerful accelerator, but acceleration without direction is just faster entropy. The agent writes the code, but the human decides which code to write. This decision requires experience that no prompt can replace. IA amplifies your skills. If you are a weak engineer, it just helps you generate tons of bad code faster. I experienced this firsthand at times while working on a project—not knowing what the next step should be simply because I didn’t have a clear understanding of the situation, so I had to take a few steps back and think. What am I not seeing here? Where am I right now? What do I need to do to find a solution to this problem?

  3. TDD Protects Evolution: Without tests, every merge is Russian roulette. I followed Dijkstra's wisdom: "Tests show the presence, not the absence, of bugs." TDD wasn't just a safety net, it was the engine that allowed me to refactor the media pipeline three times without breaking the core audio spine.

  4. Context is the Secret Sauce: AI doesn't learn between sessions, it is static and needs to be fed clear rules every time. I learned that the "Magic" isn't in the model alone, but in the precision of the context I provide.

  5. Architecture over Functionality: I constantly asked myself: "Is it more important that the system works, or that it is easy to change?" In the fast-moving world of frontier APIs, I chose the latter, building a modular bridge that could adapt to API shifts in minutes.

🔮 What's next for DoodleSoul

  • Cloud Persistence: Migrating from in-memory stores to Google Cloud Firestore.
  • GCS Integration: Storing Imagen/Veo assets in Google Cloud Storage with signed URLs.
  • Multi-Drawing Memory: Allowing the AI to remember characters from previous drawings to build long-term therapeutic rapport.

🚀 Created with passion for the #GeminiLiveAgentChallenge, Bridging imagination and healing through Multimodal AI.

Submitted to

Links

Built With

  • fastapi
  • google-adk
  • google-cloud-logging
  • google-cloud-run
  • google-genai
  • motion-react
  • react
  • tailwind-css
  • typescript
  • uvicorn
Share this project:

Updates

Submission history