Vibe Canvas

Inspiration

Traditional pediatric distraction—like handing a child a tablet to watch videos—is passive and often fails the moment an injection actually happens. As a medical sciences student, I’ve seen how kids "check out" when they are entertained, but "tune in" when they are scared. We wanted to flip the script. Taking inspiration from the "Minecraft Factor"—where kids lose themselves in "Creative Mode"—we realized that active construction occupies more brain bandwidth than passive watching. Vibecanvas was born to move the child from a passive patient to an active architect, using AR to biologically inhibit pain signals by engaging the prefrontal cortex.

What it does

Vibecanvas is an AI-powered spatial sanctuary. Currently, the platform allows children to interact with a persistent AI Assistant to manifest their imagination into the physical room. Conversational Architect: Using a high-performance voice stack, kids can talk to an AI assistant to brainstorm ideas. Voice-to-3D: Say "Give me a friendly robot," and the system uses generative AI to spawn a 3D model directly into the XR environment. Web-to-XR Bridge: We have successfully deployed the web interface and model hosting, allowing the assistant to live-trigger 3D assets in real-time.

How we built it

We focused on a low-latency "Agentic" architecture to keep the magic alive: Orchestration: Pipecat acts as the glue for our voice-to-agent flow, with CI/CD and deployment handled via Railway. The Brain & Voice: Deepgram for lightning-fast STT, OpenAI/Grok for intent inference, and Cartesia for human-like TTS. The Assistant: AIAvatar powers the visual representation of our guide. The Workshop: Stability AI and Meshy 3D API for the procedural generation of 3D assets.

Challenges we ran into

The 3D Latency Gap: Generating high-quality 3D models in real-time is computationally heavy. We spent significant time optimizing the pipeline to ensure the "Wait time" didn't break the child's immersion. Noise in the Ward: Hospital rooms are rarely quiet. We had to work on NLP filtering to ensure the AI responds to the child’s intent rather than background medical equipment or clinician chatter. Deployment Sync: Getting the web-based assistant to communicate seamlessly with XR glasses required a robust Remote Service Module setup in Lens Studio.

Accomplishments that we're proud of

End-to-End Pipeline: We successfully reached the milestone where a child can speak a thought and see a 3D asset appear in their room via our deployed website. The Assistant's "Vibe": Creating an AI interaction that feels supportive and creative rather than "robotic" or clinical.

What we learned

Technical: Gained a deep understanding of connecting LLMs to spatial environments using Lens Studio and handling real-time asset deployment. Design: We realized that in healthcare AR, "User Agency" is the most effective form of anesthesia. When a child feels they have the power to change their environment, their stress levels plummet.

What's next for Vibe Canvas

Spatial Drawing & Editing: The next phase is moving beyond just "spawning" objects. We are building the ability for kids to physically "draw" in 3D space and edit the generated models using hand gestures. Full XR Interaction: Deepening the physics-based interaction so kids can play with the objects they create. Biometric "Calm" Mode: Integrating heart-rate sensors so the Vibecanvas environment shifts to cooler colors and slower music if it detects the child's heart rate rising during a procedure.