Animo

🚀 Inspiration 3D animation usually takes hours of rigging, keyframing, and heavy software. We built Animo to turn 3D storytelling into an instantaneous, natural process. By pairing large language models with real time text to motion generation, Animo lets anyone direct multi character scenes step by step or chat directly with interactive 3D avatars straight from the browser.

💡 What it does Animo features two distinct modes running on a single engine:

• Create Mode: Describe a scene and direct it beat by beat. Add characters, introduce objects, trigger interactions, and watch the entire composition assemble live on a shared playback clock.
• Chat Mode: Talk directly to an on screen character who speaks aloud, gestures, and physically demonstrates actions on command (like teaching a squat) using integrated ElevenLabs voice synthesis.

🛠️ How we built it Animo combines a lightweight static browser frontend with a high performance Python GPU backend:

• Scene Director (Gemini): Gemini handles scene decisions. It reads user prompts and outputs strict JSON defining stage layouts, prop placements, lighting, and timeline instructions without handling heavy rendering.
• Motion Engine (NVIDIA Kimodo and Llama 3): The backend hosts NVIDIA Kimodo on a GPU server (deployed via Vultr / RunPod), utilizing Meta-Llama-3-8B-Instruct as the primary text encoder to translate instructions into raw BVH motion capture data.
• Browser Spatial Engine (three.js and three-vrm): The browser loads BVH motion clips and retargets them directly onto VRM character models. A custom JavaScript layer handles spatial math, calculates spawn offsets, clamps travel distances, and smooths pose transitions over a 0.4 second crossfade window.
• Export Engine: WebM video recording captures the canvas with camera orbit support, pairing the output with optional Gemini generated background audio.

⚡ Challenges we ran into

• Coordinating Isolated Motion Clips: Text to motion models generate single body movements in isolation without spatial awareness. We wrote custom scene logic in JavaScript to cap travel distances, assign spawn points outside prop footprints, and reorient characters toward each other so independent motion files read as real interactions.
• Cross-Rig Axis Alignment: Bone rest axes vary between VRM models. Instead of running heavy per-frame rotational calculations, we skinned VRM meshes directly onto the BVH skeleton hierarchy to keep browser performance at 60 FPS.
• Scene State Persistence: Managing independent camera states, timeline event rows, and animation blend trees between Create and Chat modes while maintaining a single playback engine.

🏅 Accomplishments that we're proud of ✅ Built an end-to-end text-to-3D animation pipeline running live inside the browser with no build step required on the frontend. ✅ Solved multi-character positioning and interaction alignment entirely in client-side JavaScript without requiring multi-person model training. ✅ Integrated fallback motion support so the application remains functional even when disconnected from the GPU backend. ✅ Delivered real-time character voice response and automated WebM video rendering with full soundtrack synchronization.

📚 What we learned

• Offloading spatial math, layout calculations, and rendering to client-side JavaScript while restricting LLMs strictly to structured JSON decisions yields massive performance gains.
• How to retarget motion capture data onto VRM avatars directly in web-native graphics pipelines.
• Techniques for synchronizing decoupled AI models (text-to-motion, LLM scene planning, and text-to-speech) under a single master clock.

🚀 What's next for Animo ✨ Expand the prop and avatar library with custom user uploads. ✨ Implement multi-user persistent memory (Backboard) and cloud project saving (TigerData). ✨ Introduce multi-character conversational Chat mode where multiple AI avatars interact with each other and the user simultaneously.

Built With

• gemini
• javascript
• kimodo
• python