motion-engine

Empathic avatar gemini live (no dedicated GPU)

Comment

Inspiration

We tried asking Gemini Live to animate a 3D avatar while holding a conversation — generating gestures on the fly via system prompt. It failed: the model got distracted, tool calls threw errors on the preview model, and animation generation was too slow for real-time.

That failure gave us a principle: don't make the LLM do work it doesn't need to do. Every task we move off the model means fewer tokens, lower latency, and better conversations.

What it does

MotionEngine is a plugin for TalkingHead that gives LLM-driven 3D avatars a rich body language — without burdening the LLM.

Instead of making Gemini reason about animation data, we give it a dictionary of 137+ named motions. The LLM just names the gesture it wants inside its speech using lightweight markers. The frontend picks up those markers from the transcription and animates the avatar instantly. The model never breaks its conversational flow.

For moments when the avatar is listening (not speaking), we go further: a local algorithm using MediaPipe reads the user's facial expressions through the webcam and generates empathic avatar responses — nodding, smiling, mirroring emotional states — all on the client, with zero LLM involvement.

Two avatar states, one seamless experience: | | Animation source | Uses the LLM? | |---|---|---| | Speaking | Markers in speech | Yes (gesture names only) | | Listening | Empathic vision (local) | No |

Use cases: healthcare assistants, digital sales kiosks, AI companions, empathic robotics — any scenario where an AI needs expressive body language beyond a text box.

How we built it

Markers instead of tool calls — We embed ::marker:: tokens in the LLM's speech stream. The model doesn't read them aloud. The frontend detects them via regex in the transcription, triggers the animation, and strips them from the output. No tool calls, no latency, no interruption.

Motion dictionary — 137+ gestures defined as JSON data. A multi-track system (pose, mood, action) lets persistent moods like thinking stay active while temporal actions like nod_yes play on top.

Empathic vision — MediaPipe extracts facial blendshapes from the webcam. A lightweight classifier maps them to emotional states. In empathic mode, the avatar doesn't clone the user's face — it reacts naturally: a soft smile, a nod, a tilt of the head.

LLM context injection — getLLMContext() generates a compact motion catalog for system prompts, keeping token overhead minimal.

Built with vanilla JavaScript. The reference implementation (doacam.com) uses Gemini Live API via the Google GenAI SDK, deployed on Google Cloud Run + Firebase Hosting.

Technical details and full API documentation: README.md

Challenges we ran into

LLM as animator was a dead end — Tool calls broke the real-time flow. The model hallucinated morph target names and took too long. We had to rethink the entire approach.

Keeping markers invisible — Getting Gemini to embed markers consistently without reading them aloud required careful prompt engineering. Too many and the model narrates them; too few and the avatar feels lifeless.

Empathy, not cloning — Mirroring the user's face 1:1 looked uncanny. Designing attenuated reactions with complementary gestures (the _react schema) made it feel natural.

What we learned

Delegate everything you can. Moving expression detection, empathic responses, and head tracking to the client made conversations faster and more natural — without changing the model.

The interface transforms the experience. The same Gemini model powers every chatbot. Wrapping it in a reactive avatar with body language creates something emotionally different.

Markers beat tool calls for real-time. In-stream markers keep animations coupled to the exact moment in speech where they belong.

What's next for MotionEngine

On-demand vision — Stop sending webcam frames while the avatar speaks. Let the LLM request images only when it needs them. Empathic vision handles the rest locally.

On-device micro-LLM — Delegate animation decisions to a small local model like Gemma, pushing the philosophy even further: the main LLM only talks, everything else runs on the device.

Beyond avatars — The same semantic motion layer could drive physical robots. Empathic reactions, gesture vocabularies, and marker-based animation apply to servos and actuators just as they do to 3D meshes.

Open SDK — Package the full pipeline (TalkingHead + MotionEngine + FaceMirror) as a standalone toolkit for developers.

Built With

Share this project:

Updates

Submission history