QUANTUNM 3D AI

Riko is an interactive 3D VRM AI companion bringing conversational intelligence to life with real-time WebGL rendering, neural voice lip-sync, and dynamic multimodal emotional expressions on the web.

Comment

🌌 Riko | Quantum: Building a 3D AI Companion That Actually Feels Alive

💡 Inspiration

Let’s be honest: chatting with AI in 2026 is amazing, but staring at a plain white text box gets boring fast.

We’ve had text chatbots for years now. When you talk to another human, you don't just read words—you look at their eyes, watch their facial expressions, hear the tone in their voice, and react to how they move. More than half of human communication is visual and emotional.

I wanted to change that. I wanted an AI that doesn't just reply with text, but actually exists in 3D space, looks at you, talks back out loud, and shows real emotions. And I wanted it to run right inside the browser without asking people to install a 50GB app or own a $3,000 gaming rig.

That’s why I built Riko | Quantum.

🧠 What It Actually Does

You open the website, and Riko is right there in full 3D:

  • She looks and moves naturally: She blinks, breathes, and tracks your cursor/head gaze. Her hair and clothes react naturally to movement using 3D physics.
  • She speaks and moves her mouth in sync: When you speak to her, she listens through your microphone, thinks, and replies with voice while her lips move in real time to match the sounds.
  • She shows real emotions: If you tell a joke, she smiles. If you ask a deep question, she pauses and looks thoughtful. If something unexpected happens, she looks surprised.
  • Zero downloads: Everything runs client-side in the browser on vmrchat.vercel.app.

🛠️ How I Built It

I built the whole app using a mix of Vue 3, Vite, Three.js, and WebGL.

1. The 3D Rendering & VRM Physics

I used Three.js along with the @pixiv/three-vrm loader so anyone can load humanoid 3D VRM avatars. To make the hair and clothes sway naturally instead of looking stiff, I hooked into the spring bone physics engine modeled as a damped oscillator:

[ m \ddot{\mathbf{x}} + \gamma \dot{\mathbf{x}} + k (\mathbf{x} - \mathbf{x}0) = \mathbf{F}{\text{ext}} ]

(Where (m) is the bone mass, (\gamma) is damping so hair doesn't shake forever, and (k) is the spring tension).

2. Real-Time Lip-Sync & Visemes

Getting the mouth to move naturally with audio was tricky. If you just open and close the jaw randomly, it looks like a cheap sock puppet. I mapped the incoming speech phonemes into 5 core Japanese/English visemes ((A, I, U, E, O)) and smoothed the blendshape transitions:

[ B_k(t) = (1 - \alpha) B_k(t - \Delta t) + \alpha V_{\text{target}} ]

With (\alpha = 1 - e^{-\frac{\Delta t}{\tau}}), the mouth morphs smoothly between sounds without sudden jarring snaps.

3. Latency Budget

Conversations only feel natural if the response is fast. I optimized the whole loop to stay under human conversation pause thresholds:

[ \tau_{\text{total}} = \tau_{\text{STT}} + \tau_{\text{LLM}} + \tau_{\text{TTS}} + \tau_{\text{render}} \le 450\text{ ms} ]

🧗 Challenges I Ran Into (And How I Fixed Them)

  1. The "Sock Puppet" Mouth Problem:
    At first, linking audio amplitude directly to mouth opening looked terrible. I had to rewrite the audio analyzer to calculate spectral energy across frequency bins and map them to actual phonetic vowel shapes.

  2. Mobile Frame Drops & Crashes:
    3D VRM models have thousands of polygons and high-res textures. On an iPhone or budget Android phone, the tab would lag or crash. I had to manually optimize the WebGL shaders, compress textures, and drop down bone calculations when running on mobile devices to lock in a clean 60 FPS.

  3. Keeping Audio & 3D Render Loops Synced:
    JavaScript is single-threaded, so heavy Three.js rendering would sometimes cause audio glitches or vice versa. I decoupled the audio viseme calculations from the main rendering tick so neither slows the other down.

🎓 What I Learned

  • WebGL & Shader Tuning: Building a 3D avatar that runs at 60 FPS in a web browser taught me way more about memory management, draw calls, and vertex shaders than any tutorial ever could.
  • Small Details Matter: It turns out adding tiny things like random micro-blinking, subtle eye saccades, and slight head tilts makes the character feel 10x more alive than just having perfect speech.

🚀 What's Next

  • Spatial Computing / VR: Porting Riko to WebXR so you can put on an Apple Vision Pro or Quest headset and have her standing in your room as a life-sized holographic assistant.
  • Camera / Vision: Letting Riko use your webcam (with permission) so she can see your face, smile back when you smile, and recognize objects you hold up.
  • Custom Avatar Uploads: Letting users drop in their own .vrm 3D files to customize their companion however they want.

Built With

Share this project:

Updates