FaceMOD

facemod avartar to mark , with expression blendshape
faceswap to a boy
faceswap to a girl
Mark loves this.

Inspiration

FaceMOD is built on a core idea: if reality is our ultimate game, then why can't we mod our own character? We were inspired by the concept of "face mods" in video games, where you can swap a character's appearance. We asked: what if we could do this live, to ourselves, in Mixed Reality? Our goal is to let users replace their live visage with a dynamic, expressive avatar, and explore visual face-swapping effects.

What it does

FaceMOD is a pioneering MR application for Meta Quest 3 that allows users to replace their real-time face in passthrough with a virtual avatar. It uses real-time computer vision to capture your facial expressions and perfectly map them onto the avatar's blend shapes, enabling live facial puppeteering. The application also explores real-time visual face-swapping.

How we built it

The system is a complex integration of three core technologies:

AI-Powered CV (MediaPipe): We integrated MediaPipe Face Landmarks to run real-time facial detection and landmark tracking directly on the Quest 3.

Passthrough Camera API (PCA): This provides the raw video feed to MediaPipe. The crucial challenge was in the precise coordinate system transformation between the PCA's camera space, MediaPipe's 2D/3D landmark output, and Unity's 3D world space to anchor the virtual face perfectly in the real world.

Avatar Animation Pipeline: The mapped landmarks drive a custom rig that controls a high-fidelity avatar's blend shapes in real-time.

Challenges we ran into

The primary challenge was the non-trivial integration of AI (MediaPipe), the Passthrough Camera API, and 3D rendering. Each operates in a different coordinate space. Ensuring stable, low-latency alignment of the virtual avatar with the user's real head position and orientation through precise coordinate transformation math was our biggest technical hurdle.

Accomplishments that we're proud of

We believe FaceMOD is likely one of the first applications to successfully replace a user's real-time face with a fully animated avatar in a consumer MR experience. While visual fidelity has room for improvement, we have conclusively proven the technical feasibility of seamless, real-time AI-driven avatar replacement using the Passthrough Camera API on a standalone device.

What we learned

This project was a deep dive into integrating on-device AI models with the Passthrough Camera API. The key learning was mastering the pipeline of coordinate system transformations required to bridge the 2D camera feed, 3D CV landmarks, and the final 3D MR scene. We learned how to make AI "see" the world through the headset's cameras and correctly place its output back into that world.

What's next for FaceMOD

Our roadmap focuses on community, next-gen AI, and expanded functionality:

UGC & Avatar Platform ("FaceMOD Studio"): Allow creators to design, rig, and share their own avatars and face assets.

AIGC & Real-Time Face Generation: Explore diffusion-based models for photorealistic, identity-preserving face swaps and generation in real-time.

Expanded Feature Set: Develop new modes like "FaceClone" for realistic persona replication and explore emotional recognition to drive more nuanced avatar reactions.