Inspiration

Livestreaming is everywhere, but it’s mostly limited to flat cameras or game capture. Mixed Reality headsets like the Quest 3 open up something different: the ability to share not just your gameplay, but the world around you as seen through passthrough video.

We started wondering — what if you could take that view and bring it straight to Twitch? It might unlock new kinds of content: hands-free IRL streams, live cooking or DIY sessions, or maybe even guided tours where the audience can look through the creator’s eyes.

Of course, livestreaming the real world also brings challenges, especially around privacy. That’s why we built in automatic face-blurring and anonymization from the start — so experimenting with MR streaming can feel safe, both for creators and for the people around them.

What it does

LiveLens streams the Quest 3’s passthrough view directly to Twitch while automatically protecting bystanders:

• Real-time face anonymization — Detects faces in the headset view and applies pixelation/blur, or (in the future) playful AR masks on-device before frames ever leave the headset.
• Creator-first UX — Hands-free start/stop your stream & Twitch chat overlay

How we built it

Engine, Packages, Assets & Technologies:

• Unity
• Meta Quest SDK
• Passthrough Camera Access
• Unity Inference Engine (BlazeFace model)
• WebRTC
• Quest Camera Kit

Challenges we ran into

• Latency The WebcamTexture that we use to access the passthrough camera image has a few frames of delay, which makes it difficult to do actual real-time detection, even if the AI model runs fast enough
• Alignment: Even outside of this latency issue, aligning the detected bounding boxes to the actual passthrough view composited by the OS was more challenging than expected.

Built With

• blazeface
• meta-quest
• passsthrough-camera-access
• twitch
• unity
• unity-inference-engine
• unity-sentis
• webrtc