Holo Hangout!

A past live Twitch broadcast of HoloHangout, demonstrating real-time Twitch chat integration with a WebXR mixed-reality environment.
HoloHangout's Community Sandbox: Dynamic AI-powered 3D co-creation between Twitch streamers & viewers.
OpenAI Codex testing & validating HoloHangout's 3D interactions agentically using the Immersive Web SDK's XR emulator.
HoloHangout's system architecture: Twitch, WebXR, and Quest-to-OBS streaming.
HoloHangout's multiplayer Streamer vs. Chat challenge, showcasing live streamer-audience interaction on Twitch.

Inspiration

You've probably come across VR streamers on Twitch, sharing gameplay from immersive virtual worlds while their audiences watch from behind their screens with limited opportunities to actively participate.

As developers who've enjoyed building VR & AR applications ourselves, it got us thinking: What if Twitch viewers could actually become part of their favorite streamer's world for a moment? What if they could socialize, play, and create together in the same mixed reality space – without needing a VR headset or any additional hardware, just Twitch chat?

What It Does

Imagine watching your favorite VR streamer on Twitch and suddenly becoming a character inside their world, without ever putting on a headset. That's HoloHangout!

Using a VR headset and WebXR, streamers can launch their own shared mixed-reality space, connect their Twitch channel, and broadcast the experience live through OBS. Instead of simply reading chat, they can now reach out, grab, move, and physically interact with their viewers, represented as animated 3D characters in their real surroundings.

For viewers, Twitch chat becomes their controller! Without needing a VR headset, downloads, or additional hardware, viewers can type !join to enter the streamer's world as animated characters, control their avatars using commands like !wave and !dance, and play, socialize, or co-create with the community in real time.

What makes HoloHangout exciting is the two-way interaction between Twitch and the physical world. Viewers shape the mixed-reality experience through chat, while streamers physically interact with their avatars and creations. Everything is livestreamed back to Twitch so everyone can watch and participate in real time. Our goal is to give VR/XR streamers a fun, recurring experience where audiences aren't just watching, they're part of the action!

How It Works

HoloHangout bridges Twitch's real-time chat infrastructure with Meta Quest's mixed reality environment, to turn livestream viewers into animated 3D characters that exist in the streamer's physical space. Powered by Meta's Immersive Web SDK (IWSDK) and WebXR, the experience allows viewers to control their avatars through Twitch chat while streamers can physically grab, move, and interact with them using hand tracking.

1. From Twitch Chat to Interactive Mixed Reality

HoloHangout integrates Twitch OAuth, Helix APIs, and EventSub WebSockets with a Node.js backend to transform live chat messages into real-time spatial interactions.

When a viewer types !join, Twitch delivers the event to the backend, which identifies the viewer, assigns an avatar, and sends a structured WebSocket event to the IWSDK-powered WebXR runtime on Meta Quest. The viewer instantly becomes an animated 3D character inside the streamer's passthrough environment.

Commands like !wave, !dance, and !run control individual avatars, while IWSDK's Entity-Component-System (ECS) and hand-tracking GrabSystem allow the streamer to physically pick up, reposition, and release those same characters.

2. Interactive Streamer-Audience Experiences Beyond Chat

Twitch VR streamers can use HoloHangout as a recurring interactive segment in their livestreams, bringing their communities into shared mixed reality experiences that encourage engagement and give viewers a reason to return.

Here are some examples of the unique interactions possible in a shared mixed reality space:

Community Sandbox for Real-Time Co-Creation: Viewers can type !build <idea> to request a 3D creation. The streamer approves the request by physically holding that viewer's avatar. HoloHangout then uses Gemini to generate a structured voxel blueprint, validates its geometry, and assembles it into a grabbable 3D object inside the Quest environment.

Streamer vs. Chat Challenges: Viewers can launch virtual projectiles using Twitch chat commands, while the streamer fights back with a physically grabbable blaster in mixed reality, creating fun, playable challenges between the streamer and their audience.

Community Shoutouts: Streamers can give viewers personalized shoutouts by physically interacting with their avatars, creating memorable moments of recognition and connection with their community.

3. Challenge: Bringing the Actual Mixed Reality Experience to Twitch

One of our biggest engineering challenges was broadcasting the actual Quest mixed reality view, including camera passthrough, tracked hands, and virtual objects.

Instead of relying on desktop window capture, HoloHangout uses a custom Android companion application (Codex helped us build this!) that captures the Quest's composed display through the MediaProjection API, following system capture consent. The video is hardware-encoded as H.264 and transmitted over USB to a local receiver. FFmpeg then repackages the video into an OBS-compatible stream without re-encoding it.

OBS broadcasts the result to Twitch, where viewers can watch their characters interact with the streamer and send new commands. This completes a continuous loop: viewers influence the mixed reality world, and that same world is livestreamed back to them.

How We Used OpenAI Codex as an XR Developer & Tester

One of the coolest things Codex helped us do was actually run, interact with, and test our 3D mixed reality world! Using Meta's Immersive Web SDK and browser-based XR emulator, Codex launched a virtual Quest session, rendered and visually inspected our 3D environment, simulated hand gestures to grab and move virtual objects, and verified the interactions by inspecting live ECS state. It wasn't just writing XR code; it was operating, testing, and debugging the experience itself! (Watch our demo video to see Codex doing this in action.)

Beyond that, Codex helped engineer our entire real-time system, from Twitch OAuth and EventSub WebSockets to chat-controlled avatars and Quest-to-OBS streaming. When our original video capture kept freezing, Codex helped build a custom H.264 streaming pipeline, created automated video tests, diagnosed failures, and iteratively verified fixes. Using Codex through VS Code also made the entire workflow incredibly convenient, allowing us to build & iterate, without constantly switching between tools.

What We're Proud Of & Looking Forward

HoloHangout was a lot of fun to build as we challenged ourselves to push the boundaries of what's technically possible across different mediums, bringing Twitch and mixed reality together to make livestream audiences active participants in a streamer's physical world.

We're proud of combining real-time Twitch interactions, hand tracking, and 3D experiences into one working system. For larger streamers with thousands of viewers, we envision HoloHangout as an exclusive community segment, giving loyal community members special opportunities to join, play, and interact directly with their favorite streamer. We're excited to see how HoloHangout could grow with new interactive features, multiplayer experiences, and more ways for streamers to engage with their communities!