Playable Livestream

Unity
Realtime Rendering

Inspiration

As a VTuber and livestream creator with a growing community (2.6k followers on Twitch and over 190k followers on TikTok), I’ve always believed that streaming should be more than passive viewing. My goal was simple: bring my audience into my 3D world and let them truly participate. This idea became the foundation for Playable Livestream, an experiment that transforms traditional livestreams into shared, immersive spaces where viewers join as real-time avatars and shape the experience together.

The inspiration came directly from livestream culture itself. While platforms like Twitch and TikTok continue to grow, most streams remain fundamentally passive. Viewers type in chat, send emojis, or react — but they never step inside the experience. As a creator with a background in Unity, VR, and community-driven content, I wanted to break that barrier. I wanted viewers to stop being names in a chat and instead become visible, active participants inside the scene. This aligns strongly with Meta’s vision of real-time presence and shared immersive spaces.

What it does

The system I built in Unity uses a real-time integration layer that connects Twitch and TikTok with an in-world avatar system.

Key components include:

Unity-based 3D “livestudios” designed to feel like playful and immersive virtual sets
Custom chat commands allow viewers to spawn as small characters (currently snails) inside the scene
Real-time input parsing from chat messages from Twitch, YouTube and TikTok
Event-driven mechanics, such as triggering buttons, lights, fireworks
Customization of the viewer’s avatar (color changes, hats, etc.)
VR integration, allowing me as the VTuber to experience my own stream from inside the virtual world
Experimental viewer VR mode, where viewers will be able to join the livestream directly with a Meta Quest headset and appear inside the shared world

A working prototype has already been showcased publicly on Austria's FM4 “Gamingroom,” where viewers spawned inside my livestream as interactive avatars and explored the virtual space together.

How I built it

I built Playable Livestream as a modular Unity-based system that connects livestream platforms with a fully interactive 3D world. The core architecture consists of three layers: input layer, interaction layer, and presentation layer, all working together in real time.

Challenges we ran into

Balancing 2D Stream UX with 3D In-World Interaction
A key challenge was designing mechanics that feel meaningful for viewers typing commands and for future VR participants. Movements that felt intuitive in VR could overwhelm the 2D stream visually, so I had to create a shared interaction layer that keeps the world readable and entertaining from the main camera perspective.

Performance & Scalability
Because every viewer can spawn an avatar, performance optimization became critical. I had to redesign movement controllers, reduce physics overhead, and switch to MaterialPropertyBlocks to avoid instantiating unnecessary materials. This was essential for ensuring that 30–50 avatars could move at once without stutters.

Accomplishments that I'm proud of

I’m proud to have created a completely new format of live entertainment: a Playable Livestream where viewers don’t just watch but enter the world as real-time avatars, interact with the environment, and soon even join in VR. By unifying Twitch, TikTok, VR presence, and interactive 3D worlds into one seamless experience, I’ve built something truly unique — a hybrid between livestreaming, gaming, and social VR that turns audiences into co-creators.

What I learned

Through building this project, I learned how complex it is to merge livestreaming, real-time interaction, and VR into one coherent system. I gained deep insight into low-latency input handling across different platforms, how to optimize performance for dozens of simultaneous viewer avatars, and how to design interactions that stay readable and enjoyable for both 2D viewers and future VR participants. I also learned how differently audiences behave once they gain a visible identity inside a shared world — their engagement, creativity, and sense of presence increase dramatically. This confirmed to me that interactive, co-created livestreams are not just a feature, but a new way of experiencing digital spaces.

What's next for Playable Livestream

The next step for Playable Livestream is to expand the experience from 3D avatars to fully immersive participation. I’m currently developing a feature that will allow viewers not just to spawn as 3D characters in the world, but to join the livestream directly in VR using a Meta Quest headset. This will give them a true sense of presence inside the shared environment, letting them move, explore, and interact alongside the creator and other participants. Beyond VR joining, I plan to add more collaborative mini-games, customizable avatars, spatial interactions, and a streamlined toolkit that makes the system accessible for other creators who want to build their own playable livestreams.