Inspiration
What it does## Inspiration
What if Twitch chat could create the game around the streamer?
I wanted viewers to do more than watch or trigger a predefined effect. They could imagine an ice kingdom, ask for a crystal sword, or turn a battle into a peaceful lantern ceremony—and see their collective decisions become something the streamer could physically play.
Backdrop started as an experiment with virtual backgrounds. It grew into a studio where the community helps write the adventure.
What it does
Backdrop turns ordinary chat messages into shared, interactive experiences.
Codex uses chat as inspiration to propose four distinct possibilities. Viewers vote, and the winning idea becomes the next chapter. Codex can write interaction rules for that chapter, while Tripo supplies generated environments and 3D assets.
The streamer appears inside the scene without a green screen. Tracked hand, head and body movements drive props and interactions. A sword can charge, a guardian can expose a weak point, or a cupped hand can gather light and release a lantern.
The Mac app combines scene creation, audience controls, voting overlays, saved adventures and clean OBS output. Streamers can require approval, restrict participation, pause audience control and replay saved experiences. A companion Codex skill also provides conversational control of the local studio.
How I built it
I used OpenAI Codex throughout development to build and refine the camera pipeline, renderer, Twitch integration, voting system, interaction runtime, Mac application and tests.
The app combines a Python backend with a JavaScript/WebGL frontend and an Electron desktop shell. Robust Video Matting separates the streamer from the camera background. MediaPipe supplies tracking, and Depth Anything V2 adds relative depth to background images.
At runtime, the signed-in Codex CLI plans audience requests and authors bounded JavaScript interaction programs. These programs run in an isolated Worker with validated inputs and outputs. Tripo generates images and textured 3D models; completed assets are cached locally for reuse.
Twitch authentication and EventSub connect real channel chat to the director and voting system. OBS receives the composed scene for recording or broadcasting.
Challenges
The hardest part was connecting creative, unpredictable requests to a reliable live experience.
Generation takes time, tracking can disappear, and viewers may suggest conflicting ideas. I separated lightweight voting proposals from preparing the winning scene, added progress and approval controls, and made saved experiences replayable.
Prop alignment, hand occlusion and convincing placement also required repeated testing. The backgrounds use image-based 2.5D depth rather than full room reconstruction, so interactions use camera-relative positioning.
What I learned
The exciting part is not simply generating an asset. It is turning an audience idea into an understandable interaction with a goal, feedback and a payoff.
I also learned that creative freedom needs clear streamer controls. Approval, permissions, cancellation and reusable assets are central to making an AI-directed experience practical.
What is working today
Backdrop runs locally as an Apple Silicon Mac prototype. Real Twitch chat and an initial live broadcast have been tested.
The trailer demonstrates two prepared experiences: Frostbound Throne and Lantern Lagoon. It uses local recording, simulated audience participation and previously generated scenes for repeatable capture. The underlying app also supports fresh Codex-authored interactions and optional Tripo generation.
What's next
I want to improve spatial placement, expand interaction possibilities and test longer sessions with real communities. The goal is a stream whose next chapter comes from its audience.
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for Backdrop — Chat imagines. Codex builds. You play
Built With
- codex
- obs
- twitch