Fourth Wall

Twitch viewers become tiny characters you catch, pass between your real hands, and rescue—with one webcam and a chat command.

Comment

Inspiration

A Twitch chat is full of people, but on screen they usually occupy a column of text. I wanted to make their presence physical: what if a viewer could jump out of chat and land in the streamer's hand?

Fourth Wall turns that idea into a small, shared rescue game. A viewer types !join. A tiny character with their name appears. The streamer catches them on a real palm, passes them to the other hand, and tries to bring them home safely. The joke, the tension, and the celebration all happen in the same webcam frame.

What it does

Your chat. In your hands.

  • Join from Twitch: viewers type !join in the connected broadcaster's chat. Their characters carry their display names and chat colors. Viewers install nothing.
  • Play with real hands: one ordinary webcam tracks two palms. Catch falling viewers, carry them, pass them between hands, or tilt a palm to release them.
  • Give viewers agency: !wave, !heart, and !spin animate the viewer's own character while the streamer holds them.
  • Make a moment into a game: a 60-second rescue run adds first-catch and first-handoff points, rescue streaks, and a device-local best score.
  • Set the atmosphere: choose Grove, Orbit, or Sunset, toggle names and sound, and enter a clean stage view for browser Window Capture in OBS.
  • Try it immediately: rehearsal works without a camera or Twitch login, with draggable palms and a guided handoff using the same physics engine.

The intended use is a short, repeatable community ritual between games or during a chat break: try to save the people who are actually there.

How I built it with Codex

I built Fourth Wall from scratch with OpenAI Codex. I directed the concept, tried it with my own webcam and Twitch account, and reported what felt wrong. Codex implemented the React and TypeScript interface, Canvas game, tracking worker, Twitch integration, automated checks, and Vercel deployment. We iterated on working builds rather than stopping at a mockup.

The pipeline is deliberately small:

Twitch chat → EventSub WebSocket → named character → tracked palm → physics → catch, handoff, rescue.

Twitch OAuth authorizes the broadcaster's own chat with user:read:chat. EventSub delivers chat messages, with duplicate protection, bounded intake, cooldowns, and reconnection handling. Fourth Wall does not send chat messages.

MediaPipe Hand Landmarker runs in a worker with GPU acceleration and a CPU fallback. Only one inference frame is in flight at a time. A 120 Hz physics simulation handles moving support surfaces, gravity, catches, handoffs, and rescues. Camera frames and landmarks stay on the device; there is no application backend or camera upload.

Challenges I ran into

The hardest part was making two independently tracked hands feel like dependable platforms. In the first real webcam tests, the second hand appeared late and a viewer could fall during a transfer. A smooth animation was not enough: tracking delay and physics had to agree about where support existed.

With Codex, I moved inference onto the GPU worker where supported, improved hand identity matching, and added a short, deliberate nearby-palm handoff. Brief detection gaps receive bounded tolerance; absent hands eventually lose their platforms. A regression test also exposed crowd spacing pushing a character off a palm edge, which we fixed.

Accomplishments and what I learned

A recorded test with my own Twitch account used real !join and !wave messages, physical palms, and the deployed app. That take produced one handoff and two rescues with zero misses. The current build passes 58 unit tests; a 13-test browser suite covers the broader experience, and the final production handoff regression passed with two transfers and two rescues.

The tests separate real-model image fixtures, synthetic movement, and physical webcam checks. Lighting, occlusion, and computer performance still matter. My biggest lesson was that automated tests and human playtesting reveal different problems—and Codex was most useful when I gave it a concrete failure it could reproduce and verify.

What's next

Test with independent viewers and more camera setups, improve difficult hand crossings, and build more community games around the same catch-and-pass mechanic. The longer-term idea is a playground of physical audience interactions; this submission delivers the palm-catching foundation.

Try Fourth Wall

Open the live app and select Watch a handoff for a no-login introduction. For your own hands, select Your camera, keep both open palms visible in good front lighting, and bring the empty palm just underneath a held viewer. Connect Twitch links the broadcaster's chat so viewers can type !join.

Built With

Share this project:

Updates

Submission history