Streamer vs Chat

Twitch chat is the boss's brain; your body is the controller. Chat votes the attacks or calls the combos, and you guard, lean and punch on your webcam, live on stream. One viewer is enough.

Comment

Inspiration

I'd never streamed before this, and the first time I did it was with my mate to test out the app. In that first test run I realised the pacing was way too fast and there was no way I could move fast enough, which led to the difficulty levels.

The whole point of this is that you can use it on a stream with real people and you being in there as an actual person to dodge.

What it does

Two ways to play, both driven by Twitch chat and both played with your body on a webcam (keyboard works too):

  • Boss Fight. Chat runs the boss. Every few seconds viewers vote its next attack (!jab, !left, !right, !combo, and !super once hype is full). The attack is drawn by vote share, so every voter counts. The stage calls the defence (GUARD!, LEAN LEFT!, LEAN RIGHT!), the streamer blocks with their hands or dodges with a lean, then gets a moment to counter-punch. Clean defences fill FOCUS for a FINISHER. When nobody has voted yet, the stream tells viewers what to type.
  • Crowd Trainer. Chat calls boxing combos (!12, !112, !12s with a slip, !12g with a guard, !faster, !slower) and the streamer throws them: three 60-second rounds with rests, scored clean, partial or missed, then a workout summary. A recorded coach voice calls each combo.

I added it in because I thought it would be a really good addition, especially as I'd already had quite a lot of the wiring. That was probably the biggest thing I added in the last day or so. It's for people who are focused purely on fitness: that'd be good for fitness influencers. The Boss Fight one is just a fun one that anyone can use, but this one would be good as a fitness one.

If you've got more people in it, then it goes down as a vote. If, say, X amount of people typed jab, then the one with the highest vote gets it in. That's the same thing that happens for Forge as well. You need to get enough people voting for it, so it helps get engagement, because people need to vote for the different things they want.

On your own channel there is nothing to install: connect Twitch in Chrome, go live straight from the page (Twitch's WHIP ingest; only the fight stage goes on stream), and play. The chat bot posts round results, KO and PERFECT moments become Twitch clips, chat can design a new boss with !forge (it must pass a balance check first), and viewer memory is opt-in with !forgetme. All of this worked live on twitch.tv/kashruns on 2 Nov: the bot posted each round result, KO and PERFECT clips, and "Boss Forge: chat built The Vermicelli Vortex! It fights next round.", and the forged boss fought the next round.

The homepage runs the whole game with simulated chat, always labelled SIMULATED, so anyone can try it in seconds without Twitch.

Webcam, honestly: the game only reads what a normal laptop webcam can see without tracking gear. That is your hands, your head and shoulders, a guard held up, a lean to either side, and a straight punch. It does not read footwork, kicks or the exact punch type. Calibration takes five short steps. Video never leaves the browser.

How we built it

  • One pure game reducer runs everywhere: the balance simulator, the no-login demo and the hosted page, so what judges play is what streamers run.
  • Twitch: EventSub WebSocket chat, Send Chat Message, Create Clip, browser OAuth, WHIP ingest from a browser tab, and a Cloudflare Worker with a Durable Object per channel.
  • Pose: MediaPipe Pose Landmarker in the browser with per-player calibration. Guard and lean decide every hit; punches are a bonus in Boss Fight and the whole point in Crowd Trainer.
  • Voices: 4 boss voices and 4 coach voices recorded with ElevenLabs (872 lines), chosen by the player.
  • A balance gate in CI replays 4,000-round simulations at 0, 3, 50 and 500 viewers and fails if casual win rates leave 30–70%. It rejected our own features twice.
  • 1,281 automated tests, local browser runs of every path (9 homepage flows, 41 Crowd Trainer checks, 9 mocked Twitch steps), a 1,000-chatter load test and a 30-minute soak.

How Codex was used

Built with OpenAI Codex, in one main thread: 77 turns and 57 code commits. Kash set the direction, approved every brief and did all the camera and Twitch testing. Codex planned each task, split the work across subagents running in parallel, tested it in browsers, reviewed each commit with an independent subagent and wrote the code. Rules for the agents: AGENTS.md.

In Kash's words:

I wish I had Pro, but I wanted to build this with Codex, as I've used it for other stuff in the past and at an OpenAI hackathon a few weeks ago, and enjoyed using it. I went through all of my credits from an OpenAI hackathon, I did a full-week reset as well, and I only stopped using it because I'd got down to my last 4%. I really enjoy using Codex, and it helped me in three ways.

  1. You can see what's going on. Especially in the Codex app, you can see how it opens test browsers, and the IDE comes up so you can see the editor. That's really helpful.
  2. The connection to my phone. That was quite helpful when I was trying to do some of the testing: I did the testing myself, and I was able to run it from elsewhere.
  3. The cloud instances, which I found super helpful, and using as many subagents in parallel as we needed to get this moving.

Decisions Codex made inside the briefs:

  1. FINISHER retuned, gate untouched. The balance gate rejected the briefed 8-damage FINISHER (72.6% > 70%), so Codex cut it to 4 instead of loosening the gate (review, tests).
  2. CHAT SYNC dropped after two tuning attempts failed the gate (review).
  3. A lean-invariant guard, after live tests showed the block flickering during leans (replay tests).
  4. Resting hands are not a guard: an anatomical wrist/elbow gate, after a resting streamer read as blocking everything.

Finished with Claude Code. The entry was built on a Plus plan, so once the Codex usage ran out near the end, the last fixes from live tests and the Crowd Trainer mode were finished with Claude Code. Every one of those commits is labelled "(Claude Code)". The full list and the measured split are in docs/codex-evidence.md.

Challenges we ran into

  • Pace: "In that first test run I realised the pacing was way too fast and there was no way I could move fast enough, which led to the difficulty levels." Webcam players start on Easy.
  • Punches on a real webcam: "For punching, it didn't seem to pick it up unless I turned my whole body. Just pushing out a normal punch wasn't picked up." A punch straight at the camera barely makes the arm longer in the picture. Reading the forearm getting shorter as the fist comes towards the camera took the catch rate on Kash's recorded punches from 13 of 44 to 42 of 44, and after a 2 November fix 51 of 59 in Kash's fast one-two drills, with 7 false punches over all the guard and lean footage.
  • Getting on stream: "Setting that up was quite difficult. It's been half an hour, and even then, I still don't really know what's happening." (Kash, on OBS.) So the page can go live itself, and OBS stays as the advanced route.
  • A feature we switched off: we built a live voice AI corner coach to talk to between rounds, then turned it off. "I don't know how much the coach actually helps, because there isn't really much you can do other than play the game."

Accomplishments that we're proud of

  • Going live on Twitch from a browser tab with no OBS, with only the fight stage on stream, in sync with the game (checked against the Twitch recording: the picture matches the page and the sound is about 0.2 s behind).
  • A balance gate that said no to our own features, and we listened.
  • Every live test on Twitch changed the game. In the later live tests, the chat was typed by Kash on a second Twitch account.
  • The first real test with a friend as chat: "They really enjoyed the concept, and they found it engaging enough to watch. They liked making me move around, especially because it was very quick. Having me run around because they were controlling me was great."
  • Chat forged its own boss live (!forge pasta attack became The Vermicelli Vortex), it passed the balance test between rounds, and Kash fought it the next round.

What we learned

It should be something that anyone can pick up and can start using straight away.

That changed the front page: it now leads with playing on your own stream, shows what works without Twitch, and explains what the webcam can and can't see. Small details matter on stream too: after a live test, "hands up" every second "got grating", so the callouts now name the move and the voice plays once a round.

On the webcam itself: "Punch detection seemed to work more when I punched upward, kind of, or off screen. Straight, like a normal jab or cross, was hard for the camera to pick up. I understand now why you have to have sensors for a lot of these other games. This was just using a regular webcam I have, and it can still be workable, as I showed. It might just be a bit more exaggerated with the movements, but it's still a good workout."

What's next

Fitness channels: "fitness type channels could actually use this as a workout or show their fighting skills." In my words:

  • Better calibration for being able to work things out of video, because it's quite difficult to do that unless you have some kind of sensors or something to hone in on. That's the reason why a lot of different pieces of software that do this give you some kind of controller to measure with, rather than doing it purely off video.
  • Putting in different types of punches.
  • Different kinds of kicks, if there's space. There's not a lot of space where I am, which is why I only stick to punches. If there's more space, I like to do more Thai, with kicks and things like that. Maybe not wrestling: I don't think we'd fit that in when you're running the computer.

Also: more games on the same chat-votes loop, and a Twitch Extension so viewers can vote by clicking the video.

"Doing it with video, this is probably the best I can do right now. If I had more time, I'd want to start looking into how to get the calibration working even better so that this could work purely off video."

Built With

  • claude-code
  • cloudflare-workers
  • durable-objects
  • elevenlabs
  • mediapipe
  • node.js
  • obs
  • openai-api
  • openai-codex
  • twitch-eventsub
  • twitch-helix-api
  • typescript
  • vite
  • vitest
  • webrtc
Share this project:

Updates

Submission history