Inspiration

A Twitch viewer can send chat messages, emotes and bits, but nothing the streamer can hold. We wanted to try a small action that ends in the streamer's hand: a viewer sends a snack, the streamer holds out a palm, and a purple character hops over to eat it.

It fits into content the streamer is already making, and viewers only have to type one word.

What it does

Chumi is a 3D character that lives on a Twitch channel. Chat controls it with three commands:

  • !snack: Chumi waits for an open palm, hops onto it, eats a star and hops home.
  • !surprise: Chumi reveals a gift.
  • !cheer: Chumi celebrates.

The streamer's webcam finds their hand, and Chumi sits on the palm. It turns mid-hop, squashes on landing and balances as the hand moves, with a soft shadow under its feet. If the hand leaves, Chumi hops back home.

Chumi also has front, three-quarter and back views. The back shows a symbol for its mood, and the glow around the face changes colour with each baseline expression.

The streamer can pause interactions, dismiss the current action or turn on quiet mode for a serious conversation. A separate broadcast window carries the camera and Chumi for OBS, and the controls stay outside it.

How we built it

Chumi runs locally. A Node.js server reads Twitch chat through EventSub and accepts the three commands. Each viewer has a three-second cooldown and one outstanding contribution, and the queue holds five actions.

Three.js draws Chumi from procedural geometry, so the expressions, hop, feeding and gift reveal are all code. MediaPipe finds hand landmarks in the browser, and smoothed measurements set Chumi's apparent size. Camera frames stay on the streamer's computer. The browser mixes the webcam and character into a 1280 by 720 output for OBS.

We built it with OpenAI Codex, which helped implement the Twitch connection, the renderer and the interaction logic, and helped with tests and review.

Challenges we ran into

A fixed pose made the 3D model look flat. We added directional turns, a feeding tilt and a landing squash, and kept the purple body colour, using the rim glow and the back symbol for mood.

Hand tracking needed rules for interruptions. When the streamer pulls their hand away, Chumi goes home. Every interaction has a defined ending, and cancelled actions do not replay when the streamer resumes.

Testing changed the pacing. Our first cooldown was 20 seconds, which broke the flow between commands. At three seconds, participation felt quick and the queue limit still held.

Accomplishments that we're proud of

Real Twitch messages triggered all three interactions, and a real palm completed the feeding, including a run where we pulled the hand away midway. The repository has 35 automated tests, all passing, covering camera cleanup, interaction timing, queue limits, authorization and the broadcast output.

What we learned

Viewers and streamers need different controls. A viewer needs one command they understand at once. A streamer needs a way to accept that input without giving up the stream.

The palm gives each side a role: chat starts the visit and the streamer finishes it. Movement did more for the character than extra model detail. A small turn, a squashed landing and a shadow made Chumi feel like it had weight.

What's next for Chumi: Chat Came Over

We could not build every augmented reality feature this week, so this version starts with the palm. The next step is depth from the camera, so Chumi can sit on a shoulder, perch on a palm and react to cues in the real world. Shoulder perching needs body tracking and is not built yet.

Beyond that, channels could pick their own welcome, celebration and farewell animations, and several viewers could contribute to one gift reveal, with limits so one account cannot take over. Channel collaborations during raids would need extra Twitch permissions and the approval of both streamers.

We have not measured retention or revenue. Before adding anything, we would ask streamers to run Chumi during five minutes of normal content and see which actions they repeat and when they pause it.

Built With

Share this project:

Updates

Submission history