Stream Splat

A Twitch Extension to make your streams a MESS.

Comment

Inspiration

This entire project got started by me being mad at my computer and being mad at my college football team.

While watching Alabama just be kind of trash but somehow beating Texas A&M, I was doing research and design work for this hackathon. I knew I wanted to:

  1. Make a Twitch Extension
  2. Keep everything within the Twitch Extension Ecosystem
    • No Extension Backend Service and try to use the Extension Configuration Service for everything
  3. Maybe use Bits maybe?

So while watching mid football I started coming up with ideas, and every single one of them wouldn't work. Each one either needed expanded capabilities or the Extension couldn't get the data or couldn't send metadata with a Bits transaction or just some other random thing. Every time the idea went nowhere after a bunch of research, and eventually I just wanted to throw something at the TV. Now, I am glad I didn't do that (I just got an OLED screen for the first time in my life) but eventually I just got the idea that what if I made an Extension just about chunking stuff at the stream. To show the streamer "HEY I'M MAD AND I NEED YOU TO KNOW I'M MAD." At the same time, I wanted the streamer to not necessarily MIND that you were that mad, and therefore if they profited off it then it would be fun for everyone. That was the initial starting point and thankfully Alabama had a bye week the next week.

Tech Stack

Once the general idea behind the project and design was decided, I started making some rules about what the tech stack needed to look like. Being in the Twitch Extension ecosystem does mean that a lot of these decisions are made for you, but I did have some additional requirements:

  • Written in TypeScript
  • Use WebGL and WebGPU in order to make the rendering of everything as efficient as possible when available on a client's machine
  • Not be affected by whatever else is happening inside the iframe that Twitch loads extensions in
  • Have good i18n support so it could support all the languages Twitch supports
  • Have easy to use development environment with as few dependencies as possible

This led me to picking Lit as the framework for the project. While I do like Lit and use it in other projects, the main reason I picked it was for ShadowDOM encapsulation. This lets us worry less about what all is going on inside the iframe or outside it as the ShadowDOM will isolate everything else from what's all going on there. In addition, this gave me access to @lit/localize for handling the localization logic. Overall this should lead to not needing to do a lot to maintain the base features over time.

For the various UI elements that would be needed from both the extension as well as the development harnass, I chose Web Awesome as the library to use. Web Awesome is Lit based (so you get the same benefits of ShadowDOM encapsulation) and pretty much had all the UI elements this project needed. It also has excellent accessability support built in and is very easy to use. Finally, it has agent skills built in to help Codex understand what all was available in the library and quickly grab new UI elements as I asked for them.

And yes, before adding the skill Codex did decide to.... not use any of it. We had to have a DISCUSSION about it and explain the importance of good i18n support that we don't have to maintain ourselves.

When it came to the rendering engine, I knew pretty early that I wanted to use PixiJS as the library to use. PixiJS excels at 2D animation, and everything in this project is some form of 2D something. While an agent CAN write raw WebGL, it's not fun to look at or understand, so I wanted something that made sense to both the robots and me. This would also give us access to WebGPU when available on a client's machine so you can just go crazy with the number of shots.

Bun was added as the development harnass runtime to keep the dev server fast and lean (and their new formatter and TypeScript checker was nice too and I'm looking forward to the linter coming in v1.4.5).

Design

Visual Design

Since this was a QUICK hackathon, there wasn't time to get any assets commissioned for anything for the project. Therefore, the main design philosophy initially started with everything needed to be able to be expressed with math and code and be rendered in engine. This was how the idea of "Splats" came about. A splat is just a fancy SVG that can be streached and distorted with math. Then this extended to the cannon, which is just a bunch of rectangles with border radiuses and gradients that a bunch of math could move around as x and y positions changed. This kept the design simple and also kept the actual rendering simple as well so it could handle a lot more splats. Even though it's silly and janky, it's grown on me. Then again, I could just be staring at it too much.

This eventually extended to all of the promotional materials like extension listing materials and the images in the DevPost Image Gallery. Every visual asset is something rendered in engine and captured using Playwright rather than being mocked up by Codex (well, once I got mad at Codex and told it it's not allowed to mock things up and to use that rendering engine we had spent all that time developing) or a copy of Affinity Designer. That way if I had some listing idea I needed to prove that it was actually possible, and actually found a few rendering bugs this way by asking for stuff the engine couldn't handle until it got fixed.

Behavior Design

When it came to how this extension acts, I really wanted everyone watching the stream to see the same thing. I didn't want the shooter to see one thing and the rest of the viewers to see another. After all, if a Splat just so happened to just NAIL the streamer in the face, all of Chat needs to pop off and not just 1 person. At the same time, Extensions don't let you attach metadata to Bits purchases and there is a set amount of data per SKU and the purchases really just have a limited amount of data received over Extension PubSub.

This drove all of the behaviors to be pseudo-random and in reality everything is deterministic based on metadata that was sent over Extension PubSub. Splat locations, splat colors, and splat distortions are all based on metadata that every viewer receives and is then transformed into rendering data so that every viewer sees the same thing. While this started as a limitation, it ended up being a feature of the extension as this removed a lot of additional clicks and decisions for shooters. Instead of sitting there and deciding on shot locations and shot colors and a bunch of other things, they just decide how many Splats to shoot and when to fire. This leads to Splats being more of an impulse buy and overall lowers the amount of friction for all the interactions.

Additional Things I Learned Working On This Project In No Particular Order

  • Codex likes using all my GitHub Actions minutes if I don't stop it
  • DaVinci Resolve and how its MCP server works (it's pretty neat)
  • That my PC is bad at rendering DaVinci Resolve previews (it's fine at final renders though)
  • That NVIDIA Broadcast and DaVinci Resolve will fight each other for GPU resources like spoiled children

What's Next

  • A nap.
  • Get the extension past Extension Review (if it hasn't made it by the time you read this).
  • Wait for some more people to install it and yell at me. Cuz they will.

Built With

Share this project:

Updates

Submission history