I have a research background in the realm of interactive video on the web from my time as a graduate student at NYU. During that period, I worked with Dan Shiffman on Hello Processing and with Dan Shiffman and Lauren McCarthy on Hello p5.js, both of which paired video captures with real-time code running in the browser.
Heat is an initial attempt at bringing some of these ideas to Twitch. I've always been impressed with the tools and experiences built around Twitch Chat, but wanted to provide a more "tangible" experience.
What It Does
Simply put, Heat captures click data on the video feed and routes it to the broadcaster. Heat currently provides a number of front-end demos and also includes instructions for creating custom experiences with a simple protocol based on socket.io.
How I Built It
The EBS is a bit more complex. It listens for click data (verifying that the payloads are valid and managing cool downs) and then broadcasts that data over socket.io to any experiences that are listening. It is written in Node.js and deployed to AWS Elastic Beanstalk where it makes use of several AWS services, including EC2 and ElastiCache.
The individual viewing experiences vary in complexity based on content, but the general flow involves connecting to the EBS via socket.io and then subscribing to a specific channel's data.
Challenges I Ran Into
The biggest overall challenge has been video latency. Using Low Latency mode helps massively, but it unfortunately has limited browser support and there's still a noticeable lag. The short-term answer has been to create experiences where immediate visual feedback isn't essential, but it's still the most problematic part of the system.
I've been considering ways to give more visual feedback to the viewer regarding the timing of their clicks in the future.
Accomplishments That I'm Proud Of
One of the things I pride myself on in creative work is my ability to pull together a lot of disparate components and make them work together to create something new. Heat has a lot of moving parts spread across the Twitch front end, the EBS and the experience views, but the end result has been performant and reliable thus far.
Beyond that, though, the most rewarding part has been the surprise and delight shown by my viewers at their new ability to participate directly in what I'm streaming.
What I Learned
While I've done a fair amount of Node.js work in the past, this was my first time deploying to AWS. At first, I found it difficult to navigate the huge array of services and features it provides, but once I got everything configured and deployed the service has been rock solid.
What's Next For Heat
In the near term, I'd like to come up with an implementation for mobile and explore possibilities for monetization. I'm also very interested to see what other streamers create with Heat.