ClipMaster

Inspiration

"Someone clip that!" is an expression often heard on streaming platforms. With our system, that will no longer be necessary. Instead, a team of agents will be constantly monitoring your stream for those juicy clips.

As Streamers often post many of their stream clips across all their short form platforms like Youtube, Instagram and X. We wanted to make that process as seamless as it could possibly be. Instead of a streamer yelling to his moderators to "clip that", we built a tool that can get that done for you.

This is similar to the goal of Medal, the clipping app for competitive games. Although similar in concept, the underlying architecture is very different. Medal hooks directly into game applications to provide clips whereas Twitch doesn't really have much to go off of except for the stream itself and the chat.

What it does

We devised a system to monitor the stream for "hype moments", moments where the energy of the stream and the chat is highest. We do this by analyzing transcripts of the stream and of chat to figure out timestamps for these moments. The system follows a director-critic model where the director will constantly feel for new clips, cut them, and edit them and the critic will independently evaluate the clips to see if they are worthy of a post.

How we built it

This application is built on a Rust and Tauri backend with a React frontend. It uses OpenAI models as the backbone for analysis and decision-making, but operates in a highly structured manner.

We operate on a sliding window technique; 12 second windows wiht 6 seconds of overlap. These windows are transcribed through the Whisper model, have their audio energy analyzed, and have their chat logs pulled to create an EvidenceWindow. The evidence loop is then fed into an agentic feedback loop:

Observer: detects whether a coherent moment may be developing.
Director: identifies setup, escalation, payoff, reaction, and an initial cut.
Editor: tightens boundaries, removes dead air, and chooses layout/hook text.
Critic: independently attempts to reject weak, repetitive, misleading, unsafe, or poorly contextualized clips.

All Agent's transcripts, decisions, manifests, etc. for each session are stored in a lightweight SQLite database. This allows decisions and reasoning to be traced.

The whole thing runs in an orchestrated Pipeline module which handles preparation of the media source, initializing the windows, passing control between agents, and finally rendering and publishing a finished clip.

Challenges we ran into

Our biggest challenge was managing API costs with clip quality. Originally we wanted to feed in visual streams and were using high end models (like Terra and Sol). However this was completely unsustainable and used a lot of credits within a short span of time.

We reduced costs by switching off visual stream input, switching to a text based prediction model, and using Luna. While the costs were much more manageable now, we still had very high API costs.

We further streamlined our API usage by making changes to main agentic workflow. In particular, we added a new agent called the AudioAnalyzer in between the Director and Editor. This means that audio analysis runs only if the director or observer didn't give a confident rejection. Furthermore, a confident rejection at any stage skips the rest of the stages. Lastly, the evidence sent to models is sampled instead of sending everything:

Up to 64 transcript segments
Up to 80 chat events
Up to 12 visual samples

These architectural changes minimized our API costs and made the project feasible.

Accomplishments that we're proud of

We are proud of taking on a project that is able to analyze a stream in real time. At the high level, this involved a lot of good design to ensure tha thte system is architected purposefully and we really had an emphasis on that.

We are also proud of being able to address our challenges and reducing our API costs.

What's next for Twitch Capture

Our next step is completing the publishing workflow, including title generation and direct posting to social platforms. We also want to support multiple creator accounts with distinct personalities, audiences, and content niches. Each account could have its own vocabulary, preferred humor, clip style, title style, and posting strategy.

Also the sliding window strategy is not the best and introduces timestamp distortion errors between the Critic's final output and what actually gets clipped. We would like to unify time objects into one MediaTimeline object that every part of the system must report to to unify timestamps and get rid of messy logic between various elements of our system.