ClipScout

ClipScout turns Twitch, Kick, and YouTube VODs into ranked, editable highlights using audio peaks, chat surges, and GPT-5.6, then exports creator-controlled 16:9 or 9:16 MP4 clips.

Comment

Inspiration

The best livestream moments are often buried inside hours of footage. Creators remember the clutch plays, reactions, jokes, and moments when chat exploded—but finding them again after the stream is slow and repetitive.

Existing highlight tools often make black-box recommendations or force horizontal gameplay into an awkward vertical crop. We built ClipScout to make discovery fast while keeping the final editorial and framing decisions in the creator’s hands.

What it does

ClipScout converts long Twitch, YouTube, and Kick livestream VODs into ranked, editable, post-ready highlights.

Creators can analyze a VOD using three independent discovery modes:

  • Audio peaks for loud reactions and high-energy moments
  • Comment surges for moments when chat activity suddenly increases
  • Hybrid mode combining both signals

ClipScout displays the evidence behind each candidate instead of presenting unexplained recommendations. GPT-5.6 interprets nearby chat context and candidate metadata to generate an editorial title, explanation, short-form hook, moment type, and semantic highlight score.

After selecting a moment, the creator can preview the video while editing and choose from:

  • Original 16:9 output
  • Direct 9:16 vertical crop
  • Twitch-style two-panel 9:16 layout for gameplay and the streamer

The gameplay region supports pan and zoom without stretching the original footage. The streamer region can be moved and freely resized. ClipScout then renders a downloadable MP4 ready for YouTube Shorts, TikTok, Twitch, or other social platforms.

A built-in judge demo provides a complete rights-safe sample workflow without requiring an external VOD or account.

How we built it

Codex was our primary engineering and product-design collaborator throughout OpenAI Build Week. It helped with product architecture, implementation, testing, cloud deployment, responsive UI design, and the submission workflow.

The frontend uses React, TypeScript, and vinext. A FastAPI worker handles media ingestion, audio analysis, and video rendering. yt-dlp supports public VOD sources, while FFmpeg analyzes audio energy and generates the final clips.

The deterministic pipeline first identifies candidate windows from audio energy and chat velocity. Only compact evidence—timestamps, signal scores, and nearby chat samples—is sent to the OpenAI Responses API.

GPT-5.6 (gpt-5.6-sol) returns a strict structured result containing:

  • Semantic highlight score
  • Editorial title
  • Evidence-based explanation
  • Short-form hook
  • Moment category

The final ranking blends the measurable signal score with GPT-5.6’s editorial judgment. If the OpenAI API is unavailable, ClipScout preserves the deterministic candidates and clearly reports that AI enhancement was not applied.

The media worker runs on Google Cloud Run, while the public frontend is hosted with OpenAI Sites. API credentials remain server-side and are never exposed in the browser.

Challenges we faced

The most difficult challenge was designing a vertical editor that feels natural for horizontal gameplay.

A simple center crop frequently removed important gameplay information. Our final composer keeps the gameplay crop predictable, supports direct pan and zoom, allows free resizing of the streamer layer, and updates the vertical output preview while the source video is playing.

Another challenge was making AI-assisted selection explainable. We kept the measured audio and comment signals visible and constrained GPT-5.6 to a strict editorial schema. This lets creators understand why a moment was selected while retaining full control over the final clip.

Accomplishments that we're proud of

  • A complete judge demo requiring no account or third-party media
  • Separate audio, comments, and hybrid discovery modes
  • GPT-5.6 structured editorial curation with visible supporting evidence
  • A playback-aware 16:9 and 9:16 crop editor
  • A Twitch-style two-panel vertical layout
  • Pan, zoom, and freely resizable streamer framing
  • A real downloadable MP4 export
  • Rights-safe sample content and deterministic AI fallback
  • A production deployment with server-side secrets and tested failure states

What we learned

AI works best in ClipScout as an editorial layer rather than the only ranking engine.

Deterministic signal processing provides speed, reliability, and traceability. GPT-5.6 adds contextual understanding, naming, storytelling, and editorial judgment. Combining both approaches produces results that are more useful and easier to trust.

We also learned that a strong judge experience must be self-contained. The built-in demo avoids dependence on third-party VOD availability, platform restrictions, and copyright-sensitive footage.

What's next for ClipScout

Next, we plan to add creator accounts, processing credits, saved layouts, brand presets, subtitle generation, team review, scheduled VOD processing, and direct publishing integrations where platform policies allow them.

ClipScout is designed to become a creator subscription service with a limited free tier and usage-based video processing.

Built With

Share this project:

Updates