StreamClip AI

Turn live Twitch moments into viral 9:16 Shorts & TikToks in 1 click—with a 30s browser buffer, live! clip chat triggers, custom watermarks & OpenAI Codex hooks.

Comment

Inspiration

Twitch streamers generate hours of high-energy live content every day, yet discoverability happens on vertical short-form platforms (TikTok, YouTube Shorts, Instagram Reels).

Currently, turning a hype live Twitch moment into a vertical 9:16 Short requires downloading massive VODs, importing them into heavy desktop video editors, manually cropping 16:9 gameplay + facecam, adding streamer/clipper attribution watermarks, and brainstorming viral titles—or paying steep monthly subscription fees for cloud GPU clipping services.

We asked: What if a live Twitch viewer typing !clip in chat—or a sudden surge of POG / LUL chat hype—could instantly slice the last 30 seconds of a stream directly in the browser, auto-crop it to 9:16 vertical format with facecam + gameplay, stamp the viewer's handle on screen, and generate a viral hook using OpenAI Codex—with \$0 server bandwidth fees?

What it does

StreamClip AI is a 100% client-side, zero-server-cost live clipping and 9:16 vertical production engine built for Twitch streamers and their communities:

  1. Dual Staggered 30s Rolling Browser Buffer: Continuously captures the rendered 720x1280 (9:16) vertical canvas stream and audio tracks in memory using dual staggered MediaRecorder slots (2.5 Mbps VP8)—using zero disk space until a clip is triggered and exporting ready-to-play .webm files in under 0.3s.
  2. Zero-OAuth Live Twitch Chat Trigger (!clip): Connects directly to any live Twitch channel via Twitch's anonymous IRC WebSocket (wss://irc-ws.chat.twitch.tv:443). When any viewer types !clip in live chat, StreamClip AI automatically captures the last 30 seconds and credits the viewer on-screen (⚡ Clipped by @username).
  3. 🔥 Automatic Chat Hype-Spike Detector: Tracks a 7-second sliding window of live Twitch chat velocity and hype emotes (POG, LUL, KEKW, OMG, CLUTCH, CLIP, 🔥). When chat velocity hits 100%, StreamClip AI automatically triggers a 30-second vertical clip (🔥 Clipped by @ChatHypeSpike) even if nobody typed !clip.
  4. 4 Real-Time 9:16 Smart Crop Layouts + Interactive Framing:
    • Center Crop (Crosshair / Action): Crops 16:9 gameplay to vertical 9:16 focused on the action zone.
    • Split: Top-Right Facecam + Gameplay: Stacks the streamer's top-right webcam in the top 34% and center gameplay in the bottom 66% with a neon Twitch divider.
    • Split: Top-Left Facecam + Gameplay: Dedicated split layout for streamers with top-left webcams.
    • Pillarbox Blur Background: Centers the full 16:9 frame over a dynamically blurred vertical backdrop.
    • Smart Video Framing (Pan X / Pan Y / Zoom + 📺 Fit YouTube / Twitch Tab Preset): Allows instant 1-click framing to crop out white webpage sidebars when capturing non-fullscreen browser tabs.
  5. OpenAI Codex Viral Hook Banner & Caption Engine: Renders an editable TikTok/Shorts hook banner directly onto the 9:16 video canvas and generates 3 copy-ready viral titles and hashtag packs tailored to the streamer and clipper.
  6. Multi-Clip Session Vault & Cover Extractor: Stores every captured clip during a stream session with instant in-browser playback, 1-click .webm video download, and an automatically extracted 720x1280 .jpg cover thumbnail.

How we built it

We used OpenAI Codex as our core engineering co-pilot to architect, debug, and optimize the real-time media and networking pipelines in our repository (GitHub Repository):

  • Real-Time 9:16 Canvas Compositor (CanvasPipeline): Used Codex to design the 720x1280 vertical compositor that transforms any 1280x720 (16:9) screen/tab capture or synthetic stream into 4 switchable vertical layouts with glassmorphic streamer badges, viewer attribution pills, and lower-third viral hook banners.
  • Dual Staggered Rolling Recorders (RollingClipBuffer): Standard MediaRecorder chunk-shifting corrupts .webm files because the EBML initialization header only lives in chunk[0]. With OpenAI Codex, we engineered a Dual Staggered Recorder architecture (Slot A starting at 0s and Slot B offset at 15s, rotating every 32s) so every sliced clip retains a valid initialization header and exports instantaneously without waiting on single-threaded WASM re-encoding.
  • Native Twitch IRCv3 WebSocket Parser & Hype Engine (TwitchIRC): Used Codex to write a zero-dependency WebSocket client that performs the anonymous justinfan handshake over wss://irc-ws.chat.twitch.tv:443, responds to PING/PONG heartbeats, parses IRCv3 messages in real time, and calculates rolling chat hype velocity.

Challenges we ran into

  1. WebM Header Corruption in Rolling Buffers: Simply calling recordedChunks.shift() after 30 seconds strips the WebM EBML header from chunk[0], producing unplayable video downloads. Solving this with dual staggered MediaRecorder instances ensured every 15–30s window starts with a clean keyframe and initialization segment and exports in < 0.3s.
  2. Background-Tab requestAnimationFrame Throttling: When capturing another browser tab via getDisplayMedia, Chrome focuses the captured tab and throttles requestAnimationFrame() in the background StreamClip AI tab to 0 FPS, causing frozen video exports. We solved this by pairing requestAnimationFrame with a 33ms (30 FPS) background timer driver and asynchronous canvas.toBlob() thumbnail extraction so recording stays butter-smooth at 30 FPS even when StreamClip AI is in an unfocused tab.
  3. Cropping Out Browser UI on Non-Fullscreen Streams: Capturing a standard YouTube or Twitch tab often includes webpage margins or sidebars. We built real-time Pan X, Pan Y, and Crop Zoom sliders plus a 1-click 📺 Fit YouTube / Twitch Tab preset (panX: 28%, panY: 34%, zoom: 1.5x) to lock straight onto the video player.

Accomplishments that we're proud of

  • \$0 Cloud GPU Costs & Instant < 0.3s Exports: Everything runs natively in the browser at 30 FPS using HTML5 Canvas and MediaRecorder without server uploads or slow transcoding queues.
  • Dual Community Triggers (!clip + Automatic Chat Hype-Spike): Combines intentional viewer clipping (!clip) with automatic emote-velocity detection (POG / LUL waves) so streamers never miss a clutch moment while focused on gameplay.
  • Built-In Community Attribution: Viewers love seeing their handle (⚡ Clipped by @Viewer) permanently baked into a streamer's official TikTok or YouTube Short, turning passive viewers into active clip hunters.
  • Zero-Friction Live Testing: Works out of the box with live screen/tab capture (getDisplayMedia), real Twitch chat channels, and a built-in 60 FPS synthetic gameplay + chat hype simulator for instant browser testing.

What we learned

  • How to synchronize HTML5 <video>, <canvas> compositing, background-tab timer drivers, and MediaRecorder bitrates (2.5 Mbps VP8) for smooth, zero-freeze vertical video output.
  • How effectively OpenAI Codex can diagnose and solve subtle browser media pitfalls—including WebM EBML header preservation, background-tab frame throttling, and real-time IRCv3 WebSocket stream parsing.

What's next for StreamClip AI

  • OBS Studio Custom Dock & Browser Source: Embed StreamClip AI directly inside OBS Studio so streamers can run it without an extra browser window.
  • Microphone Audio-Peak Auto-Trigger: Combine our Twitch Chat Hype-Spike Detector with Web Audio API decibel spike detection (streamer reactions/laughter) for multimodal auto-clipping.
  • Direct TikTok & YouTube Shorts OAuth Publishing: One-click upload directly from the StreamClip AI Session Vault to YouTube Shorts and TikTok drafts.

Built With

Share this project:

Updates

Submission history