Inspiration
A streamer's best moment dies in ten seconds. Chat erupts (LULs, clips, Ws) and then the scroll swallows it. Turning that hype into a story means hours in an editor after the stream, and small channels never get the TV-production polish that big broadcasts have. We wanted the hype itself to have a director's chair.
What it does
DirectorCut is a live cutscene engine for streams:
- Viewers type
!cutscene [style] [what happened]in Twitch chat: noir, anime, hollywood, baywatch, or sportscast. The optional viewer pitch ("one HP clutch with a frying pan") is story material the AI director weaves into the scene: capped at 100 characters, scrubbed of links and @mentions, blocklist-checked, and never rendered on the overlay, it shapes the script and shows only in the operator console's scene details and history. - An AI director (any OpenAI-compatible LLM) writes a cinematic scene as validated JSON: caption, badge, sting, duration, reacting to the actual chat hype.
- The cutscene renders live over the stream (OBS browser source): letterbox bars, film treatment, word-by-word narration, blinking
● REPLAYbadge, and an "EXECUTIVE PRODUCER: @viewer" credit for whoever called it. - Chat-hype auto-trigger: when hype in a rolling 30-second window crosses a threshold, the director cuts a scene on its own, credited to CHAT ITSELF.
- A WebAudio-synthesized sting scores every scene: no copyrighted audio, no external assets, nothing to load.
Demo video note: the video runs the real server (npm start) with its chat client connected anonymously to Twitch, and fires scenes through the local trigger API using sample viewer handles; the terminal, WebSocket frames and test run are live, and the opening chat panel is illustrative. Real Twitch chat commands use the same pipeline via IRC (tmi.js); setup is in the README.
How we built it
- The core engine (Twitch IRC integration, the LLM director with a tolerant JSON contract, the WebSocket overlay renderer, the WebAudio synth, the rehearsal simulator) was written by OpenAI Codex in a single session from a written build brief (session ID
01a19dc4-534f-7690-b36b-33615188f337). Human role: directing. Codex role: engineering. - After the Codex build, directed polish passes added output moderation (blocklist + OpenAI moderation, fail closed), chat rank gating, a server-side take counter with late-join replay, film-grain tiles, viewer-pitch intake (length cap, scrubbing, blocklist), and a JSON-only trigger API (requests without a JSON content type get a
415), all verified by the same network-free test suite. - Constraints we set and kept: plain Node.js ESM, three dependencies (
tmi.js,ws,dotenv), zero external assets, one process serving overlay + API + WebSocket + chat, secrets only in.env. - Reliability is the design: model output is validated and clamped (5–12 s), captions filtered, credits always rebuilt server-side from the real trigger, and if the LLM is unreachable a deterministic template director still cuts the scene. The show must go on.
Challenges
- Making an overlay that renders zero pixels when idle yet erupts broadcast-grade mid-scene, inside OBS's restrictive browser-source environment (blend modes, autoplay policy, transparency).
- Treating chat as untrusted input end-to-end: URL/mention stripping, 100-character pitch caps, JSON tolerance, caption truncation, moderation that fails closed, rate-gated replies.
- Keeping worst-case latency bounded: the entire director pipeline (LLM + moderation) fits in a fixed timeout budget, with a fallback that never leaves chat without a cutscene.
Accomplishments
- Five distinct broadcast treatments with procedural stings that look directed, not generated.
- A provider-pluggable AI director (OpenAI, or any local OpenAI-compatible endpoint) with a contract a judge can read in one file.
- 49 fast, network-free tests covering the JSON contract, fallback rotation, cooldown/queue boundaries, hype-window math, moderation fail-closed paths, pitch cleaning and blocklist rejection, the JSON-only trigger API, WebSocket role counting, late-join replay, and timeout aborts.
What we learned
Constrained AI beats deterministic code for creative runtime content, but only when the AI's output is treated like any other untrusted input: validated, clamped, filtered, and rehearsed with a fallback. And in OBS-land, "invisible when idle" is a feature you have to engineer as deliberately as the scene itself.
What's next (roadmap only, none of this is in the submitted code)
- Vision director: the director sees the actual frame via OBS source screenshots and narrates the real moment.
- Real slow-mo replays cut from the OBS replay buffer.
- Channel Points redemptions ("Executive Producer" as a reward) with automatic refunds, and credited Twitch clips.
- First-class local profile (Ollama default) so the whole thing runs with zero paid APIs.
Built With
- codex
- javascript
- node.js
- obs
- openai
- tmi.js
- twitch
- webaudio
- websocket