"Let viewers throw hearts with !love, two per person in any rolling 30 seconds." 16.5 seconds later, OpenAI Codex had compiled that sentence into a handler, a permission manifest and its own tests. Riff verified it 10/10 in a sandbox, rehearsed it privately, and one click later real Twitch chat was throwing hearts across the stream.
Inspiration
Chat is the best writers' room on Twitch. "Throw tomatoes at me." "Let us vote on the next round." "Red vs blue!" The best moments of a stream are invented live, by the audience.
Almost none of them make it on screen. Turning one idea into a real interaction means a bot command, an overlay, a rate limit, an OBS source and an hour of setup. By then the moment is gone.
AI coding could close that gap in seconds. But letting a model write code that runs live, in front of an audience, is exactly what no streamer should do blindly. Untested, unbounded, impossible to stop.
We wanted both: the speed of "just describe it" and the safety of "nothing goes live that I didn't approve." That tension is Riff.
What it does
Riff turns a creator's sentence into a working, controlled Twitch interaction in five steps.
- Describe. Type the idea, or hold to talk. Example chips get a first-time creator started.
- Compile (OpenAI Codex). Codex compiles the sentence at runtime into a JavaScript handler, a permission manifest and declarative tests. When a fact is missing, it asks one precise question instead. When something is impossible, it says why and proposes the closest supported alternative.
- Test. Riff re-verifies everything itself and shows the evidence in plain language: when it runs, what it does, who can trigger it, how much, and a table of what you asked vs. what will run.
- Preview. A private rehearsal stage with test viewers (Alice, Bob, a subscriber, a moderator). One click runs a proof sequence: accepted, limit reached, unrelated chat ignored, duplicate delivery ignored.
- Approve & play. Approval activates only the exact tested artifact. Real Twitch chat (EventSub) then drives the effect on an OBS overlay. Pause, rollback and Stop everything are always one click away, including
!riff panicfrom chat.
Beyond one-shot effects, the same pipeline powers:
- Stateful games and formats. Team battles, community goals, shield walls (N distinct viewers within a window), castle sieges with health and elimination, best-of-N matches, milestone unlocks, a pixel mural painted by chat, a staged song vote and quizzes. Scores and resources live in the host, never in generated code.
- Live evolution. Change the rules of a running round ("switch to
!splatand give everyone two more throws") without wiping viewers' progress. Riff generates a migration, proves it on empty, populated, exhausted, paused and finished rounds, and lets the creator choose keep progress or fresh round. - Audience co-creation. Viewers propose mechanics with
!riff suggest. Moderators shortlist, chat votes, and the creator turns a winning idea into a brief that Codex compiles. Chat never compiles or activates code by itself. - Twitch-native actions. Polls, announcements, shoutouts, Channel Points, Predictions (the creator always settles), triggers for subs, gifts, resubs, cheers and raids, and named timers.
- Studio tools. A scenario studio with saved regressions, a visual overlay canvas, shared rounds across channels, an adaptive pacing director, remixable recipes, OBS / Streamer.bot / Firebot bridges, a public viewer guide in four languages, and aggregate insights that never rank viewers.
How we built it
OpenAI Codex, twice
Codex built Riff. We developed Riff in Codex threads. The main build thread (01a19212-fdf0-7000-89d6-43b58f1719fe) carried features F31 to F40 across 1,278 Codex tool calls. Each feature followed the same loop: Codex proposed the next feature, we agreed the design, then Codex wrote failing tests, implemented, reviewed and committed in focused slices.
Codex is Riff's runtime compiler. The server drives the @openai/codex-sdk with a deliberately tiny world:
- A fresh temporary workspace per compile holding only
TASK.md,SDK.md, two.d.tsfiles and curated examples. No repository files, credentials or secrets. - A fresh
CODEX_HOMEcontaining only the login, a read-only sandbox, no network, no web search and no approvals. - Structured output against a strict JSON Schema: candidate, question or unsupported.
- Exactly one repair turn, fed with the host's own failure report (at most 4 KB), and a 180-second timeout.
- A compile queue (1 per channel, 4 global). Codex's reasoning summaries stream live into the console, and every run records its thread ID, attempts, prompt-pack version and token usage.
The trust model: Codex writes, the host decides
- Admission before code. Duplicate suppression, exact command match, audience roles, rolling per-viewer windows and channel budgets are enforced by the host. Each decision is written to SQLite before any effect leaves the server.
- A real sandbox. Handlers run in QuickJS (WebAssembly), with a fresh runtime per event, 64 MB of memory, 50 ms of CPU, 2 s of wall time and 25 bridge calls. There is no network, filesystem, timers or
eval. Any fault quarantines the version. - Independent verification. Manifest schema, then restricted syntax and least privilege, then sandbox compile, then Codex's generated scenarios, then a host policy suite that Codex cannot see or edit, plus deterministic intent checks.
- Exact approval. Approval binds the artifact hash, the test-report hash and the expected live version. A stale review gets a 409.
- An off switch that always works. Stop flips an in-memory fence first and persists second, so it works even when the database or Twitch is degraded.
Stack
A strict TypeScript monorepo:
- Web: React 19 and Vite, with four entries: the creator console, the OBS dock, the read-only overlay and the public viewer guide.
- Server: Fastify 5 with WebSockets, and SQLite through Drizzle (21 migrations).
- Runtime and Twitch: QuickJS (WASM) and Zod at every boundary, plus Twitch EventSub over WebSocket and Helix.
- Hosting: AWS Lightsail, managed by Terraform. Secrets live in Secrets Manager, and GitHub OIDC CI/CD needs no stored keys. Each streamer gets an isolated container behind Caddy HTTPS, with daily backups.
What is live and what is simulated
| Verified live on production (real Twitch, real Codex) | Built and tested against Twitch's documented payloads, not yet run live |
|---|---|
| Codex runtime compiles and repairs | Polls, Channel Points and Predictions (need an Affiliate/Partner channel) |
| EventSub chat driving the OBS overlay | Sub, gift, resub, cheer and raid triggers (exercised with labelled synthetic events) |
| Chat replies and announcements via Helix | OBS, Streamer.bot and Firebot bridges (protocol tests) |
!riff help / !riff language answered in German |
Voice transcription with a real microphone |
| A 5-minute timer surviving a redeploy; stop and resume | Shared rounds between two real channels |
Overlay images in the gallery marked sample data were rendered from sample overlay messages, not a live round.
Challenges we ran into
- Trusting generated code without trusting the model. A passing test written by the same model proves little, so the host derives its own expectations from the manifest. The first real production compile of a timer interaction exposed a host bug in how timer actions were counted against budgets. We fixed the host, not the prompt.
- Changing the rules mid-round. Viewers' scores are sacred. Migrations run in an effect-free isolate and are checked against five round states. Riff then re-runs them against live progress at the exact moment of approval, so chat accepted during review is never lost.
- Bot loops. Riff's own chat replies came back through EventSub as broadcaster messages and could trigger commands. An echo ledger (Helix message IDs, plus text fingerprints for announcements) and a hop limit stop the loop.
- Exactly-once effects on an at-least-once platform. EventSub redelivers. Admission is durable, duplicate identities survive restarts, and a per-round journal lets the creator press Verify round to replay every decision and prove the scoreboard.
- Making it fun to watch. Our first overlay was honest but looked like grey debug boxes, and long quiz explanations were cut off. We rebuilt it for broadcast with team colours, countdowns, answer reveals and a text-fit engine. We also redesigned the whole studio to be responsive from 375 px up.
Accomplishments that we're proud of
- One sentence to live on Twitch in under a minute. The compile in our demo took 16.5 s, followed by 10/10 checks and a real chat-to-overlay run.
- 1,294 automated tests, plus 11 Playwright end-to-end runs with axe accessibility checks.
- A security model where generated code never holds credentials, network access or authority, with a stop button that cannot be blocked.
- 40 features tracked in a public feature matrix with honest Implemented / Bounded / Live columns, built from 31 October to 2 November 2026.
What we learned
- AI-generated behaviour becomes trustworthy for a live audience when the host owns verification, limits and the off switch, and shows the evidence in plain language.
- For runtime generation, structured output plus one bounded repair turn beats an open-ended agent loop: it is predictable, auditable and fast.
- Twitch EventSub is at-least-once by design. Duplicates, reconnects and restarts have to be first-class from day one.
- Real production runs find bugs that unit tests don't. Every live pass found something.
What's next for Riff
- Twitch sign-in so any streamer can onboard themselves, with stronger isolation for generated code.
- Live validation on an Affiliate channel for polls, Channel Points and Predictions.
- Sound effects behind creator consent, and more effect assets.
- A public recipe marketplace for remixing interactions between channels, and a Twitch Extension panel for viewers.
Built With
- aws-lightsail
- caddy
- codex-sdk
- docker
- drizzle-orm
- fastify
- javascript
- node.js
- obs-studio
- openai-codex
- playwright
- quickjs
- react
- sqlite
- terraform
- twitch
- twitch-eventsub
- twitch-helix-api
- typescript
- vite
- vitest
- webassembly
- websockets
- zod