ModScrim

ModScrim: a fire drill for your Twitch mod team. Real screens from the demo room.
Live on Harsh's real Twitch channel: signed in as harshbhx, Twitch connection Connected, EventSub chat subscription Active.
Twitch's own settings page lists ModScrim under the apps authorised on the account.
The problem: filters look for exact words, raiders add spaces, and most mods never get to practise.
How it works: Drill, Grade, Fix.
The drill on a mod's phone: every catch scored, false bans flagged, nothing posted to Twitch.
After the drill: 11 of 14 attack lines get past; fixes with 0 real-chat matches; 4 to 17 blocked on a fresh wave (demo, local estimate).
What makes it different: other tools act during the raid; ModScrim works before it.
Safe by design and built with Codex in four days: 119 tests passing.
Try it: the public demo needs no Twitch account. Scan the QR code.

ModScrim: raid drills for your mod team

Your mods have never practised a raid. Now they can. ModScrim runs fire drills for a Twitch channel's moderators: a private practice chat floods with spam built to slip past this channel's own filters, the mods race to clear it, Twitch grades the filters, and the fixes are proven before the streamer pushes them with one click. Try a full drill in about a minute in the no-account demo.

Inspiration

Twitch mod teams are volunteers, and the only practice most of them get is a real raid, with real harm and an audience watching. Streamers tune their filters by guesswork: a new blocked term might ban a regular, and nobody can see that risk until the term is live.

A blocked term catches "free nitro" but the same scam with number swaps, stretched letters or spaced letters slips past

We started from the research, not from a hunch:

Mods learn on the job. In a survey of 1,053 active Twitch moderators (Seering and Kairam, 2023), about 12% had trained another mod in the past month, and most learn from streamers (76%) and other mods (69%) rather than Twitch's tutorials (49%).
Filters leak both ways. An audit of AutoMod with test accounts let through up to 94% of hateful messages on some test sets and blocked up to 89.5% of harmless messages that used sensitive words; spacing and punctuation changes lowered what it caught.
Raids are coordinated, and bots adapt. A study of hate raids on Twitch describes real-time attacks run by people and bots together, and volunteer mods report spam bots that reword and respace messages to dodge filters. That is exactly the behaviour ModScrim's drill imitates.

Schools run fire drills because nobody is in danger. We wanted the same for chat: a fake raid, real filters, real feedback. Every source we used is listed, with a note on why it matters, in docs/references.md.

What it does

Drill, Grade, Fix: the seven steps of a drill

Drill

ModScrim reads the channel's public blocked terms and AutoMod levels. The streamer picks tactics, speed and a slip window, and shares a mod link with a six-digit code. The practice chat fills with attacks aimed at those exact filters, mixed with harmless banter. The tricks (spacing, dotted letters, number swaps, stretched letters, look-alike letters, emoji inside words, zero-width characters) live in tricks.ts:

blocked term:  free nitro
plain:         free nitro for the first 50 -> nitrodrop.invalid      BLOCKED
number swap:   fr33 nitro for the first 50 -> nitrodrop.invalid      SLIPPED
spaced:        f r e e nitro, claim yours at freebits[.]invalid       SLIPPED (needs a bot rule)

Mods delete, time out or ban from any device (D, T, B on the keyboard, tested here). The first action wins, a line nobody touches slips through, and banning a harmless regular counts as a foul. The streamer watches every move live, and an OBS overlay shows viewers the drill as blurred bars, never readable attacks.

Grade

Every attack line goes to Twitch's Check AutoMod Status, so the score comes from Twitch, not from us. The rule finder then proposes up to six blocked terms that catch what got past, and rejects any term that would match the last 7 days of the channel's real chat or a harmless banter line.

Fix

The proposed terms are tested on a fresh wave of 30 attacks (waves.ts), built from a new seed with tricks the rule finder never saw. Push stays disabled unless real-chat matches are zero. The streamer approves, ModScrim adds the terms on Twitch, and Undo removes exactly the terms that drill added. An after-action report keeps per-mod scores, grades and results.

What makes it different

Live bots and AutoMod act during a raid, and Twitch's AutoMod testing tool checks one typed phrase at a time. ModScrim is the rehearsal and the filter test that come before a raid: it trains the team, aims attacks at the channel's own filters, has Twitch grade them, and proves the fix on real chat. It never acts on real chat itself.

How we built it

With OpenAI Codex, from a written brief

We treated Codex as a teammate that needs a precise brief, so we wrote the brief first:

Specs first. The PRD sets scope, copy and safety; the SRS sets the architecture, data, interfaces, algorithms and tests.
Rules for the agent. AGENTS.md tells Codex which spec wins, which safety rules it must never break, and that it may not add a feature, endpoint, dependency or table the specs do not list.
Tickets, tests first. Work was split into numbered tickets, each a GitHub issue. Codex wrote the tests the SRS names, watched them fail, then implemented; npm test and npm run typecheck had to pass.
Humans decide. We reviewed every change against the safety rules, tested it, and kept a dated trail of decisions and test evidence in docs/LOG.md.

On Twitch

OAuth sign-in, with tokens sealed with AES-256-GCM; they never leave the server, and validating them hourly keeps them honest.
Get Blocked Terms and Get AutoMod Settings (moderation guide) to aim the drill. AutoMod is read-only: we never call its update endpoint.
Check AutoMod Status to grade, paced to the per-channel limits Twitch announced.
Add / Remove Blocked Term for push and undo, only after the streamer approves. All Helix calls are in helix.ts.
EventSub webhooks for the 7-day chat log, with every signature checked on the raw body before it is read, as Twitch's webhook rules require.

On Cloudflare

Architecture: clients, one Cloudflare Worker with the ChannelRoom Durable Object, D1, Queue and R2, and Twitch OAuth, Helix and EventSub

One Worker serves the Hono API and the React app. Each channel gets one Durable Object with its own SQLite database that owns the drill clock, the hibernating WebSockets for the console, mods and overlay, the chat log, grades and reports. D1 holds users and sealed tokens; a Queue and R2 carry report exports. The engine is pure TypeScript with a seeded random generator: no AI runs at runtime, and the same seed always builds the same drill.

Challenges we ran into

Twitch said "slow down" while our counter said "go". Grading once sat at "0 of 14 checked" (issue #249): we sent a whole phase in one request, but Twitch counts its limit per line. For a normal channel that limit is 5 lines a minute, so grading the 14 attack lines of a drill takes

$$t = \left\lceil \frac{14 \text{ lines}}{5 \text{ lines/min}} \right\rceil = 3 \text{ min}$$

We now pace requests by line, tell the streamer when the next batch goes, and fall back to a labelled local estimate if Twitch keeps refusing.

One alarm per Durable Object. Cloudflare gives each object one alarm, and it can fire twice. A drill needs a line clock, grading ticks, reports and clean-up, so every job runs through one schedule table and is safe to run twice.
Redaction had to be structural. "Never show attack text on stream" is easy to break by accident, so overlays only ever get a RedactedLine of { lineId, user, len }, and a test checks every overlay frame of a full drill.
Proving a rule without flattering it. Testing terms on the lines they were learned from would overstate them, so the proof uses a held-out wave.

Accomplishments that we're proud of

It works on a real Twitch channel. On our own channel the console is signed in with Twitch, the EventSub chat subscription is active, and Check AutoMod Status grades the drill live in paced batches, a few lines a minute (see the gallery).
Anyone can try it in a minute in the no-account demo, which never calls Twitch.
Safety is code, not a promise. Every generated link ends in .invalid (test), slurs exist only as placeholders drawn as blocks (test), no token ever reaches the browser (test), nothing reaches Twitch without approval, and every push can be undone.
119 tests in npm test (engine and Worker with the Durable Object), plus Playwright runs of the full demo loop and the safety checks against the live app on every push to main.
A deterministic engine: any drill can be replayed exactly from its seed.

What we learned

Specs are the best prompt. A precise PRD, SRS and AGENTS.md kept Codex in scope and turned every review into a check against something written down.
Tests first matters most for safety rules. Redaction, .invalid links and undo should be proven, not remembered.
Honest labels build trust. "Local estimate" and "needs a bot rule" help a streamer more than a confident wrong number. Some tricks, like spaced letters or zero-width characters, cannot be written as blocked terms at all, and ModScrim says so.
What we have not measured yet: the full comparison of Twitch's grading against our local engine on a test channel is still to run, so we make no claim of measured improvement on a real channel. Our known limits are written down.

What's next for ModScrim

Run drills with real mod teams and learn whether they would schedule them without a hackathon reason.
Rules suggested per tactic, not only per drill, and future waves aimed at the tricks a team misses most.
A Twitch Extension, once there is time for Twitch's 5 to 14 day review.
A path to a product: who might pay, tiers and risks are sketched in docs/business.md.

ModScrim

Your mods should not meet their first raid live on stream. With ModScrim they meet it in a practice chat nobody else can see, Twitch tells you which filters held, and the fix is proven, approved and one click from undone. Built by Harsh Bhanushali and Dhairya Patel with OpenAI Codex.

Blow the whistle before the raiders do.