Inspiration

The best moment of a stream scrolls away. The clutch comeback or the raid that lands at the wrong time ends up in a chat log nobody re-reads and a VOD most viewers never open. Raided and late-arriving viewers also land mid-story with no idea what just happened. We wanted to turn a stream's best moments into something a newcomer can read in seconds, without the streamer doing extra work.

What it does

ComicToons captures a live Twitch stream and its chat, drafts a 4-6 panel comic page from them, and holds every page for the streamer to approve. Approved pages show in an overlay and in a catch-up view for latecomers, chat can vote on the lettering, and pages bind into a shareable episode. Chat is treated as reaction evidence, never as proof of what happened on screen.

Try it in two minutes with the keyless Sample demo (npm ci, then npm run demo): three invented stories, no Twitch account, no API keys.

How we built it

  • Capture: Chromium screenshots the playing Twitch video every 10 seconds, keeping each frame's hash and video clock. Chat arrives over IRC and is filtered: commands are dropped, and deletions and opt-outs are honoured. Any page built on a withdrawn quote is withdrawn too.
  • Plan: a vision model returns a 4-6 panel plan. A validator rejects plans with missing frames, fewer than four distinct frames, or changed chronology.
  • Draw: each session picks a provider (OpenAI, Gemini, Pollinations, or Cloudflare Flux). A source mode composes the real captured frames with exact, attributed chat lettering and calls no image API.
  • Review: a second vision pass checks faces, lettering, chronology and fidelity against the source frames. It fails closed: anything uncertain is held and can never be released.
  • Release: only the streamer's approval sends pages on to the overlay, catch-up view and export.

TypeScript throughout: a Fastify server, a Vite web app, and shared story and layout rules, plus Twitch EventSub, polls and an OBS browser-source overlay.

Built with Codex. We supplied product direction, visual requirements, approval rules and testing authorization. Codex inspected and edited the repository and implemented and debugged the Twitch/OAuth/chat integration, role separation, evidence capture, provider adapters, comic lettering, moderation protections, catch-up refresh, chat voting, tests and documentation. It used the local UI and OBS for authorized QA and inspected generated images against the captured frames. Parallel Codex fix and QA agents handled the Cloudflare and regression-test tasks. Codex session: 01a1989f-bf57-71f0-a88f-c10dad4473ec.

Challenges we ran into

  • AI illustration vs. fidelity. Image models produced pages, but our strict review held every one for source, face or lettering mismatches. We chose not to release or claim an AI-drawn page that didn't pass, which is why the source-frame mode exists. Provider quotas also limited testing.
  • Chat is not evidence. It is useful as reaction but unreliable as fact, so the writer never invents streamer speech.
  • Real streams are messy. Reliable capture, deletions, opt-outs and reconnects took far more work than the happy path.
  • Time. We cut scope and kept the docs honest about what was verified.

What we learned

  • Trust matters more than polish: a fail-closed review and a human approval gate make this something a streamer can rely on.
  • Real captured frames plus exact chat can be more faithful than generated art for a recap.
  • Writing down honestly what works and what doesn't made the project easier to test and trust.

What works, and what doesn't yet

  • Works: the keyless Sample demo, the Studio approval queue, overlay, newcomer catch-up and chat votes (on sample data), real capture of our own stream, and read-only viewing of another live channel.
  • Not yet: a fresh live end-to-end proof in source-frame mode, an AI-drawn page that passes review, and speech transcription (not implemented).

What's next

Finishing the live end-to-end proof, getting AI-illustrated pages through review, and adding speech understanding so recaps can reflect what the streamer actually said.

Built With

Share this project:

Updates

Submission history