Understudy

An AI co-host that learns a streamer's bits, vibes, and voice — then stands in live when they're away, keeping chat company until the star returns.

Comment

Inspiration

Every streamer knows the guilt: life happens, you go offline, and chat is left staring at a dead channel. Regulars trickle away, the community drifts, and the creator burns out trying to be always-on. We asked a different question: what if the channel could hold the fort — not with a VOD rerun, but with a co-host who actually knows the bits, the in-jokes, and the vibe?

What it does

Understudy is an AI co-host that stands in for a streamer when they step away:

  • Learns the show — the streamer writes their persona (bits, catchphrases, tone, community lore) in a control room, and Understudy performs their voice, not a generic assistant's.
  • Goes live in their place — a PNG-tuber-style avatar with emotional states takes the stage, talks with chat, runs polls, and keeps the community company until the star returns.
  • Discloses honestly — the overlay and chat announce that Understudy is an AI standing in. No deception, ever.
  • Knows its place — moderators get a kill switch to end the takeover instantly, and the streamer can reclaim the stage at any time.

How we built it

We built it with OpenAI Codex as our pair programmer — the core of the project was written inside a Codex session, from scaffold to ship:

  • src/server.ts — TypeScript/Node core: profile validation, Twitch chat wiring, takeover lifecycle, moderator stop handling, poll management, and model/voice endpoints.
  • Persona engine — inspired by our earlier "Pantheon: Council of Immortal Minds" experiments, it models the channel's voice instead of a one-size-fits-all chatbot.
  • Control room — a web UI where the streamer writes the persona, flips the takeover switch, and watches their understudy work.
  • PNG-tuber avatar — layered sprites with emotional states generated with Pillow, so the stand-in feels like a cast member, not a widget.
  • CODEX_LOG.md — a running log of the Codex session, because the build story is part of the product story.

Understudy is free by default: everything runs locally on the streamer's machine, with an optional OpenAI integration they can switch on. (Codex asked us before adding any paid API — we said yes only as an opt-in.)

Challenges we ran into

  • Persona fidelity — making the stand-in sound like the streamer, not like a helpful robot. Early drafts were far too polite; real streamers are gremlins.
  • The ethics of an AI on stage — we decided disclosure is non-negotiable: the overlay says "AI understudy," chat is told, and mods can pull the plug at any moment.
  • 4-day scope — cutting a full VTuber rig down to a PNG-tuber with emotional states kept the build shippable inside the hackathon window.

What we learned

  • Codex is shockingly good at owning an entire server end-to-end when you hand it a crisp brief and clear guardrails.
  • The hardest part of an AI stand-in isn't the AI — it's the handoff: when does the human come back, and how does the AI leave the stage gracefully?

What's next

Voice for the understudy's mic, "while you were away" recap clips, and a multi-understudy mode for co-op channels.

Built With

Share this project:

Updates

Submission history