• 

  Twitch live

• 

  Song generated

• 

  Chat writing song

• 

  Voting

About the project (Markdown)

Inspiration

Every stream has that moment: chat is on fire, someone types the perfect line, and it scrolls away forever. I wanted chat to make something that lasts. Not a poll or a meme, but a real song, written together, live, with every writer's name on it.

What it does

Verse vs Verse turns Twitch chat into a songwriting room.

• Write. Viewers type !line <lyric>. Every line shows up on stream with its writer.
• Battle. An AI judge shortlists the best lines and they go head to head. Chat votes !1 !2, with live counts on stream and the options posted in chat.
• Sing. About 12 seconds later, an AI band sings the winning line word for word.
• Watch. Every line becomes its own animated scene, and the scenes follow the words: "crown me king" puts a crown on the character's head, "dancing on the moon" has it dancing on the moon. Lyrics play karaoke-style, and the character's mouth moves only on sung words.
• Release. Each round the song grows and plays again. At the end chat gets a finished song with a title, writer credits, a share page and a music video.

It runs on real Twitch: EventSub reads chat, a bot replies through Helix, the overlay drops into OBS as a browser source, and a built-in broadcaster can even go live without OBS.

How I built it

OpenAI Codex built Verse vs Verse end to end, stage by stage, in one long thread:

1. Foundation: Twitch OAuth, EventSub over WebSocket, the round engine (write, vote, produce, play, release), the OBS overlay and the streamer dashboard.
2. Models: an OpenAI-compatible judge that ranks lines and names songs, moderation, and the music and speech adapter.
3. The scene engine: an LLM plans each line as a scene (keyword, mood, character pose, props) and a pure-SVG renderer draws every frame, with MP4 export through headless Chrome and ffmpeg.
4. Live operation: retries, release recovery, lyric fidelity scoring, loop mode, live timers, and the chat commands !help, !queue, !song and !skip.
5. The look: a hand-built library of 40 props with anchors and motions, a character that dances on the beat, and props and camera moving with the music.

Music, speech-to-text and voice run through Pascal, a voice AI platform I built that provides music generation, speech-to-text and text-to-speech through one API. The music model is a single setting, so a better model plugs straight in.

Challenges I ran into

• The AI wanted to write its own song. With song-section tags, the music model rewrote 40–80% of the lyrics. A strict prompt with plain lines brought that to zero invented lines in 10 test renders, and every clip now gets a fidelity score, so chat hears exactly what chat wrote.
• Keeping the picture on the beat. Long renders only time some lines, so I built a word-level aligner that finds every repeat of each line and keeps scenes and karaoke in sync.
• Live pacing. The first version waited 13 minutes for the final song. Using the last round's clip as the release made the end instant. ## Accomplishments that I'm proud of
• A full song written live on twitch.tv by chat, sung word for word.
• Scenes that follow the lyrics instead of just decorating them.
• An animation engine where the art style lives in its own folder, so new styles drop in.

What I learned

Viewers care far more about hearing their own words sung than about audio polish, so faithful lyrics beat everything else. And for live shows, the fastest path wins: every second of waiting on stream is a second chat gets bored.

What's next for Verse vs Verse

• Channel vs channel: two communities write against each other, and the best lines win the song.
• More styles: anime, painted and pixel styles, plugged into the same engine.
• Clips: every finished song as a ready-to-post short.

Built With

• codex
• express.js
• ffmpeg
• helix
• node.js
• openai
• sqlite
• twitch
• typescript
• websocket