Streamer Twitch Convos

Clone streamers voices. Generate conversation between two streamers, giving a topic to discuss and a emotion. Output audio is playable via a in-browser music player.

Comment

Inspiration

I was inspired by the Ai Spongebob Livestream and wanted to build something similar but with Twitch Streamers.

What it does

Give the app a streamer user name - it will then clone the streamers voice based off twitch clips. You can then select that streamer and another, give a topic, and a emotion. A 30 second audio clip will then be generated based on the information given. It will also generate elevator music based on the emotion that will play in the background of the audio clip.

How we built it

The frontend is build with nextjs. The backend is in Python.

Frontend: Drizzle ORM, madzadev/audio-player, Tailwind, NextJS14.

Backend: FastAPI for building the API, pydub for audiomixing, PeeWee ORM , xtts_v2 for the cloning of streamers voice / text to speech. Qwen2.5-72B-Instruct for Convo generation between streamers. Using twitch api to download and store streamer clips which are then used for the text to speach. Fullstack: neon.tech (postgres) for database, AWS sqs for sending generate request from nextjs to python, AWS S3 for storing the generated audio. In the database storing title of generated audio and S3 URL.

Challenges we ran into

Deploying the Python backend was a challenge. Using two languages for a full stack - would prefer to do everything with just NextJS but Python was easier especially for the cloning of streamers.

Accomplishments that we're proud of

Audioplayer in browser. Using SQS to talk between frontend and backend. Cloning the streamers voice with Twitch API - Clips. Convo generation.

What we learned

Being specific with prompts for AI to get JSON format of convo between the two streamers.

What's next for Streamer Twitch Convos

Video would be nice. Deployment of the backend. Twitch stream where users can watch. Podcast / audio live streamer that plays the generated content. Add auth. Would like more advance / better prompt for convo generation.

Built With

Share this project:

Updates