Waifu Maker

Inspiration

Anime, idol culture and AI tinkering leading to wanting to make real anime girl (and boy) companions

What it does

Provides a way to use LLMs through a talking AI companion with realistic voice and have it become an immersive real experience.

How we built it

Combining Elevenlabs, Mistral, Perplexity and FastAPI along with three.js into a full application where we animate Vroid (vtuber) models with speech. Also includes an AI memory system.

Challenges we ran into

Issues with engineering saving of memories, issues with web services running differently to local processes, some CORS issues and slow voice processing making it difficult to get the whole process running. Immense difficulty with timing of mouth movements.

Accomplishments that we're proud of

Getting the models running live and talking as a full integration at https://ai-waifu-platform.onrender.com/

What we learned

Learned about how to create realistic mouth movements and overall engineer a feeling of realness through the models. Also how to construct AI memory.

What's next for Waifu Maker

Add a subscription model for voice usage and a cheaper voice model as a part of the live app. Make it available as a beta for people to use and play with.

Built With

• elevenlabs
• fastapi
• javascript
• mistral
• perplexity
• python