Melby-sama

• 

  Melby-sama watching a MLB YouTube video

Inspiration

We were inspired by the growing popularity of virtual influencers and the desire to enhance fan engagement in the digital age. Traditional mascots have limited reach online, and we saw an opportunity to create a dynamic, interactive character that could connect with fans on a deeper level during MLB livestreams. The goal was to bridge the gap between the excitement of live sports and the immersive potential of AI-driven virtual experiences.

What it does

Melby-sama is an AI-powered VTuber designed to be the next-generation mascot for MLB livestreams. Melby-sama interacts with fans in real-time, providing commentary, answering questions, and personalizing the viewing experience. This creates a more engaging and interactive environment for viewers, fostering a stronger connection with the sport and the teams.

How we built it

Melby-sama's AI capabilities are powered by Google Generative AI API, which can make fun conversations with the fans. Azure speech synthesizer is then used to generate the voice of Melby-sama. We used Vtube Studio, and OBS for avatar creation, motion capture, and streaming. Lastly, pytchat captures the audience's comments in the live chat so that Melby-sama can interact with the audience directly. It is thanks to all these technologies combined, that we can build a project with such complexity within two weeks.

Challenges we ran into

• Time Constraints: Availability of team members required us to prioritize features and work efficiently.
• Technical Difficulties: Integrating different software components (VTuber software, AI platform, streaming tools) presented technical hurdles. We encountered challenges with live chat integration, AI response latency, and ensuring smooth streaming.
• Balancing AI and Personality: Creating a believable and engaging personality for Melby-sama was crucial. We worked to fine-tune the AI's responses to ensure they felt natural.

Accomplishments that we're proud of

• Functional Prototype: We successfully developed a working prototype of Melby-sama, demonstrating the core functionality of real-time interaction with fans.
• Integrated AI: We integrated generative AI to enable dynamic responses and personalized interactions.
• Unique Character Design: We created a visually appealing and engaging character design that embodies the spirit of MLB.
• Team Collaboration: We effectively collaborated as a team, overcoming technical challenges and meeting the hackathon deadline.

What we learned

• VTuber Technology: We gained practical experience in avatar creation, motion capture, rigging, and streaming.
• AI Integration: We learned about using AI for natural language processing and real-time interactions.
• Google Cloud's API: We acquired the skills to use Google Cloud's API seamlessly.
• Fan Engagement Strategies: We explored best practices for online community building and fan engagement within the sports context.
• Project Management: We honed our skills in rapid iteration, task prioritization, and teamwork under pressure.

What's next for Melby-sama

We plan to further develop Melby-sama by:

• Expanding AI Capabilities: We want to enhance the AI's knowledge base, improve natural language processing, and explore more advanced features like personalized content generation.
• Improving Interactivity: We aim to create more interactive experiences for fans, such as polls, games, and Q&A sessions.
• Integrating with MLB Platforms: Our long-term goal is to seamlessly integrate Melby-sama into official MLB livestreams and digital platforms.
• Exploring Sponsorship Opportunities: We believe Melby-sama has the potential to create new sponsorship opportunities for MLB and its partners.

Built With

• azure-cognitiveservices-speech
• google-genai
• mss
• opencv-python
• pyaudio
• pytchat
• python
• python-dotenv