Vocal Pro X

Karaoke on-the-go powered by AI! State-of-the-art deep track splitting, lyric extraction and alignment, and image generation all combined to make the perfect impromptu vocal performance!

Comment

Inspiration

As avid music lovers, we wanted to use AI to create a fun and interactive way for people to enjoy their favorite music. Since most people interact with their favorite music by singing along, we thought it would be cool to try to improve the karaoke experience!

What it does

Our mobile app allows people to search YouTube for their favorite song (no matter how obscure or whether it's a cover of a song) and turn it into an instrumental karaoke version + accompanying lyrics to sing along to, for any language supported by Whisper. The app will also generate an associated image, based on the lyrics of the song, using Stable Diffusion.

How we built it

  • Front-End
    • Using Swift to build the front-end of the mobile application
    • Set up front-end to allow for embedded YouTube video search
  • Back-End
    • Set up a Flask server on a Deep Learning VM on Google Compute Engine
  • Inference Pipeline
    • Use pytube to download the audio stream of the selected YouTube video
    • Use AudioConverter to change mp4 format to wav
    • Use demucs to split the audio into vocals + instrumental tracks
    • Use a modified version of whisper to transcribe vocals into lyrics, with associated word-level timestamps
    • Use Stable Diffusion to generate an associated image using the lyrics as the prompt

What we learned

  • How to set up a Flask server on GCP Infrastructure
  • How to deploy LLM models (Whisper, StableDiffusion-2.1) to an API endpoint for efficient prediction serving
  • That there are a lot of cool LLMs out there that deal with audio processing

Challenges we faced

  • Setting up deep learning models in production for scale (running out of GPU memory)
  • Getting highlighted lyrics lined up word-for-word with audio

What's next for Vocal Pro X

  • Faster track splitting + lyric transcription inference
  • Faster + higher quality image generation based on lyrics
  • Record yourself singing along
  • Share recordings on social media
  • Add vocal effects to singer’s voice in real-time
  • Ability to pitch-shift the instrumental in real-time

Built With

Share this project:

Updates