We all played and loved Rock Band back in the early 2010s. One of us (Jason) played it so much that he became an actual drummer. How many more people might become musicians if they could play Rock Band anywhere, on any song, with friends anywhere in the world, while playing real instruments?
What it does
- Submit a music video from YouTube
- Share a lobby URL with your friends, video chat while you wait
- Everyone picks an instrument (drums, piano, vocals, bass, other)
- Press START to start recording your part
- Each player will be shown 2 amazing things:
- The original music video, WITHOUT their instrument's audio
- Their instrument's audio, transcribed into notes and scrolling across the screen
- Afterwards, we turn your group into a new and improved music video, with these parts:
- All instrument parts that were recorded
- The original audio of the parts that were not recorded
- The original music video, sans audio
- The FOREGROUND of each player's video feed (we used video segmentation), superimposed over the music video
At the end, players can download their generated music video. The results are truly hilarious and very shareable.
How we built it
- Agora for multiplayer and recording audio and video
- youtube-dl to download YouTube music videos
- An audio source separation AI (Spleeter)
- A drum transcription AI (ADTLib)
- A piano, voice, bass transcription AI (CREPE)
- canvas for displaying scrolling MIDI tracks
- Video segmentation AI (OpenCV)
- ffmpeg for mixing audio tracks and videos.
Challenges we ran into
- Video segmentation was low quality and noisy at first; Josh found a way to smooth it out
- Working with CV and music IR is difficult because iteration cycles are slow (even with <10s song clips)
Accomplishments that we're proud of
- We finished it! Holy cow was this an ambitious project.
What we learned
- Audio source separation and music information retrieval AIs have gotten amazingly good in the last 2 years.
What's next for Audia
- We think this is genuinely motivational and we are going to share this as widely as we can. Hopefully it will inspire more people to explore their musical sides. Also, it's really fun.