Our team sat and discussed ideas for a long time, and as a team of music-lovers, we kept drifting towards music-related projects. Jacob threw this idea out while we were brainstorming, and it seemed interesting enough to at least look into. We then stumbled across some recent research and development into "unmixing" methods, which provided a huge boost to our drive and gave us the starting point we needed.
What it does
unmixr is a website. Given a song (from a YouTube video), it will isolate or remove an instrument from the song. You then have the option of playing the new song alongside your inputted video or downloading your newly made audio file. This is all done by the use of machine learning: the audio file is analyzed and separated into perceived instruments, which are then removed or made into their own audio files.
How we built it
Our project largely consists of two major components:
- A standard Flask + Angular web app that we built from scratch
- The advanced machine learning methods of open-unmix that we modified and integrated with our web app As well as many of other tools:
- Google Cloud products were essential for deployment & processing.
- We hosted the front-end on Google App Engine
- The back-end on Google Compute Engine.
- We configured our compute engine instance with a GPU to strengthen processing power.
- We used Python to scrape the audio from YouTube videos
- We used the Youtube API to load the video & sync our newly created audio with it
Challenges we ran into
Integrating the aforementioned research on unmixing was a major challenge. Converting this model and these ideas into our website proved challenging, and required a lot of studying and tweaking for us to get right. Another difficulty was connecting to a domain name. The connection of google cloud with domain.com was giving us trouble, and we ended up unable to use 'unmixr.tech' as we had planned.
Accomplishments that we're proud of
We're proud of our idea. We're proud of our teamwork and communication in a stressful environment. We're proud of our UI. We're proud of how much we've learned. Most of all, we're proud of creating a polished and interesting tool, that we would use, in the short amount of time given to us.
What we learned
Each of us learned different things. All of us made an effort to work in stacks that we aren't used to, as a chance to grow and learn (and of course there were some things none of us are used to). We learned some valuable teamwork skills as well - little sleep in an uncomfortable setting really puts a team to the test in the morning.
What's next for unmixr
In the future, we'd like to further study and improve upon the machine learning methods we used, to better handle lossy files and improve upon the quality of our unmixes. We'd definitely like to add an input for actual files as lossless file types such as .wav (instead of what we retrieve from YouTube) unmix at a much higher quality. Ben envisions turning this into more of a music production tool. Many artists would benefit a lot (particularly in sampling, remixing, and studying mixes) from what we've already made, and even more so if we allowed the isolation of more instruments. Ultimately, our goal is to make the most accessible and versatile tool that we can from our concept.