Musync

The home page our site
The login page of the website
The final video view

Inspiration

As musicians, the COVID-19 pandemic has disrupted our ability to physically perform pieces together in bands, orchestras, and small ensembles. Meeting in online video calls to perform is also difficult due to issues with audio lag. Another option is to record videos individually and then manually edit them together which is a very laborious and painstaking process. This inspired us to create Musync: an app that synchronizes recordings via an easy-to-use interface and efficiently integrates them into a polished video.

What it does

Each musician creates a Musync account and joins a "class". The lead musician can then upload a lead recording (or a metronomic click-track) for the class to play along with. Each member records himself playing an individual part (while listening to the lead recording with headphones), then uploads the video file to Musync. Our app synchronizes the parts by listening for a distinctive clap at the beginning of each recording. The final output is a synchronized music video which features the musicians on the screen (grouped by instrument).

How we built it

To create this application, we used Bulma for our frontend CSS framework, Google Cloud Services for our backend, and the ffmpeg file editing library to manipulate the video and audio files. To process the recordings, we start by splitting the mp4 video from each user into a video and audio part. For each of the audio files, we detect the distinctive clap by analyzing spikes in volume and trim the audio and video files at the clap time so they are all synchronized. Then, we overlay the trimmed audio files into a master audio file and arrange the trimmed video.

Challenges we ran into

One of the hardest part of developing this application was trying to use ffmpeg and the google cloud server because of the time it took to load dependencies each time we loaded the application to test the functionality.

Built With

bulma
ffmpeg
pydub
python

Submitted to

PennApps XXI
- Winner Route: Best Music Hack

Created by

I worked on the splitting and trimming of the video and audio files using ffmpeg and pydub so they are synced to the clap of the musicians.

Ayush Varshney
I worked on grouping the instruments and stitching the final video together with ffmpeg. I also wrote the Bulma front end.

Aaron Mei
I worked on the Google App Engine web server, and wrote most of the backend. I also worked on integrating my teammates' code to make a complete application.

Ezra Bernstein
I wrote the code to detect the clap at the start of the recording and worked on the Google Cloud Storage database. I recorded myself on the violin and put together the video for Devpost.

Gabriel Wu