What it does

A music player that allows you to adjust the volume of each instrument like the conductor of an orchestra using an Arduino hooked up to an IMU.

How We built it

We used a Python Deep Learning algorithm to split the tracks into separate components and then we used an Arduino hooked up to an IMU to then provide control data. This was all then combined into a PyQt5 application which allows the user to search for, download and split tracks from the Octave API. We then played all the components in parallel and controlled the volume of each using hand gesture controls on the Arduino.

Challenges We ran into

Getting the Octave API to work correctly took a great deal of time as there were plenty of compatibility issues. We managed to rebuild most of the player provided to us and used it to provide '.ogg' files through user search. The Arduino IMU had also given us some issues since the Library did not effectively handle I2C communication.

Accomplishments that We're proud of

Finally getting a functional end product which did something unique and fun.

What We learned

How to use Deep Learning and TensorFlow, Arduino programming, Threading and scheduling, Linux commands and Octave API.

What's next for conductify

This system can be built completely on a smartphone and would provide the user with a fun and interactive new method for appreciating their favorite music. It would be great to train the Deep Learning model further as well to get more consistent results.

Share this project: