We wanted to get into music making, but were lacking proper software for it.

What it does

Our app takes lines drawn on a paper and turns them into melodies. The process goes as such:

  1. Draw colored lines on a paper in red, green, and blue
  2. Take a picture of your masterpiece and upload it to
  3. Listen as your drawing comes to life!

How we built it

We used openCV running on an azure server to identify the lines and separate colors, then move that data onto a python app that generates midi and wav files, that we then play back to the user.

Challenges I ran into

Our original plan was to allow the user to take a picture in our android app, and it would upload to our server, then we would play it back to them. We couldn't get android and python to communicate the picture among each other.We solved that by opting to make the entire service a web app, that can be accessed by both personal computers and mobile devices.

Accomplishments that we're proud of

The openCV code was very well optimized. We brought processing time down from nearly a minute to just a couple of seconds. The midi generation library was very difficult and counter intuitive to work with, but we managed to edit its source to fit our needs. The website design is very sleek, benefiting the user experience. Can be used to transfer information, like a prettier QR code.

What we learned

Each one of our team taught the others a bit about their specialty. Joe taught the team computer vision and algorithms, Chris taught us music theory and digital music processing, and Reshef taught the team about design and UI/UX.

What's next for Melody

With more time to learn about android and iPhone development, a proper app could be made, so the users would not rely on a web app.

Try to standardize the format a bit more, and explore possibilities in machine readable labels, that look better and more memorable than QR codes.

Share this project: