Melodica

• 

  Main Melodica Interface for Music Visualization

• 

  Melodica Loading Screen

• 

  Song Upload Screen

• 

  Moises Workflow Interface

• 

  Moises API Output Interface

• 

  Basic Pitch Interface

Inspiration

As a group of aspiring musicians and a newly formed band, we looked for a fast and useful tool to help us learn and analyze songs we wanted to play for practices and performances. We wanted the project to be able to visualize and isolate different instruments and elements of the songs to allow band members to play along and practice to songs without losing other instruments or having to play over existing plays of their parts. The opportunity to make a tool to help us potentially solve our problem gave us an incentive to try and make something we could use and be proud of, so the goal of this project is to aid us and any other aspiring musicians with their practices.

What it does

Melodica is a web app that allows users to upload a song or music track and will separate and visualize the different instrumentals and audio tracks, and provides a timed chord feature that allows users to sing and play along to the song. Melodica offers audio levels and pitch visualization for each instrument, and offers the ability to both manipulate and download each instrument track.

How we built it

Melodica uses two main artificial intelligence technologies to analyze the waveform of a user-inputted song. Moises separates the various instrument stems and extracts the song's qualities using multiple pre-trained recurrent models and Spotify's open-source MIDI converter Basic Pitch converts the isolated audio data into a MIDI format. The web application's front end uses a Flask architecture that embeds a P5.js script to draw various audio visualizations of the song. Three.js was also used to make the haha funny spinning band for our loading screen while the program processed the song on the back end.

Challenges we ran into

• P5.js being inconsistent in different environments
• Pushing the limitations of P5.js
  • Synchronizing different elements of the visualizer with real-time audio manipulation
  • Attempting to create functional versions of necessary user interfaces
• Creating the library that converts the raw song input to an analyzed instrumental MIDI output
• Piecing together individual components of the project that each member contributed
  • Difficulties merging the many different elements due to variations in languages and APIs used

Accomplishments that we're proud of

• Fully functional and visually pleasing front end and user interface
• Managing to interweave several different software and technologies for a polished final product
• Finishing a project

What we learned

• We learned to better plan the software we want to use
  • With a more ambitious front end and graphics needed, we found that older and less powerful software such as P5.js has its limitations
  • Important to coordinate the various environments to limit compatibility issues
• Integrate different subsystems of varying languages and structures together
• Task delegation, a collaborative programming workflow, and project management skills

What's next for Melodica

• Better Midi Visualization
• Midi to Music Sheet Transcription
• Metronome and Play Along Feature

Built With

• basicpitch
• bulma
• flask
• moises
• processing
• python
• three.js