The inspiration was Google's magenta API and the sub-project A.I. duet. The aim was to give a machine pre-trained to produce music, image based cues extracted from the said image using feature extractors and let it generate music from it.

What it does

The algorithm takes in an image and produces a string of music from it. The instrument options are - single piano, - piano medley, - piano and drums

How I built it

I extracted high level features from an image using a CNN and naively took the top 10 most significant features to generate a midi key (note). I added the relative minor third of the major key and made a list of notes which are fed as the 'primer' to the main RNN generating the music.

Challenges I ran into

  • finding an open source audio editor to play midi files
  • handling midi playback for multiple files at the same time
  • overlaying different midi files and wrapping up the code for the whole project in time ## Accomplishments that I'm proud of ----ACCOMPLISHMENTS---- Generating 'option' based music (single piano, double piano, drums) based on cues from any given image. The algorithm is naive right now, but is flexible in its treatment of generated variables as either 'hardcoded parameters' or learned hyperparameters - as future improvements. ## What I learned --TAKE-AWAYS--
  • using google-magenta
  • training an generative RNN
  • handling midi in code and overlaying multiple files, though im still doing it by hand and not by code
  • extracting significant features from high level layers of a CNN
  • timidity and linux based midi manipulation ## What's next for audio-filters 'or' im2sound --WAY-AHEAD--
  • learn the parameters that are currently hardcoded to be truly 'AI' generated music
  • use emotion detection (MS azure) or sentiment analysis (CNN) t detect the mood of he image and pass that as a contextual cue
  • generate genre based music which is learned either from the context or user specified
  • basically do anything possible with generating music

Built With

Share this project: