Musimage

Without the use of any machine learning, this project parses through an image using math and music theory to create a new piece of music for every image. Come and listen to your favorite images!

Comment

Inspiration

We had previously been able to change images into audio, but it was often hard to listen to and not very musical. This time, we wanted to see if we could generate pleasant sounding music using the Fourier transform and 2D derivative of an image.

What it does

Our project takes in an image from the client and plays a song by scanning through the x and y axes of the derivative of the image. Each pixel is associated with a frequency from the pentatonic scale and its loudness is determined by its brightness. This allows listeners to "hear" where the image is changing, and how bright it is. If the image is periodic, so will the sounds. The frequency is then transformed into samples using a custom waveform derivation based on the Fourier transform.

How we built it

We used Python and JupyterLab for analysis and testing, and Flask to build a web app. We also used Purdue's computing resources in the form of Anvil.

Challenges we ran into

We ran into many difficult challenges as this is a very mathematically intensive and yet creative work. As an example, one of our challenges was to reconstruct and verify that the Fourier transformed signal can be deconstructed back with high fidelity.

Accomplishments that we're proud of

  1. Our music actually sounds so good
  2. It doesn't rely on any ML or AI.
  3. All of it is done ourselves, including raw sample data.
  4. Features of the image are actually evident in the music!!!!

What we learned

We learned the complex math behind Fourier transforms along with a lot of music theory.

What's next for Musimage.

We hope this could find some application beyond just entertainment as a novel way to communicate a sense of vision through hearing.

Built With

Share this project:

Updates