Inspiration

We were inspired by the original AI Duet project https://experiments.withgoogle.com/ai-duet. and we wanted to extend it to allow users who don't access to a MIDI instrument to input raw audio via file uploads or microphone recording.

What it does

It allows users to input raw audio and generate a musical continuation of it using a generative model.

How we built it

We used the Magenta libraries Onset and Frames for the audio -> midi transcription and the Transformer model on the backend to generate longer musical continuations than the original project.

Challenges we ran into

We experimented with many different approaches such as doing real time music transcription on the backend and completely changing the front end visualization. Our biggest challenge was figuring out how to connect all the different components we were working with and understanding all the intermediate representations we had to know how to handle.

Accomplishments that we're proud of

Most of us do not have any prior music background so learning how to manipulate music data was a steep learning curve for us.

What we learned

We learned learned a lot about various music file formats, digital representations of music, parameters and outputs of generative music models and front end development.

What's next for AI Duet with Audio Input

We would like to publish the app and allow people to experiment and expand on it.

Built With

  • magenta
  • onsetandframes
  • tensorflow
  • transformer
Share this project:

Updates