Personal Pianist

User Interface
Interactive Keyboard
Generated sheet music (PDF)

Inspiration

We were inspired by creative learning methods as well as the rising impact of artificial intelligence. We combined these two concepts to build a new and innovative way to experience, learn, and play music.

What it does

Our project is a web application that allows users to upload an audio file and select an instrument from a dropdown menu, transforming the audio into a version played by the chosen instrument. Users can download the new audio file and a PDF of the corresponding sheet music directly from the website. For piano, an interactive keyboard shows which keys are played alongside the music. Our project helps new musicians by generating playable versions of the songs they love while providing interactive demonstrations to guide them in learning their instrument. For experienced musicians, our application offers a truly enriching musical experience, allowing them to learn and play any song imaginable.

How we built it

We used AI and sound filtering to extract the notes and their durations from audio files. We then used these notes to generate a new audio file (MIDI) and a pdf of the sheet music. We utilized React to configure the user interface, including the interactive keyboard and Django to manage file transfers.

Challenges we ran into

Note detection was a big problem, as even state-of-the-art pitch detection algorithms were extremely inaccurate due to the inherent randomness in sound waves. To mitigate this problem, we used a machine learning model to separate the pitches of the melody in the audio from any background sound. We then applied further processing by using a key detection algorithm to filter out notes with a high probability of being incorrect, leading to our final audio transcription.

Accomplishments that we're proud of

We are proud of creating a product that has the potential to truly help people learn and understand music in a new way. Putting all of our components together, from audio translation to sheet music generation to displaying the user interface took a lot of work, so we are really proud of that.

What we learned

We learned about the challenges of connecting frontend and backend code. We also learned to think creatively about how to cohesively utilize different libraries to achieve our goals.

What's next for Personal Pianist

We want to incorporate more accurate audio translations for a more robust range of sounds. Right now, our app is best suited for the keys of C major and A minor, but we plan on adding more diverse pitch correction techniques to increase its accuracy over a larger range of music.