Inspiration
We were tired of the barrier of entry into music. To create amazing songs, you need to be an amazing musician, and the amount of effort it takes to learn an instrument is a reason why students rarely continue into music education.
What it does
HumStudio is an AI-powered digital audio workstation (DAW), purpose-built to uplift new or inexperienced musicians. With our signature "Hum" feature, users can hum a tune that's been stuck in their mind and have it immediately transcribed into music that the computer can play. They can also ask agentic "Theory Agent" to explain the music theory behind chord progressions, and suggest new places to take their song.
How we built it
This app is built with Next.js, Tone.js, Typescript, the Gemini API, and a custom note-recognition algorithm inspired by YIN and Spotify's Basic Pitch.
The note-recognition algorithm takes in an audio stream from the microphone and first feeds it into Pitchy, a Javascript pitch detection library. We measure the loudness of the audio using RMS to determine the exact note length and articulation (is the note a quarter note or two eighth notes?) We also implement note segmentation, translating raw frequencies hummed by the user into the discrete notes that you find on a piano. In other words, we "round" the frequencies to the nearest note.
We then apply quantization to place every note on the beat, and generate an accompanying chord progression.
The AI agent was made using the Gemini API. We create a representation of the entire project, feed it to the LLM, and instruct it to provide the user with explanations of the theory behind it.
Challenges we ran into
Dealing with noise during humming. To generate the correct sequence of notes, there must be very little noise in the room or in the recording. To counteract this, we implemented a "sensitivity" feature, where the user can specify how fast the notes change in their recording. Lower sensitivity gives more accurate results, but cannot transcribe faster rhythms. High sensitivity can gauge a wide range of note lengths, but it also picks up on background noise.
Building a working interface. This was unexpectedly one of the hardest parts of the project. A prerequisite for hum detection and agents was a robust data model for the song. With such a model, adding and deleting notes would be trivial. But in early prototypes, the way we represented notes was very simple and couldn't support the demands of complicated rhythms. We opted to completely rewrite this section of the code.
Accomplishments that we're proud of
- Working hum detection. Users can sing into their microphone and generate an accurate transcription in a matter of seconds. This is a major improvement over the hours it took to encode complicated melodies by hand in a DAW.
What we learned
I think all of us learned a lot about working on a team and building with agentic systems. Concepts we learned in the CS classroom (algorithms, object-oriented programming) truly came to life in this project.
What's next for HumStudio
We want to add the ability to create your own digital instruments using Theory Agent. We also want to add interactive lessons and quizzes to help build the student's musical competency. And we also plan to get rid of the "sensitivity" feature, which is more of a compromise than a feature. Better signal processing and more intelligent segmentation would help us capture the user's original idea more accurately.
Log in or sign up for Devpost to join the conversation.