I came up with the idea for this app when I got into a conversation with my violin teacher about harmonizers in the music industry. Commercial harmonizers, I learned are large pieces of hardware that listens to a vocal line or a solo instrument and harmonize in real time, typically at a fixed musical interval below the soloist. I was disappointed to learn that they were very expensive - not to mention the non-portability, and the worst of all in my opinion, the lack of harmonic complexity that can by achieved by harmonizing at a fixed interval relative to the soloist. We challenged ourselves to create a portable and free harmonizer that leverages a database of chord progressions from tens of thousands of songs to generate a complex chord progression that fits soloist's line.

What it does

The main difference between Isolde and a conventional harmonizer is that Isolde doesn't process audio in real time. However, without this restriction it's able to utilize the full context of not only the preceding and future notes, but also the harmonic progression. By knowing ahead of time the context and length of the solo, Isolde can guarantee it will resolve at the right time and in a satisfying way.

How I built it

The main tasks involved in building Isolde were parsing the tones accurately and determining which chords groups of tones could belong to. The tones are parsed using the WadJS library and the tones under 100ms in duration are discarded (these short undesired 'ghost notes' occur frequently at the start of a tone). The raw audio is also recorded and saved as a glob for downloading later. In terms of the chord synthesis, one issue with commercial harmonizers is that they have difficulty telling the difference between notes in a chord and decorations or suspensions. In order to get around this issue, and to account for the occasional ghost note or overtone picked up by our parser we gave each bar of notes a score between 0 and 1 for how well it 'fits' a certain chord. This score is influenced by whether each note is in that chord's triad, and is weighted by the note's duration in the bar. The hooktheory API, on the other hand, can yield the most common continuations given a sequence of chords by consulting its database of chord progressions from over 25 000 songs. We compare the most common continuations suggested by the hooktheory API and choose the chord that fits both these criteria (slightly prioritizing the consonance over the popularity).

Challenges I ran into

The first main challenge we encountered was getting the code to actually compile. We had envisioned coding a blazing fast WebAssembly application that would be compiled from C++ using Emscriptem. I'd chosen an audio processing library and had begun to write some tone detection code when I realized my code was compiling with gcc but not to wasm with emcc. After some screwing around for a while trying to manually edit the library's source to get it to compile, we decided to change languages last minute and use JavaScript. Luckily, I was able to find an analogous library and we made everything work out. In hindsight, the main bottleneck of our program is still the API calls so the performance improvements made by using WebAssembly would've been marginal at best.

Accomplishments that I'm proud of

I'm proud of getting a good amount of sleep yesterday.

What I learned

Looking back to the challenges I faced, I created an awful lot of trouble for myself by prioritizing efficiency over pragmatism, when in reality efficiency wasn't an issue in the first place. I need to remind myself that a perfect application won't spring out of my efforts fully formed, and the actual functionality of a project is more important than the efficiency of the code. In the words of Donald Knuth, "premature optimization is the root of all evil".

What's next for Isolde

I envision adding options for more diverse types of generated accompaniment; perhaps making some recordings of piano, violin and guitar tracks of all the supported chords that can be modulated and sped up / slowed down to fit with a track. In addition, I'd like to add an option to harmonize the music in different styles such as jazz or atonal.

Built With

Share this project: