Mistral guitar tabs generator

WeApp screenshot
Team Logo

Inspiration

Today, more and more LLMs are being used in creative activities. We were inspired to experiment with generating music using a large language model, but we found that the results were not impressive without fine-tuning the model. Despite our limited knowledge of music theory, we are familiar with guitar tablature, which can be represented in ASCII format. This allows us to input it into an LLM effectively.

What is a guitar TAB

What it does

Our chatbot takes four measures of guitar tablature , the desired style and key as input. It then generates the following four measures and produces a MIDI file, enabling playback of the tablature in computer music software.

How we built it

We used an open-source dataset with guitar recordings to ensure the use of royalty-free music. This dataset includes the guitar frets played over time. We preprocessed the dataset to create a collection of ASCII tabs, each labeled with its style (rock, funk, etc.) and key.

We fine-tuned the Mistral-7B-Instruct-v0.3 model using QLoRa (Quantized Low-Rank Adaptation) for music note generation. The model was trained to take the first four measures of a piece as input and generate the next four measures. This fine-tuning process enabled the model to learn the patterns and structures of music in a given key, allowing it to produce coherent continuations based on the input measures.

To ensure the model generates exactly the next four measures each time, we enclosed the guitar tab input with [start] and [end] tokens. This helps the model better understand the input structure and produce outputs that are easier to compute. We used the prompt like this example to provide clear instructions to the model:

<s>[INST] Give me the guitar four measures following this one in the key Eb and in the style Funk. Stop generating after four measures. Here is my guitar measures :

[startt]e|--------------|--------------|--------------|-----9--9-----|
B|-----6--------|-----6--------|--------6-----|-----7--7-----|
G|-----7--------|-----7--------|--------7-----|--------------|
D|-----6--------|-----6--------|--------6-----|-----8--------|
A|--------------|--------------|--------------|-----6--------|
E|-----6--------|-----6--------|-----6--6-----|--------------|
[endt] [/INST]

Challenges we ran into

The main challenge was preprocessing the data. Initially, the output measures were empty because our dataset contained many instances of empty music.

We also needed to add special tokens to indicate where the tabs start and end. This was challenging because a tab is represented on six lines. Consequently, the first line generated by the LLM continues from the first line in the input tab, the second line continues from the second line, and so on. Additionally, ensuring the LLM knew when to stop generating was a significant challenge, which we addressed by using our custom tokens.

Accomplishments that we're proud of

We are proud that we built a complete application and our application can generate guitar tabs based on the tabs and indicated key, style in the prompt.

What we learned

We learned how to build a complete project utilizing all the tools we wanted and those offered by the sponsors, creating a beautiful interface and providing a comprehensive description. Additionally, we gained experience in fine-tuning a large LLM (7 billion parameters) for the first time.

What's next for Mistral guitar tabs generator

By using a larger dataset, we could achieve better results and create a more comprehensive partition generation model.

Additionally, we can adapt this project to generate tablature for any stringed instrument with frets.

Our graphical charter

We have established our own visual identity guidelines, which include our logo and color scheme. For this, we took inspiration from the Mistral design. Link for our graphical charter

Citations

Guitar Set : Q. Xi, R. Bittner, J. Pauwels, X. Ye, and J. P. Bello, "Guitarset: A Dataset for Guitar Transcription", in 19th International Society for Music Information Retrieval Conference, Paris, France, Sept. 2018.