VoiceSynth

Inspiration

Learning modern music software is tough, getting the sounds you want is even harder. When using FL Studio synthesizers, all the knobs and sliders look random but are actually able to create most sounds you imagine.

What it does

A fully voice controlled groovebox, with 4 instrument tracks: 1 Drum and 3 Synthesizers. You can use the voice command "Groove" and followed by the command/thing you wanna do. You can really ask groove to do anything you want, but everything is controlled by the voice and you need to sing the notes you want to hear. Groove can help you clean everything up and make it sound like how you imagine, but it won't create something you haven't directed it to.

How we built it

We created a web app inspired from a groovebox called SLOOP. We have a LangChain Deep Agent that takes in voice input and controls the application for you. The internal model is a Gemini model. You can describe sounds you want and we use an algorithm from research papers we found that automatically tunes the synthesizers 79 parameters to match the description. You can say when you want to record, and sing the tunes into the program with the instrument of your choosing, or you can create drums also using your voice.

Challenges we ran into

The hardest challenge we ran into was translating the singing voice or beats created by human voices to good sounding music. We iterated a ton of times with different ways of calibrating delays, background audio and overall sound generation.

Accomplishments that we're proud of

How much fun we had while making the project. Once the project could start generating any sort of music, we started having so much fun each time we tested, generating something stupid or trying to make sounds and beats that popped into our head.

What we learned

We learned a lot about audio to text and deep agents. We spent time researching papers and current software while talking to people from Langchain, Costar, and Chatoverflow on their opinions on how we should tackle our project. We learned that specification is key, collecting existing resources and opinions helped us direct our project and make less mistakes then if we went in blind.

What's next for VoiceSynth

Our goals would be to expand the amount of tracks, improve the groove deep agent's reliability and tools to allow for more creative inputs and overall increase the instrument flexibility past synths or the synth generation to allow for any type of instrument to be generated more reliably.

Most importantly, we can't wait to keep using this project ourselves!