Inspiration
After learning the capabilities of ElevenLabs API, we were inspired to expand beyond just speech and into music. We wanted to see if we could turn simple text and beats into a complete song, just for fun, creativity, and experimentation.
What it does
Our project is a web based music creator that lets users build a full song from scratch. Users can create and customize beats using the provided sounds such as kick, snare, hi-hat, and more with full control over both the drums and melody section.
The interface allows the user to adjust the songs tempo (BPM), and add more steps to sequence for more complex rhythms. Once the beat is complete, users can enter lyrics to generate an AI powered singing voice, which is then mixed with the instrumental and exported as a audio file. Additionally, users can upload a sample of their own voice to clone and use as the singing voice, to really make the song their own.
How we built it
We built the webpage using React, which made easy to create and manage the drum and melody interface. For sound playback and sequencing, we integrated Tone.js, allowing the sounds to play at precise timings. Finally, to generate the vocals, we used ElevenLabs API, which we then mixed with the beats to produce a complete song.
Challenges we ran into
One of our main challenges was getting the AI to actually sing rather than speak. Initially, the generated vocals sounded like spoken dialogue with no energy, so we had to explore and switch to ElevenLabs' music focused voice generation to achieve a more natural singing result.
Accomplishments that we're proud of
Full in-browser music production studio — real-time audio synthesis with an 8-instrument sequencer Three AI service integrations — voice cloning, TTS, and AI beat generation Intelligent beat generator — produces musically coherent patterns Voice cloning — custom AI voices from user audio samples
What we learned
Web Audio API & Tone.js: Built real-time audio synthesis and sequencing in the browser with interactive drum sequencers and melody generators State Management: Managed complex timing synchronization and audio state across React components External API Integration: Worked with ElevenLabs for voice cloning/vocal generation and Eleven Music for AI-generated beats
What's next for JamAI
Add support for more instruments and sounds Add support for uploading custom sounds Refactor the upload beat and generate beat functions so that the uploaded/generated beats show up in the create beat section to customize Add support for editing the entire beat based on user sentiment E.g. “make the beat more sad”
Log in or sign up for Devpost to join the conversation.