Jamspark

Hum a tune. Jam with AI. Instantly turn your hummed melodies into a custom DJ set. My AI listens to your idea and generates a unique palette of weighted instruments, ready for you to mix and jam with.

Comment

Inspiration

I found myself humming random melodies on my way to this AI Hackathon. I've been humming melodies in my head for years. As this is my first Hackathon, I had no idea what project I was going to do coming in. During the opening ceremony, I heard about Google's music generation AI, Lyria, and thought it was interesting. I attended the workshops hosted by Anthropic, Google, Letta, and Vapi, and these workshops taught me how to prompt AI for my purposes and how to vibe code.

After the last workshop I attended, I asked Claude for project ideas, rambling about the ideas I want to implement. It generated a list of four project ideas, and I chose the "AI Jam Session Companion" project as it was the most approachable. My project end up being a spin-off of Google AI Studio's Prompt DJ MIDI project, which is a successful application of Lyria.

What it does

Jamspark transforms any 5-second hummed melody into a dynamic, AI-generated backing track inside a live DJ mixer.

Here’s how to use it.

  1. You start with a clean mixer grid and click the "Inspire Me" microphone icon.
  2. The app records your voice for five seconds, displaying a progress bar so you know it's listening.
  3. This audio clip is sent to the Gemini API for analysis.
  4. The AI returns a list of five unique instrument suggestions (like "Plucky Bass" or "Dreamy Pads") tailored to the vibe of your melody. Crucially, it also assigns a "weight" to each instrument, representing its musical importance.
  5. Jamspark instantly populates the empty mixer grid with these new instruments, automatically setting each knob to the AI-determined weight.
  6. The result is a custom backing track, generated from your own idea, that you can immediately play, manipulate, and jam with.

Jamspark also retains the Prompt DJ feature of creating real-time music based on text prompt, through the "Add new instrument" feature.

How I built it

  1. I asked Claude for project ideas. Check out my brainstorming conversation with Claude.
  2. I asked Claude for an implementation plan, from which Claude decided to produce an entire project. Hence, I vibe coded. Check out my vibe-coding conversation with Claude. (It is a second conversation: see Challenges) for why).
  3. I acquired an API key from Gemini because I thought it would give me access to Lyria. After some issues with finding the model, my project at this stage ended up taking audio as input and outputting a textual summary of the audio recording, including instruments and key.
  4. I switched to v0, then switched to Google's AI Studio, then finally switched to spinning off of Prompt DJ MIDI.
  5. I repeatedly asked Gemini to make changes to Prompt DJ MIDI to implement the features I want, such as re-adding a progress bar to signal recording progress.

Challenges I ran into

  1. Claude's context window is limited. I quickly ran out of space to converse with Claude, especially as I was vibe coding. This pushed me to explore other vibe-coding AIs, such as v0.
  2. Both Claude and v0 were struggling to use the correct Google Gemini model to recognize Lyria's API. I tried to resolve this by manually defining the Gemini model to use, which ended up turning the recording into a textual description of instruments (and other musical features). I ended up spinning off Prompt DJ MIDI for its correct usage of Lyria.
  3. I forgot to save my work on my first spin-off of Prompt DJ MIDI, losing hours of work on AI Studio.
  4. Lyria complained about usage quotas once, scaring me. It was a non-issue.
  5. My AI Studio ended up talking about changing code but not actually changing it unless prompted. Sometimes I had to prompt it 3 times for actual code changes.

Accomplishments that I'm proud of

This is my first Hackathon, and I'm proud of myself for actually making the effort to sign up and show up!

My biggest aha moment during this hackathon that I'm most proud of is connecting my buggy "textual analysis" output from my (modified) Claude-generated code to textual prompts into Prompt DJ. This realization helped me connect my existing bugged project to an actual working usage of Lyria.

What I learned

I learned how to use AI tools to make applications. Specifically, I learned how to design my prompts well to get what I want, and I learned how to vibe code.

AI codes much faster than I expected.

What's next for Jamspark

One future path for Jamspark is to accurately detect BPM and key from the audio recording (which may need to end up being longer). Another area I can see it expand is to produce sheet music coming out of the AI-generated instruments in case live humans want to replicate that music.

Also right now there's only ever one instance of the project I think, so only 1 person can use it at a time successfully. In the future, Jamspark can take in a specific user's Gemini API (and not just use my own) for scalability.

Built With

Share this project:

Updates