CookUp

A music creation studio where humans and AI agents compose and edit music together.

Comment

Inspiration

I’ve always been such a huge fan of music, most especially the instrumental side, and I play an instrument or two myself at an intermediate level. I'm definitely no musical expert, which is why a tool like this matters so much to beginners and intermediate music lovers like me. When you look at music at its simplest root definition, it’s really just a collection of organized sounds.

So I’ve found myself wondering countless times: if AI agents were provided with these individual sound units, could they actually make music? Not just generatively, but by actually reasoning and being creative enough to compose those single units into something genuinely pleasant to the ear? Can AI actually tap into that artistic side of us and produce something meaningful and flexible enough to iterate on easily?

This project doesn't just give me the answer to that, it shows that humans and AI can actually collaborate and compose together like real musical partners. It opens up huge opportunities for indie artists, content creators, hobbyists, or anybody at all who wants to create an instrumental on their own, real fast, without the rigidness of traditional generative AI.

TL;DR

Most AI music tools are black boxes that spit out rigid audio you can't edit. This project uses AI agents as collaborative partners to arrange individual sound units, giving beginners, hobbyists, and creators total freedom to co-create custom instrumentals fast.

What it does

CookUp is a music creation studio where you can build beats and instrumentals by arranging individual instrument notes and drum sounds on a timeline.

The main difference is that CookUp does not generate the audio itself. It gives the AI agent access to individual sounds and lets it compose those sounds into an actual editable arrangement.

You can start by telling your AI agent what you want, then keep iterating through conversation:

"Create a loopable reggae instrumental"

"add a bit more drum hits, drums are too spaced out"

"decrease the sound of the guitars and add a violin"

You can also work directly in the editor yourself. For example, you can add a new track and place your own clips, then ask the agent to continue from what you just created:

"I've added some mid and high tom clips to the toms track, replicate it all through the loop and add a melodious piano track to complement it all"

The important part is that the human and AI are working on the same composition, rather than the AI generating a separate finished audio file.

CookUp also lets you play, pause, loop, edit, save projects locally, and export finished compositions as WAV or MP3.

How we built it

CookUp is a browser-based app built around WebMCP, Tone.js, and a structured composition model.

The composition is represented as structured state containing tracks, clips, notes, timing, instruments, tempo, volume, and other editable properties. That state is the source of truth for both the visual editor and the AI tools.

Tone.js handles the actual audio playback and scheduling. For melodic instruments, we use recorded instrument samples with Tone.Sampler, which lets a smaller number of recorded notes cover the playable range by tuning samples when needed. For drums, we use individual one-shot samples with Tone.js playback.

For the instrument library, we used samples from tonejs-instruments, selecting samples from 20 different instruments.

For drums, we used the TR-808 Fischer collection, which gives us 116 individual drum sounds.

The application also includes a visual timeline editor, manual editing, playback controls, looping, track controls, undo/redo, IndexedDB persistence, and WAV/MP3/project export.

For the WebMCP integration, CookUp exposes its composition and editing capabilities as tools that the agent can call. These tools can inspect the current composition and make changes to tracks, clips, sounds, tempo, and other parts of the project.

That means the flow is roughly:

AI agent
      ↓
WebMCP tools
      ↓
CookUp composition state
      ↓
Timeline + Tone.js
      ↓
Audio

The same state can also be changed directly by the human in the editor.

Challenges we ran into

One of the biggest challenges was finding the right audio assets.

We wanted individual sounds, not loops or pre-generated music. At the same time, we wanted the sounds to be good enough that the final result would not feel like a toy, and we needed licenses that would allow us to actually use the samples in the project.

We ended up using tonejs-instruments for the melodic instruments and the TR-808 Fischer collection for drums. The instrument library gave us a good range of real recorded instruments, while the TR-808 collection gave us a large set of individual drum sounds with a much simpler structure to work with.

Another challenge was sample coverage. We didn't want to ship a separate audio file for every possible musical note, but we also didn't want aggressive pitch shifting to make the instruments sound unnatural. Using Tone.js samplers let us find a balance between the number of files we ship and the amount of tuning the browser has to do.

We also had to make the AI interaction precise enough that natural language requests actually translated into useful edits to the composition instead of vague or destructive changes.

Accomplishments that we're proud of

The biggest one is simply getting the idea to work end to end.

The AI can start with an empty composition, create a piece of music from individual sounds, make changes to it, and continue working after a human has manually edited the timeline.

That last part is probably what we're happiest with. The human doesn't have to choose between using AI and using the editor. You can make a change yourself, then hand the same composition back to the agent and keep going.

We are also proud that the music is actually editable. There is no generated audio blob that you have to accept as-is. The individual sounds, notes, clips and tracks remain part of the composition.

And we managed to do all of this in a browser using real recorded samples, without needing an AI audio-generation model behind it.

What we learned

One of the biggest things we learned is that AI doesn't need to generate the final media itself to be really useful in a creative workflow.

At first, the obvious idea is to ask an AI to generate a finished song. But once we treated music as a collection of smaller building blocks, it became much more interesting. The AI can reason about those building blocks and use them to construct something, then keep changing that same composition over time.

We also learned that WebMCP feels much more useful when the agent is operating a real application rather than just returning text.

Seeing the agent add a track, place clips, change sounds, and react to a human edit made the difference much clearer to us. It feels less like asking an AI for an answer and more like working with another person in the same creative space.

What's next for CookUp

The next step is to make CookUp more useful without turning it into a full DAW.

We'd like to expand the sound library with more carefully selected instruments and sounds, and make it easier to browse and audition them.

We also want to improve the composition side so the AI can handle more complex musical structure while still keeping every part editable.

Another natural direction is better project management and sharing, so people can move their compositions between devices and continue working on them without relying only on local browser storage or manually downloading files.

And beyond that, we'd like to explore more ways for the AI and the human to work together in the same timeline, with more precise editing and better control over arrangements.

Built With

Share this project:

Updates

Submission history