Forkestra

A band of AI agents, each playing its own instrument in a Daytona sandbox. They listen to each other, edit ElevenLabs stems with real audio tools, and deliver a track verified against your exact spec.

Comment

https://www.loom.com/share/933ee4ebbb604c82a8750e1d7160f4d7

Inspiration

Game developers, video editors and content creators constantly need music that fits exact constraints: a 32-second loop at 90 BPM for a stealth level, repeating under gameplay without a click. AI music generators get close, but not exact, so someone still ends up hand-editing in audio software. We wanted to see if a team of AI agents could do that editing the way a real band works: each musician owning a part, listening to the others, and a conductor holding everyone to the brief.

What it does

Forkestra is a band of AI agents that makes music to a spec.

  • You give it a brief, like "Stealth level, rainy cyberpunk alley at night. Tense but quiet. Must loop seamlessly under gameplay: 32 seconds, 90 BPM."
  • ElevenLabs Music generates separate stems for drums, bass and keys.
  • Each stem goes to its own player agent, working in its own Daytona sandbox. Players use audio tools we built to analyze, cut, stretch and reshape their part, and they read each other's analysis so their parts fit together.
  • A conductor agent mixes the parts, measures the result against the spec, and sends a player back for a retake when a check fails. It only marks a track ready when the measurements pass.
  • You can direct the band in plain language. Type "drums: half-time" and the drummer halves its hits, then the bassist thins its line to beats 1 and 3 to follow the kick, and the keys player switches to longer held chords to leave space. The other players aren't told what to do; they react to the drummer.
  • A live UI shows each agent's reasoning, every tool call, its sandbox, and the spec checks as they pass or fail.

How we built it

  • Agents: Claude Sonnet on AWS Bedrock for the three players, Claude Haiku for the conductor. Each player runs a tool-use loop with a step limit.
  • Studio toolkit (runs inside the sandboxes): analyze (tempo, key, onsets, loudness), slice_at_beats, time_stretch, thin_to_beats, hold_chords, layer and check_spec, written with numpy and scipy.
  • Daytona: one isolated sandbox per player, running in parallel. All agent-driven audio processing happens there. API keys never enter the sandboxes, every command has a timeout, and sandboxes are deleted after each take, even when something fails.
  • ElevenLabs Music API: generates the raw stems from descriptive prompts, cached so we never regenerate the same audio.
  • SurrealDB Agent Memory: the shared band chart: spec, plan, each player's part, human directions and conductor feedback. Facts supersede rather than overwrite, so the history shows what changed, who changed it (a player, the conductor, or a human), and when.
  • Arize AX: one trace per take, with each player as an agent span and every turn, tool call and Bedrock call nested inside, so you can see exactly why each agent did what it did.
  • UI: a local web app that streams the band's conversation live, with a "Direct the band" box and before/after audio players.

How we verify the spec

The conductor trusts measurements, not the agents' own summaries:

  • Length: cut to whole bars. At 90 BPM, 12 bars is exactly 32 seconds.
  • Tempo: checked against the target within about ±4%, by testing the target tempo against competing tempos rather than guessing blind.
  • Loop seam: the jump in level where the loop wraps around is compared against the normal jumps at bar lines inside the track, plus a check that the edges aren't faded.

Challenges we ran into

  • Tempo detection on short lo-fi clips was unreliable. Rain and vinyl crackle dominated the onset signal, and a 90 BPM loop kept reading as 137. We filtered onsets to the musical band below 4 kHz and switched to checking the target tempo against competing tempos (3/4, 4/3, ±10%) instead of detecting it blind.
  • Time-stretching broke stereo. Our first stretch drifted the left and right channels out of phase so they partly cancelled. Processing in mid/side fixed it.
  • "Adapting" at first meant just turning the volume down. We added thin_to_beats and hold_chords so players respond musically (rhythm and density), not only with gain.
  • Directions can conflict with the spec. Asking for "fast paced" made players double their speed, which broke the 90 BPM loop, and the conductor correctly refused to sign off. "Upbeat" at a fixed tempo has to mean a busier rhythm, not faster playback.
  • Sandbox forking wasn't available on our account (it needs VM snapshots), so each player gets its own sandbox instead, running in parallel.
  • Tracing clutter: the Daytona SDK's built-in spans showed up in Arize as loose top-level traces. We nested everything under one trace per take.
  • Speed: a take takes about 2.5 minutes, mostly model calls and sandbox setup.

What we learned

  • A multi-agent system feels intelligent when agents react to each other, and that only shows if you make the reasoning visible. Without the UI and the traces, it looked like a plain generator.
  • Measured checks beat model self-assessment. A conductor that can say "no" is what makes the output trustworthy.
  • Sandboxes make it safe to let agents act on real files, and readable sandbox names and reliable cleanup matter as soon as you run several in parallel.

What's next

  • Timed cues like "drums enter at bar 2" or "build to a drop at 0:12", snapped to the beat grid and verified by measurement.
  • Letting a player request a new stem from ElevenLabs when its editing tools can't reach a direction (today, players reshape the generated material rather than creating new sounds).
  • Faster takes with pre-built sandbox images and sandbox reuse, plus sandbox forking so variations branch from shared state.
  • Export loops with metadata for game engines and video editors.

Built With

Share this project:

Updates

Submission history