Conductor

A job-agnostic agent cluster that turns the laptops in a room into a team of AI specialists — and makes them compose, write, and produce a song together.

Inspiration

Most "AI agent" demos are a single model talking to itself in a loop. We wanted to see what happens when you treat agents like a compute cluster: cheap local models running on whatever laptops are in the room, a coordinator handing out work, and no single agent that knows the whole plan. To make it fun — and hard to fake — we needed a job with distinct parts that have to fit together. Music was perfect. A song needs a composer, a lyricist, and a performer, and if any one of them ignores the others, you can hear it.

What it does

Conductor is a job-agnostic agent cluster. Any machine on the network can join as a worker: it registers with the coordinator, pulls the next task, runs it on a local model, and posts the result. Workers never know what job they're on — the job file defines the roles and how they chain together, and the coordinator handles ordering, retries, and re-queuing work from nodes that drop off.

For the demo, the job is producing a finished track from a single shared plan:

A theory agent designs the song — genre, mood, key, BPM, chord progression, and a bar-by-bar section layout.
That plan fans out across the cluster. A lyrics agent takes the section layout plus a music-theory library our research agent compiled, and writes spoken-word lyrics that follow the structure. In parallel, an FL Studio producer agent takes the same key, BPM, and section layout and builds the instrumental entirely from code.
The master collects everything — voices the lyrics with Fish Audio text-to-speech and mixes them over the producer's instrumental, ducking the music under the voice.

Because every role works from the theory agent's plan, the parts actually line up: the vocals land on the same structure the instrumental was built around. That shared plan is the conductor.

And the cluster isn't tied to music. The same coordinator and workers first ran a code-review job — splitting a diff between two versions of a codebase across workers to hunt planted bugs like a buffer overflow. Same workers, different cargo, zero code changes.

How we built it

Cluster: a Flask coordinator exposes /register, /task, /result, and /status, with a shared token for auth. Workers are Python processes that poll for tasks and call local models through Ollama. The whole stack ships as Docker Compose, so a new machine joins with one command — and a stub-model mode lets the full loop run with no GPU at all.
Models: Qwen 2.5 7B on Ollama — it fits fully on an 8 GB laptop GPU at 8k context. Roles can run different models, so a weaker laptop keeps a 3B model for the lighter tasks.
Knowledge library: an AutoGen research agent with DuckDuckGo search compiled a markdown music-theory library — harmony, scales from several traditions, song structures, vocal prosody, and a composer's guide. The lyrics agent's prompt draws on it.
Voice: Fish Audio TTS renders the lyrics; pydub/ffmpeg mixes them over the instrumental with the music ducked under the vocal.
FL Studio producer: FL's script sandbox blocks files and sockets, so we built a bridge that streams Python into a running FL Studio over virtual MIDI (loopMIDI SysEx), with token auth and exactly-once delivery. We reverse-engineered FL's .flp project format so arrangements are generated as code — no mouse. Demucs splits stems, librosa finds the beat grid, Whisper gives word timings, and Rubber Band time-stretches the chops. The producer then tunes its own mix by measurement — frequency-band balance, crest factor, and loudness against reference tracks — before handing the instrumental to the master.

Challenges we ran into

Silent prompt truncation. Ollama's default context window was quietly cutting our 2,800-token lyrics prompt down to 9 tokens — no error, the model just improvised with no instructions. Setting the context size explicitly fixed it.
Production cues read aloud. TTS reads every word, and the lyricist kept writing lines like "Fade out — drum roll," so the narrator said them out loud. Adding a prompt rule with examples of what not to write made it worse — the examples planted those words in the small model's head. A rule without examples, plus a conservative filter before TTS, fixed it. Our first filter also ate a real lyric ("Silence cradles dreams…"), so we tightened it and turned that line into a test case.
FL Studio is not built to be automated. One wrong byte in a mixer effect makes FL hang on load. Generated projects render only the first pattern unless Song mode is set in the header. The command-line renderer only works with FL closed. Every fix is written up in the repo's LEARNINGS docs.
Fitting on laptop hardware. A 14B model didn't fit in 8 GB of VRAM, so we measured and settled on 7B over 3B.
Switching providers mid-hackathon. We started vocals on ElevenLabs and moved to Fish Audio partway through — the master's voice step is provider-agnostic, so it was a one-function swap.

Accomplishments that we're proud of

Truly job-agnostic workers: the same cluster ran code review and songwriting with no change to worker code.
The parts fit: on the 7B model, lyrics followed the theory agent's section layout in 5 of 5 test runs, with zero production cues and zero chord names read aloud.
Code-only production: finished, mixed, and mastered tracks built in FL Studio without ever touching the mouse.
A reliable control bridge into FL Studio over MIDI — the only channel the sandbox allowed.

What we learned

With small models, what you leave out of a prompt matters as much as what you put in — negative examples can backfire.
Always check what the model actually received. Our biggest bug was invisible until we counted tokens.
Measure instead of guessing. That applied to prompts (paired A/B runs), to hardware (does the model fit in VRAM), and to mixing (band balance against references).
Splitting a creative task into roles works when the handoff is structured. The theory agent's JSON section layout is what keeps every other agent on beat.

What's next for Conductor

Let repeated hooks repeat. Right now the prompt asks every repeated section for new words, which is wrong for a chorus.
More roles in parallel: dedicated drums, bass, and melody agents on different machines, merged by the master.
Scale across the room: run the cluster on more machines on the network, assigning each node a model sized to its hardware.