The Orchestra Five — Multi-Agent Swarm Orchestrator

One mission. Five autonomous AI personas. Recursive bounded swarms with real human-in-the-loop decisions.

Comment

Project Story

Inspiration

Most multi-agent frameworks today suffer from one of two extremes:

  1. Opaque and runaway: Autonomous agents spawn uncontrolled sub-agents, burn through API budgets, hallucinate in loops, and leave users powerless bystanders.
  2. Rigid and scripted: "Agents" that are simply hard-coded linear chains masquerading as intelligence.

We asked: What if you had a superhero squad where each member had a crisp, specialized role, tasks could dynamically spawn bounded sub-agent swarms when deep investigation is needed, but the human retains absolute control over key architectural decisions?

That vision led to The Orchestra Five (Orchestra Mini) — an honest, production-ready, bounded multi-agent swarm orchestrator designed for complete visibility and reliability.

What it does

You give The Orchestra Five a high-level goal in plain English (e.g., "Build a go-to-market and technical architecture for an AI-powered fitness app"). A squad of 5 distinct personas executes the mission step-by-step:

  1. Titan (Commander): Decomposes the mission into an acyclic task graph with explicit dependencies using Kahn's algorithm validation.
  2. Nova (Investigator / Scout): Gathers research and market intelligence. When faced with complex domains, Nova can dynamically spawn a concurrent swarm of bounded sub-agents to explore parallel angles and synthesize findings.
  3. Forge (Architect): Formulates two distinct technical/strategic paths (Option A vs. Option B) with trade-offs and resource implications.
  4. Vex (Critic): Rigorously audits both options, identifies vulnerabilities, and issues a formal recommendation.
  5. Human-in-the-Loop Decision: The mission pauses in waiting_for_user state. You inspect the options and the squad's recommendation, then make the binding call.
  6. Echo (Debriefer): Synthesizes the entire mission trajectory and decision into an executive audio debrief delivered via ElevenLabs or Deepgram Text-to-Speech (TTS).

How we built it

  • Backend & Orchestration Engine: Built with Python 3.11+, FastAPI, and AsyncIO. Designed around a deliberate, predictable /advance execution loop with state persistence in memory.
  • Dynamic Bounded Swarms: Nova and Forge can recursively spawn sub-agents up to configurable caps (MAX_SPAWN_CHILDREN = 7, SPAWN_MAX_DEPTH = 2) executed concurrently with asyncio.gather.
  • LLM Integration: Unified access through OpenRouter (defaulting to openai/gpt-4o-mini), allowing zero-code model swapping across top LLMs. Structured outputs enforce JSON validation with automated retry & schema error-feedback loops.
  • Multimodal Voice Pipeline: Integrated ElevenLabs & Deepgram TTS APIs with graceful audio fallbacks.
  • Frontend UI: Responsive, zero-dependency modern web UI crafted in vanilla HTML5, CSS3, and JavaScript featuring live task graph visualization, swarm inspector modals, and interactive decision triggers.
  • Cloud & Deployment: Configured for local running via Uvicorn and serverless deployment via Vercel (vercel.json + api/index.py).
  • Testing & Quality Assurance: 16 comprehensive unit & regression tests with mocked LLM/voice engines via Pytest and Pytest-Asyncio.

Challenges we ran into

  • Uncontrolled Swarm Cascades: Swarms can easily explode exponentially. We engineered hard recursion depth guards (SPAWN_MAX_DEPTH) and per-level fanout caps (MAX_SPAWN_CHILDREN).
  • Sub-Agent Fault Tolerance: In standard asyncio.gather, one flaky sub-agent request cancels all siblings. We wrapped each child sub-agent independently so failures are recorded cleanly without crashing the overall swarm synthesis.
  • Graph Integrity & Deadlocks: Language models occasionally produce invalid or cyclical dependencies. We implemented cycle detection via Kahn's Algorithm and referential integrity checks that feed validation errors directly back to the model for one-shot auto-correction.
  • LLM Output Normalization: Handling variations like "Option A", "A", or lowercase agent names required strict sanitization layers before parsing state transitions.

Accomplishments that we're proud of

  • True Bounded Recursion: Built a genuine sub-agent swarm where recursive delegation happens organically based on task complexity without unpredictable costs.
  • 100% Deterministic & Recoverable: Handled 6 critical multi-agent edge cases with dedicated regression tests.
  • Single-Model Efficiency: Proved that 5 coordinated personas on 1 LLM model outperform heavyweight, resource-heavy multi-agent frameworks.
  • Delightful Multimodal Experience: Combining graph visualization, interactive human decision gates, and spoken audio summaries into a cohesive UI.

What we learned

  • Predictability beats cleverness: A deterministic, step-by-step advance loop is significantly easier to debug, monitor, and trust than uncontrolled autonomous loops.
  • Validation at the boundary: Validating and normalizing agent output immediately prevents cascading bugs across downstream agents.
  • Human agency is essential: AI agents are at their best when they do the heavy lifting of research and option synthesis, while keeping the human firmly in the captain's seat for pivotal forks.

What's next for The Orchestra Five

  • Persistent Storage Adapter: Plug-and-play PostgreSQL / Redis storage drivers for multi-tenant and resume-anytime missions.
  • Custom Squad Persona Builder: Enable users to define custom agents, system prompts, and swarm delegation policies via a visual team editor.
  • Tool-Calling Extensions: Integrate live web search (Brave / Tavily) and code-execution sandboxes for Nova and Forge sub-agents.
  • Live WebSocket Streaming: Stream partial thoughts and agent reasoning directly to the mission board in real time.

Built With

Share this project:

Updates

Submission history