MOSAIC

MOSAIC Statistics
MOSAIC Panels
MOSAIC Playground

Inspiration

The Starter Kit runs one Codex Agent per Playground turn. That works well for small tasks, but becomes limiting when a request contains multiple dependent work items, when different steps require different capabilities, or when an Agent fails and needs controlled recovery.

We realized that "many Agents talking" is not the same as "many Agents working reliably." A useful multi-Agent system must answer four practical questions:

When is collaboration actually worthwhile?
Which Agent should handle each task?
How can we verify that the output is real?
What should happen when a task fails?

We also noticed that "many agents talking" is not the same as "many agents working." Recent research published in Nature Machine Intelligence in 2026 suggests that multi-Agent collaboration does not always pay off: coordination costs and error propagation can outweigh the benefits. This motivated us to make collaboration a decision with explicit verification, rather than assuming that more Agents are always better.

The Starter Kit already provided the core platform: Agent creation, the Playground, persistent workspaces, Codex execution, model integration, and Docker-based runtime support. We wanted to build the missing middleware layer on top of that foundation, without rebuilding or bypassing the existing Agent platform.

This led us to build MOSAIC, a middleware layer for orchestrated, self-healing, and auditable Agent collaboration.

What It Does

MOSAIC coordinates multiple runnable coding Agents through a shared, structured Session.

A user can create a workflow from the browser and define:

Dependent Tasks
Required Agent capabilities
Task routing
Expected output files
Allowlisted verification commands
Optional turn-taking rules

MOSAIC validates the workflow and represents it as a dependency graph. A Task only becomes runnable when its dependencies are complete. Agents are selected using their declared capabilities and current availability.

The system separates open-ended model judgment from deterministic control-plane behavior:

Agents handle planning, implementation, and review.
Deterministic TypeScript handles state, validation, scheduling, limits, verification, and recovery.

MOSAIC does not trust an Agent's natural-language claim that work is complete. A Task succeeds only when:

The expected output file exists.
The Artifact is captured and linked to its source Attempt.
The allowlisted verification command passes.
The verification evidence is persisted.

Every Session records its execution history, including:

Session, Task, Agent, Attempt, and Run identifiers
Dependencies and Agent assignments
Verification results
Recovery decisions
Captured Artifacts
SHA-256 content hashes
Timeline events
Token and latency metrics

MOSAIC also supports self-healing. In our demo, a controlled acceptance-test failure is injected into a verification Task. MOSAIC records and classifies the failed Attempt, applies a recovery policy, creates a Repair Task, and continues execution. The repaired result is accepted only after the expected file and verification command pass.

The frontend exposes this process through:

A visual Task Graph
A Workflow Editor
A Task Inspector
A Recovery evidence panel
Artifact Preview with SHA-256 hashes
A persisted Event Timeline
Session and Agent metrics

MOSAIC also keeps coordination Runs separate from normal Playground conversations. Internal worker prompts and coordination context do not leak into the user's regular Agent transcript or Codex session.

How We Built It

We built MOSAIC as a middleware layer on top of the existing AgentService and Codex Runner.

The main components are:

A React coordination workspace for creating Sessions, selecting Agents, editing workflows, and inspecting evidence.
A Fastify coordination API for Session creation, Task scheduling, execution control, recovery, and evidence retrieval.
Typed TypeScript contracts for workflows, Tasks, dependencies, capabilities, acceptance criteria, Attempts, Artifacts, and recovery outcomes.
A Collaboration Gate that determines whether a task is better suited to a single Agent or a multi-Agent topology.
A Planner that produces a validated workflow graph.
A capability-aware Team Builder that selects suitable Agents.
A dependency-aware Scheduler that executes ready Tasks while respecting graph dependencies and execution limits.
An Agent executor adapter that connects coordination Tasks to the existing AgentService and Codex runtime.
A mechanical Verifier that checks expected files and runs safe, allowlisted commands such as npm test and node --test.
A classified Recovery Policy that maps different failure types to targeted actions such as retry, reassignment, Repair Task creation, replanning, human approval, or stopping.
A persistent data store for Sessions, Tasks, Attempts, Events, Runs, and Artifacts.
A session-scoped Artifact store with SHA-256 content hashing.

Our main demo workflow uses three Agents:

A Planner creates an implementation plan.
A Builder implements a small ESM feature and writes automated tests.
A Reviewer verifies the implementation and produces a review report.

The workflow runs through real Agents and an Ark-compatible model configuration. We also added deterministic fault-injection modes so that recovery scenarios can be reproduced consistently during development and demonstration.

The repository includes server and frontend tests, TypeScript checks, production builds, fixture-based evaluation tooling, and a real evaluation comparing Single Agent, Static Team, and MOSAIC.

Challenges We Ran Into

The first challenge was integrating multi-Agent coordination without breaking the existing Agent lifecycle and Playground. The original platform was designed around one Agent per Playground turn, while MOSAIC needed to coordinate several Agents inside one shared Session. We addressed this by creating a separate backend coordination path with its own Session, Task, Attempt, Run, Event, and Artifact model.

The second challenge was preserving useful context without mixing conversations. Downstream Tasks need evidence from upstream Tasks, but sharing one conversational thread would expose internal worker prompts in the normal Playground experience. We kept coordination Runs and Playground Runs separate, and passed structured dependency evidence between Tasks instead of sharing conversational history.

The third challenge was defining trustworthy success criteria. An Agent response can sound complete even when a file is missing or a test fails. We therefore made success depend on observable evidence: expected files, allowlisted verification commands, Artifact capture, and persisted verification results.

Recovery created another set of problems. A failed Task cannot simply be retried forever, and downstream Tasks may depend on its output. MOSAIC preserves the failed Attempt, classifies the failure, respects bounded attempt budgets, and creates a Repair Task or revised subgraph when appropriate. The original Task is marked as superseded so it is not recovered repeatedly.

We also had to handle real model failures. During evaluation, we encountered occasional model tool-call compatibility errors. Instead of hiding these failures, MOSAIC records them as failed Attempts and exposes the recovery behavior and final outcome. This made the system more honest and more useful for debugging.

Finally, we had to make the system reproducible within a short live demonstration. Controlled fault injection, deterministic verification commands, Docker startup, persisted evidence, and fixture evaluation modes helped us demonstrate both normal execution and recovery without relying entirely on unpredictable model behavior.

Accomplishments That We're Proud Of

We are proud that MOSAIC is a functional end-to-end middleware path rather than a static visualization.

A request travels from the browser to the coordination API, through real Agent runtimes, into persistent workspaces, through mechanical verification, and back to the user as inspectable evidence.

Our key accomplishments include:

Implementing a real dependency-aware multi-Agent scheduler.
Preserving the Starter Kit's Agent CRUD, lifecycle, Playground, and Codex runtime behavior.
Adding a visual Workflow Editor for explicit dependency graphs.
Adding capability-aware Agent assignment.
Adding structured WorkerOutput and acceptance criteria.
Adding mechanical verification through allowlisted commands.
Capturing real files as session-scoped Artifacts.
Recording SHA-256 Artifact provenance.
Implementing retry, reassignment, Repair Task, replan, human approval, and stop paths.
Making recovery decisions and outcomes visible in the frontend.
Preserving isolation between coordination Runs and Playground conversations.
Adding persisted Attempts, Runs, Events, and metrics.
Demonstrating a complete workflow with planning, implementation, review, testing, controlled failure, repair, and completion.
Completing a real three-strategy evaluation across Single Agent, Static Team, and MOSAIC.

The current validation results include:

107 server tests passing
30 frontend tests passing
TypeScript checks passing
Production builds passing
Real Static Team and MOSAIC workflows completing with verified Tasks and Artifacts
Recovery behavior verified against persisted Session evidence

The result is a system where users can inspect not only whether a workflow completed, but also which Agent performed each step, what evidence was produced, what failed, how recovery was selected, and why the final output was accepted.

What We Learned

We learned that reliable multi-Agent systems require more than sending prompts to several models. The most important engineering work is in the boundaries around the models.

Explicit dependencies make execution understandable. Typed contracts make the system easier to validate and extend. Mechanical verification prevents model-generated claims from becoming the only source of truth. Artifact provenance makes outputs auditable. Failure classification and bounded recovery make reliability behavior explicit instead of accidental.

We also learned that recovery should be visible. Hiding a failed Attempt may produce a cleaner demo, but it removes information users need for debugging, trust, and accountability. A better system preserves the failure, records the recovery decision, and clearly shows the repaired outcome.

Another important lesson was that middleware should be added at the correct architectural boundary. MOSAIC did not need to rebuild the model runtime, Playground, or container system. It added value by coordinating those existing capabilities through a dedicated control-plane layer.

We also learned to distinguish a successful demonstration from a statistically proven performance claim. Our evaluation tooling reports real metrics and failure states, but one run per strategy is not enough to establish general superiority. Evaluation results should include unsuccessful runs and limitations rather than presenting only favorable outcomes.

What's Next for MOSAIC

The next step is to make MOSAIC suitable for larger, longer-running, and more sensitive workflows.

Potential improvements include:

Repeating evaluations across more tasks, models, and runtime configurations.
Adding a richer trace view for model calls, tool calls, sandbox execution, and policy decisions.
Replacing the JSON store with a transactional database for concurrent Sessions.
Adding stronger Agent identity, ownership, permissions, and credential revocation.
Supporting more model and runtime providers through the existing executor contract.
Adding cost-aware scheduling and token budgets per Task or Session.
Improving Artifact retention, cleanup, export, and conflict handling.
Supporting workflow templates and reusable Task contracts.
Adding more recovery strategies based on historical failure patterns and Agent reliability.
Extending human approval to sensitive file, network, or external-service operations.
Improving accessibility and keyboard navigation in the Workflow Editor and evidence views.

Our long-term goal is to make multi-Agent execution as understandable and controllable as other engineering infrastructure: explicit contracts, observable execution, mechanical verification, bounded recovery, and trustworthy evidence.