ForkLab

ForkLab forks one distributed-agent failure into eight isolated Daytona worlds, then uses Braintrust to prove which interventions stop retry cascades—turning outages into causal evidence.

Comment

Inspiration

We started with an idea borrowed from immune systems: when one component recognizes danger, what should the rest of the system inherit?

Modern agent fleets usually fail in the opposite way. An agent encounters an error, retries locally, and tells nobody. Its peers encounter the same condition at nearly the same moment, make the same decision, and collectively turn a recoverable fault into a fleet-wide retry storm. Dashboards can tell us that the outage happened. Logs can suggest a story. Neither can prove which intervention would have prevented it.

We wanted to build the missing instrument: a laboratory that takes one failure, forks it into controlled worlds, and produces causal evidence while the system is still debuggable.

What it does

ForkLab is an active causal experimentation system for distributed AI infrastructure.

An operator pre-registers a hypothesis, seed, workload, and hard compute budget. ForkLab first rehearses the experiment locally, then performs a no-compute Daytona capacity inspection. Only after that inspection passes can a human authorize the immutable request.

The approved workload executes in eight isolated Daytona worlds created from the same snapshot:

  1. baseline
  2. retry jitter
  3. failure-state broadcast
  4. increased retry-queue memory
  5. increased latency
  6. broadcast plus jitter
  7. forced restart
  8. randomized placebo

Every world receives the same exogenous shocks through counter-based randomness, so the 160 episode outcomes remain pairwise comparable even when control flow diverges. ForkLab computes paired treatment effects, ranks evidence per unit of compute, and sends only the collected result—not the decision itself—to Fireworks AI for a bounded technical interpretation.

Braintrust acts as the experiment ledger. It stores the root experiment trace, one child span per counterfactual world, execution metadata, causal validity scores, placebo fidelity, positive-control direction, infrastructure fit, and deterministic regression evals.

In our live run, the untreated fleet failed in 22.5% of episodes. Failure-state broadcast reduced that to 0%. The randomized placebo reproduced the baseline at 22.5%, while the deliberately harmful latency control increased failure to 40.625%. All eight worlds completed, all workloads verified, and Daytona cleanup returned the fleet to zero sandboxes.

How we built it

The operator console is a Next.js and React application written in strict TypeScript. The experiment engine has two implementations: a TypeScript local twin for free rehearsal and a dependency-free Python workload uploaded into Daytona sandboxes. Parity tests prove that both implementations produce the same ordered episode outcomes bit for bit.

Daytona supplies the isolation boundary. ForkLab resolves a region-compatible immutable snapshot, verifies snapshot disk × peak worlds against the organization quota, and creates one short-lived counterfactual world at a time. Outbound networking is disabled. Each world is bounded by command timeout and TTL, then explicitly deleted and reconciled against the account fleet.

Braintrust is integrated at the experiment-control layer rather than added as a logging afterthought. A root span owns the hypothesis and budget; eight child spans own the world evidence. Deterministic scorers check completion, workload identity, common-snapshot provenance, common random numbers, placebo fidelity, positive-control direction, intervention effect, and infrastructure fit. Three Braintrust eval suites protect the acquisition, causal, and Daytona-capacity policies from regression.

Fireworks AI receives a typed, bounded evidence packet after execution and turns it into a concise technical readout. It is not allowed to invent missing worlds, override the statistics, or authorize additional compute.

The paid execution boundary is loopback-only and protected by same-origin JSON, a one-time 256-bit approval token, request-bound idempotency, an immutable snapshot fingerprint, an execution lock, and account-wide fleet preflight.

Challenges we ran into

Our hardest problem was not the simulator—it was making the experiment honest under real infrastructure constraints.

The first Daytona design used a Windows seed sandbox and experimental forks. That seed consumed the entire 30 GiB organization disk quota, so even sequential forks could not start. A smaller Linux VM profile appeared promising, but no compatible VM runner existed in the account's only region. We also encountered a stale snapshot that was unavailable in us.

Instead of hiding those failures, we promoted them into product requirements. ForkLab now discovers the actual snapshot region and resource class, rejects wrong-region or quota-infeasible plans before paid compute, and reports the achievable provenance honestly as common_snapshot rather than pretending the worlds share a live ancestor.

We also initially routed execution through a conversational tool-call layer. A provider failure could interrupt the tool lifecycle and leave the UI with a missing result event—the worst possible failure mode for an experiment console. Under hackathon time pressure we removed that abstraction and built a direct, typed operator path with explicit rehearsal, inspection, approval, execution, and evidence states.

Finally, ordinary seeded randomness was not enough. Different interventions change control flow and therefore consume random numbers in different orders. We replaced it with counter-based random coordinates so every world sees the same shocks independent of execution path.

Accomplishments that we're proud of

  • Executed eight real Daytona counterfactual worlds from one immutable snapshot.
  • Reduced measured fleet failure from 22.5% to 0% under failure broadcast.
  • Reproduced baseline exactly with a randomized placebo.
  • Made a positive control fail in the expected direction.
  • Captured one Braintrust root trace plus eight completed, workload-verified child spans with all causal and infrastructure guardrails passing.
  • Built deterministic causal, acquisition, and capacity eval suites in Braintrust.
  • Kept the TypeScript and Python experiment engines bit-for-bit equivalent.
  • Added 43 tests covering statistics, parity, capacity, security, idempotency, partial failure, cancellation, and cleanup.
  • Reconciled the Daytona fleet back to zero sandboxes after execution.
  • Turned multiple real provider failures into falsifiable preflight checks instead of demo-day caveats.

What we learned

Observability becomes much more powerful when it can evaluate an experiment instead of merely describing an incident. Braintrust gave us a durable causal record: the hypothesis, compute decision, world-level evidence, and validity judgment live in one trace hierarchy.

We learned that infrastructure feasibility is part of experimental validity. Region, disk quota, snapshot class, cleanup, and provenance are not deployment details when they determine whether counterfactual worlds are comparable.

We also learned that human approval works best when it is cryptographically bound to a specific plan. “Are you sure?” is theater; a short-lived token tied to the exact hypothesis, seed, snapshot fingerprint, and budget is a real control boundary.

Most importantly, AI is useful after measurement, not in place of it. Fireworks AI helps communicate the evidence, while deterministic code decides whether the evidence is valid.

What's next for ForkLab

ForkLab's retry cascade is the first experiment, not the final product. Next we want to turn it into a general counterfactual debugger for agent fleets:

  • inject model-provider brownouts, malformed tool responses, stale memory, schema drift, poisoned retrieval, and partial data loss
  • let an acquisition policy choose the next most informative world per unit of compute
  • add durable execution so experiments survive operator disconnects
  • support larger parallel fleets and true common-ancestor VM forks where the regional quota permits them
  • compare models, prompts, tools, and recovery policies under identical shocks
  • create a reviewed library of failure signatures that other agent fleets can subscribe to before they repeat the same incident

The long-term vision is an immune system for distributed software: when one agent fails, the fleet does not merely hear about it—it learns which response actually works.

Built with

  • Daytona
  • Braintrust
  • Fireworks AI
  • Next.js
  • React
  • TypeScript
  • Python
  • Vercel AI SDK
  • Zod
  • Vitest
  • GitHub Actions

Try it out

Run locally:

npm install
cp .env.example .env.local
npm run doctor
npm run dev

Then open http://127.0.0.1:3000, run the free local rehearsal, inspect the Daytona plan without spending compute, and authorize the immutable request.

Built With

  • braintrust
  • causal-inference
  • fireworks-ai
  • github-actions
  • multi-agent-systems
  • next-js
  • next.js
  • python
  • react
  • typescript
  • vercel-ai-sdk
  • vitest
  • zod
Share this project:

Updates