agent-ready

• 

  Every run has a human-readable log: each command, its real output, and the agent's own notes. No pretending.

• 

  All sponsor integrations are load-bearing: Daytona is the clean room, Fireworks is the brain and judge, Braintrust is the ledger.

• 

  Score history caught our own bad measurement: daytona's dip (70→45→70) was our disk quota, not their docs. Every point stays public.

• 

  The live scoreboard: every SDK gets a 0–100 score, checkpoint results, and a friction report. Updates itself via nightly CI.

• 

  Four sponsor SDKs, same test, very different scores. Three real docs bugs found in one afternoon — each with transcript proof.

Inspiration

Every SDK team believes their quickstart works. Almost nobody re-tests it from a truly clean machine. And in 2026, the "developer" reading your docs is often an AI agent in a sandbox — no memory of your product, no second tab, no patience. We wanted a machine that answers one question with proof: can a fresh agent, given only your public quickstart URL, reach the promised result?

What it does

One command per SDK:

1. Starts a pristine Daytona sandbox — nothing installed
2. Drops in an autonomous agent (Fireworks-hosted Kimi K2.6) with only the quickstart URL
3. The agent works like a first-time developer: reads the docs, installs, sets up keys, runs the code
4. A judge grades the transcript against five checkpoints — a checkpoint passes only if real command output proves it
5. Every run is saved to Braintrust as a scored eval, and the results become a public scoreboard with badges, readable logs, and score history

Results — from today, live at the event

Three real docs bugs in one afternoon, each found autonomously in minutes, each with transcript evidence.

How we built it

Python + uv. A small CLI (lab run targets/<sdk>.yaml) drives the Daytona SDK for sandboxes, the Fireworks API (OpenAI-compatible) for the agent loop — tools: bash, write_file, fetch_url, finish — and the Braintrust SDK for eval logging. The scoreboard is a static site on GitHub Pages with shields.io badge endpoints. A GitHub Action re-tests every SDK nightly and supports on-demand runs (paste a quickstart URL, get a scored card). Built solo, with Claude as pair programmer.

Honesty is enforced by design: secrets are redacted before anything touches disk, deviations from the docs must be declared, runs that hit the step budget end with an honest self-written summary, and a failing tool returns an error to the agent instead of crashing the run.

Challenges we ran into

• Docs drift bit us first. The Fireworks model ID we started with had been retired — the exact bug class the lab later caught in the sponsors' own docs.
• Sandboxes run as non-root. Our first run died on a /root path assumption.
• The meta-run problem. Agents testing Daytona's SDK created sandboxes inside sandboxes, leaked them, and filled our disk quota. Two scores dipped (70→45 and 75→0). The score history caught it, the transcripts explained it, we shipped lab cleanup, and re-measurement reproduced daytona's baseline exactly: 70 → 45 → 70. We kept every point public. Single runs lie. Tracked history doesn't.

What we learned

• "Agent-ready docs" is measurable, and the bar is real: only one SDK scored 100 — the one with exact samples and an /llms.txt.
• A jumping score is a finding, not noise: braintrust's volatility comes from a hard network dependency plus an undocumented flag.
• An eval product must catch its own bad measurements. That discipline mattered more than any single score.

What's next

A model matrix (does your quickstart work for K2-class, GPT-class, and small models?), an environment matrix (which sandboxes can even reach your API?), and signed vendor-neutral reports. Nightly re-tests already ship — the board updates itself.

Built With

• braintrust
• daytona
• elevenlabs
• fireworks-ai
• github
• github-actions
• kimi-k2
• python
• shields-io
• uv