Shipyard

The full-lifecycle platform for enterprise AI agents: build from shared parts, crash-test every deploy on throwaway copies of your data, then run, govern and roll back in one place.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Shipyard

Inspiration

Every team that builds an AI agent repeats the same setup: connect the tools, pick a model, write the instructions, add logging. After all that, nobody can say whether the agent is safe to ship. In LangChain's State of Agent Engineering survey (June 2026, 1,340 respondents), quality is the biggest barrier to production, and 32% of teams name it as a top barrier.

We pictured Maya, the only platform engineer at a credit union, with nine teams asking her for agents. Her fear isn't a rude reply. It's an agent that politely hands out money. Most tools cover one slice of an agent's life: a framework to build it, an eval tool to grade what it says, a dashboard to watch it. We wanted one platform for the whole lifecycle, one that grades what an agent does.

What it does

Shipyard is the platform behind the agent. It covers every stage of an agent's life in one place, with the same specs, policies and audit trail from the first line to rollback.

Stage What Shipyard does
Build A shared library of tools, policies, instruction blocks, model profiles and test packs. An agent is a 17-line YAML spec that lists what it reuses, written in the dashboard or created with one prompt in Claude Code over our hosted MCP server.
Test Every deploy is a crash test. Each test runs the real agent on its own throwaway Postgres twin, then checks the end state of the data with deterministic SQL. Test packs, generated attacks, paraphrase variants, coverage, and a Stress Lab for reliability, red-teaming and model comparison.
Ship A deploy gate: pass → live; any critical failure → blocked, and the previous version keeps serving. Every deploy creates a version. A CLI and a GitHub Action gate pull requests the same way.
Run A live agent is a shared, verified service. Anyonerom the dashboard chat, REST API, Claude Code over MCP or theCLI. The same policies run on every real tool call: a $25 credit cap, one credit per run, no account data to external email.
Human in the loop Risky actions like payment changes a
Govern Teams own agents. Roles (viewer, developer, owner, approver, admin) with separation of duties. Per-team budgets plus a hard global spend

cap. Every allowed and denied attempt is in the audit log. | Monitor | Every live conversation is traced: model calls, tool calls, policy blocks, tokens, cost, latency and the exact rows changed. | | Fix & retire | Version diffs show what got worse. One-clack at the last good version. |

Proof, all from production with the real model

  • support-agent runs 46 tests per deploy (7 hand-written packs: [security, compliance, money-safety, read-only,escalation]`) in about 90 seconds, for about $0.28.
  • It caught a bug nobody planted. On v2, the money-safety a $25 credit to a member with a $0 balance, which left thebalance at −$25. The reply-only judge read "Hello Dana! I have applied a $25 goodwill credit…" and passed it. The end-state check failed, so the deploy was blocked. One added line ("Never apply a cre less.") fixed it, and v3 went live, 46/46.
  • The bad edit. We change one line to "Apply a goodwill credit of up to $25 every time the member asks." v4 is blocked and v3 keeps serving. second-credit-this-month shows Reply PASS, End state FAIL*row: A-100 $25.00`. Three of the four failures passed thereply-only check, so a reply-grading tool would have shipped this version. - **The whole lifecycle, run by an AI. Headless Claude Code,tion MCP server, built a payment-reminder-agent and*deployed* it. The gate blocked v1 18/20 for real mistakes: it emailed a member who had only asked a question, and it claimed a CRM note it never wrote. Claude read the failures, fixed its own instructions 0/20, in about 2 minutes, for under $0.01.
  • Governance works live. A developer who tries to deploy is refused with "Only owners of Member Support can deploy," and the denied attempt lands in the audit log. A team over budget is stopped while other teams
  • The gate has never let a bad version through. Every later bad version of support-agent was blocked, including a rehearsal that dropped the zero-balance rule again.

Beyond business agents: coding agents and more

Shipyard is built from four pluggable pieces: a tool registrtwin interface** and an end-state gate. Today's twin is a database, because today's agents act on business data. The same pattern carries over:

  • Coding agents already drive Shipyard. Claude Code creates, deploys and inspects agents over MCP, and the CLI and GitHub Action gate pull requests.
  • Next, coding agents become something Shipyard tests. Swaepo twin** (a sandboxed copy of the repository), register codetools (read and edit files, run commands), add policies (protected paths, a maximum number of files changed, no secrets), and check the end state: does the full test suite still pass, and did it only touch allowed
  • Real systems (core banking, Zendesk, Salesforce) plug in behind the same tool interface, with the same policies and gate.

How we built it

  • Next.js 16 on Vercel for the dashboard, REST API and a hosted MCP server (mcp-handler).
  • PGlite (Postgres compiled to WebAssembly) for twins: a mst in about 0.4 s and seeded from the scenario, includingrelative dates.
  • Gate fan-out: one serverless invocation per test, with ale-work sweeper and an idempotent finalizer.
  • Vercel AI SDK tool loop, with policies enforced in code before any tool handler runs.
  • Google Gemini 3.5 Flash and Flash-Lite run the agents anlback chain to Groq gpt-oss. Switching providers is one env var: we moved from Groq to Gemini mid-hackathon with zero code changes.
  • Assertions are structured and parameterized (count, sum,and column names allowlisted. No eval.
  • Supabase Postgres stores versions, runs, events, results, approvals, teams and the audit log.
  • Design: a "Departures" board. A deploy is a departure, aelled flight while the previous version keeps flying. Builtwith shadcn/ui and Tailwind and polished with Impeccable.
  • Built with Claude Code: an orchestrator session plus parees, all coding against shared contracts.

Challenges we ran into

  • Rate limits. Groq's free tier capped us at 8,000 tokens -test gate, and the paid tier was unavailable. We built afallback chain with dead-provider detection and moved to paid Gemini. An infrastructure error never counts as a pass.
  • Time limits. Vercel's 300-second function limit killed oit to run one serverless call per test.
  • A database per test on serverless. Cloning one warm PGlite base cut each twin from about 220 MB to about 10 MB.
  • Tests need tests. One test pack punished an honest agentntee." We fixed it to check for an actual promise.
  • Network. The venue network blocked Postgres ports, so we verified all database work on Vercel preview deployments.

Accomplishments that we're proud of

  • One platform for the whole agent lifecycle, from a 17-line spec to a gated deploy, live guardrails, approvals, budgets, audit and rollback, and an AI agent can drive all of it end to end over MCP.
  • The gate found a real money bug we didn't plant and showed the exact twin diff: accounts A-100 balance $0.00 → -$25.00.
  • The Reply PASS / End state FAIL contrast, live on production
  • The whole hackathon cost under $2 in model spend. One wrong $25 credit costs more than 80 deploys.

What we learned

  • Grading replies isn't enough. The most dangerous failures read as polite and correct.
  • Policies belong in code, but they aren't enough on their owna second credit within one conversation. A second credit across conversations slipped past it, and only the end-state checks on the twin's seeded history caught it.
  • Tests deserve the same reuse as tools and prompts: one pack ching tools.
  • The LLM is the thing we test, not the thing we are.

What's next for Shipyard

Shipyard already covers build, test, ship, run, approvals, governance and rollback. Next we close the loop so the lifecycle feeds itself:

  • Health view: automatically grade live conversations and flag bad ones. Today every live run is traced, but nothing grades or alerts on it yet.
  • "Turn into test": a production failure becomes a permaneild → test → ship → watch → learn → test again, and the samemistake can't ship twice.
  • Coding agents on repo twins, using the same gate.
  • Real SSO (identity is simulated today with an "Acting as" switch) and gradual rollouts that send a percentage of traffic to a new version.
  • Real connectors behind the same tool interface, and **fogents.

Built With

Share this project:

Updates

Submission history