ShipBash

You ship it. We bash it.

An all-night product tester: real browser, real user journeys, video proof. You only review the results.

I work on CI/CD. AI made building and shipping dramatically faster — and honestly, it made me trust my releases less. After every deploy I'd still open production and click through the same flows by hand: login, onboarding, the workspace, whatever this release could've broken. Tests pass. Nobody re-checks the product.

I didn't want another tool that generates browser scripts I'd then have to maintain. I wanted to hand an agent the live product, let it figure out what matters, and get back enough evidence to decide whether the release is fine. That became ShipBash.

What it does

Give ShipBash one production URL. It opens the product in a real browser and explores it like a new user, then proposes a handful of user journeys — things a person should be able to accomplish ("sign in and reach your workspace"), written as outcomes and checkpoints. No CSS selectors, no click scripts.

When it hits sign-in, SSO, or MFA, it stops and hands me the browser. I log in, click "I'm done", and it keeps going in the same session. It never touches credentials.

The journeys are proposals, not decisions. I can delete one, rewrite one in plain language, add what it missed. After approval, ShipBash re-runs every journey against production and comes back with a verdict per checkpoint, screenshots, the action log, and a recorded replay of everything the agent did (sped up where nothing happens).

All that evidence exists for one reason: so you can tell a real product regression apart from a verification that just had a bad day.

How I built it

Every line of ShipBash was written in Codex with GPT-5.6 — product model, architecture, tests, deploys, all of it. Through Codex browser and computer use it also created and wired up the GitHub, Supabase, and Vercel projects, set up auth and domains, and checked the live site after each deploy.

Codex is also the product's runtime, not just my dev tool. Every discovery and verification in the demo is a codex exec session (gpt-5.6) driving an isolated Chromium through an MCP browser server, and it has to return a schema-validated result — journeys, checkpoints, verdicts — or the run gets rejected.

The debugging stories are what sold me on this workflow. An OAuth code kept landing on / instead of /auth/callback; Codex traced it. Runner jobs finished with video and result files sitting on disk but nothing written back to the database; Codex found the race. And on deadline day, runs kept dying with an opaque "failed before returning a valid result" — so I had Codex teach the runner to log its own dying words. The very next crash printed the real cause: the agent had attached proposals to a status that forbids them. One prompt fix and it was gone.

What took the most time

Not the browser automation. Deciding when to trust a result.

A verification crosses authentication boundaries, runs for minutes, records video, restarts browser processes, and depends on infrastructure that fails independently of the product under test. A dead agent process isn't a product failure. A clicked button isn't proof either — the expected outcome has to actually show up on screen. So ShipBash keeps the evidence attached to the verdict, and a human can always open the replay instead of trusting a green checkmark.

Where it goes next

Right now the loop is: discover, approve, verify, review. The next step is closing it. When a verification fails, the full failure context — actions, checkpoints, console, network, page state, video — should land directly with the coding agent already working on that repo. It reproduces the problem, ships a fix, and the new release goes back through ShipBash. Wired into the chat, issue, and CI tools a team already lives in, the release note can finally say what broke, what the evidence shows, and whether the re-verification passed.

Agents build the product. One agent should also check what actually reached production.