Browser Brain

Inspiration

Every team has one person who knows how the accounting portal works, and everyone else Slacks them. We built a Chrome extension that lets that person record a workflow once and guides everyone else through it, step by step, in their own browser. Then we asked the obvious next question: why does a person have to be first? A public website is just a graph. Send a team of agents to walk it.

What it does

Browser Brain learns how a website works two ways.

Record. Click Record in the extension and do the task. It captures what you click and which fields you fill, as labels and element fingerprints only. Typed values are never read.

Explore. Type a goal and a public URL. Several agents on Brainbase, each in its own sandboxed browser with a different strategy, race to find the path. The shortest clean route becomes a workflow. Brainbase's judge scores every run: did it reach the goal, can a person follow the instructions, how direct was it.

Either way, anyone opens the library, clicks Guide me, and the extension blurs the page and glows the next thing to click, in their own logged-in browser. Nothing is ever clicked for them.

How we built it

• Chrome MV3 extension for recording and the guide overlay. Every step carries a stable element fingerprint (data-testid, then id, then tag plus label), so replays don't need vision until a site changes.
• Node bridge that stores workflows as JSON, cleans recordings with Claude, and runs Explore mode.
• Brainbase runs the agents: one persistent agent on the Claude Code harness with Claude Sonnet 5, a fresh Daytona sandbox per thread. The sandbox entrypoint installs Playwright and a 100-line CLI we wrote: goto, look, click, fill, back, done. The agent drives the site through it, and the CLI records every step with the same fingerprints the extension uses, plus a screenshot per action.
• When a thread finishes, the bridge reads the agent's final JSON, pulls the screenshots out of the sandbox through Brainbase's task file API, ranks the trails, and saves the winner. Three model-judged evals attached to the agent score each run a minute later.
• Pages: a library, a live exploration view with one tile per agent, a filmstrip of what each agent saw, and a history with scores.

Challenges we ran into

• Getting fingerprints out of a cloud agent. Brainbase's managed browser is screenshot and coordinate based, with no DOM access. We found the Universal Harness API accepts an inline entrypoint, so we ship our own Playwright script into the sandbox instead. The whole API turned out to be documented in a public OpenAPI spec, which saved the day.
• Agents that lie. Three agents landed on iana.org's delegation guide and all reported success. The judge read the page's own text, which says delegation is distinct from applying, and failed all three. We kept that behaviour. An independent scorer overruling confident agents is the most honest thing in the demo.
• Agents that forget to finish. Some reached the goal and just stopped without printing their trail. Now the bridge nudges once, and an agent that runs out of time gets a wrap-up turn before it's interrupted.
• Hostile sites. Cloudflare challenges, sites that never answer a datacenter IP, download pages made of dropdowns. Each one became a labelled state instead of a silent failure.
• Fourteen sandboxes at once worked, just slowly. Three is the sweet spot.

What we got done in 8 hours

• Explore mode end to end: goal in, agents out, winner saved, verified live with 2 to 5 agents on python.org, MDN, Wikipedia and iana.org.
• A Chrome extension that records workflows and guides people through them, with a Discover button that sends agents to the current site.
• Library, live exploration view with a tile per agent, a filmstrip of what each agent saw, a command trace, and a history page with judge scores.
• Three model-judged evals on Brainbase scoring every run, with verdicts on the tiles and library cards.
• 50+ commits, 122 tests, a brand and style guide, and this write-up.

Tech stack and sponsor tools

Brainbase Labs is where the agents live. One persistent agent on the Claude Code harness, a fresh Daytona sandbox per explorer thread, started through the Managed Agents API. The sandbox entrypoint installs Playwright and our explorer CLI. We read each thread's status, messages and events for the live tiles, pull screenshots out of the sandbox through the task file API, and attach three evals to the agent so Brainbase's judge scores every run.

Anthropic models do all the thinking. Claude Sonnet 5 drives each explorer inside the Claude Code harness, judges the evals, cleans human recordings into steps, and picks elements from a screenshot when a site has changed and a fingerprint no longer matches.

Cloudflare and Stripe were not used. We dropped a marketplace and billing layer early to keep the demo focused.

Everything else: a Chrome MV3 extension in plain JavaScript, a Node and Express bridge with JSON on disk, Playwright inside the sandboxes, Vitest for tests.

Track: Autonomous Organizations. A team of agents coordinates on one goal with different strategies, an independent judge scores their work, and the result is handed to people as a guided workflow.

What we learned

Humans and agents can teach the same system in the same format, and a judge that reads the page beats an agent that grades itself. Also: filter buttons that toggle a hidden attribute do nothing when the card has display: flex. Ask us how we know.

What's next for Browser Brain

A nightly re-explore that detects when a site changed, patches the trail, and posts the diff to Slack. Self-improving workflows in one feature.

Built With

• anthropic
• brainbase
• chrome
• claude
• claude-code
• claude-sonnet-5
• daytona
• express.js
• html5
• javascript
• node.js
• playwright
• vitest