Clickwright — About the Project
What it is
Clickwright turns a single computer-use session into a reusable, self-healing connector. You point an AI agent at a real website once; it watches the agent complete a task, records the exact steps it took, and compiles that recording into a versioned connector. From then on, any other agent in your fleet can drive the same site through a standard OpenAPI endpoint — in about a second, with no model in the loop and no tokens spent.
The core idea is simple: computer-use agents solve a task once and then pay for it forever, re-burning thousands of tokens every time they re-read the same screen. Clickwright makes that a one-time cost.
The problem it solves
Every major AI company is shipping computer-use agents: give the model a browser, let it look at screenshots, let it click. It works. It is also expensive and slow, because every single task means:
- Render the screen — one screenshot is roughly 1,000–3,000 tokens.
- The model reads the screen and decides what to do — one inference call.
- Click or type, then go back to step 1.
- Repeat 8–15 times per task.
A 10-step task on Gemini Pro costs roughly $0.03–0.06 and takes 30–60 seconds (measured across the bundled CURA Healthcare target). Do it 1,000 times a month on the same repetitive chore — expense filings, appointment bookings, status checks — and you have spent $30–60 and waited 8–10 hours on something a script could do in seconds.
Even when a system already has an API, the agent still burns tokens figuring out how to call it: reading docs, handling auth, interpreting responses. That overhead is real. Clickwright cuts it out.
How it works
Clickwright moves through five stages:
| Stage | What happens |
|---|---|
| Explore | An AI agent gets a URL and a goal in plain English. It opens a real browser, looks at the screen, decides, clicks, types, and reads the result. Once. |
| Compile | A second pass separates the values a caller would change (amounts, references) from the structure, writes an input schema, and anchors assertions to visible page text. Steps are stored as stable locators recorded from the live page, never invented by the model. |
| Publish | The result is a versioned connector: an OpenAPI document and a skill file, generated on demand from the stored playbook, scoped to the host it was compiled against. |
| Consume | Any agent — one that has never opened a browser — loads that document and calls the site like an ordinary API. Deterministic replay: no model, seconds, $0. |
| Heal | Scheduled canaries replay every connector. When a step stops matching, the run fails at that exact step, the repair loop re-learns it, verifies the result, and publishes the next version. No human asked. |
Why it is different
| Clickwright | Screen scrapers | Agent every time | |
|---|---|---|---|
| How it drives the site | Real steps recorded from a real session | Fragile HTML parsing | AI looks and clicks, every run |
| Cost per call | $0 — no model | $0 | $0.03–0.06 per task (measured) |
| Speed per call | ~1–2 s | fast | 30–60 s |
| Token consumption | Zero after compilation | Zero | 8,000–30,000 per task (measured) |
| Survives redesigns | Yes — detects, repairs, republishes | No | Yes, at full token cost |
| Callable by other agents | Yes — OpenAPI + SKILL.md generated on demand | No | With glue code |
Computer-use agents burn tokens on every screen they see. Clickwright makes that a one-time cost.
Self-healing
Websites change their buttons, move their forms, and rename their fields. The real question is not "does it work today?" but "does it notice when the website breaks it, and does it fix itself?"
Clickwright answers with a canary-and-repair loop:
- A scheduled canary replays every connector against the live site.
- When a step no longer matches, the run fails at exactly that step — not somewhere vague, but the precise control that disappeared.
- The repair loop re-learns just that step the way a user would, checks the result, and publishes a new version. The diff and who (or what) fixed it are visible in the console.
In measured runs, Clickwright detected and recovered from a deliberately broken step on 2 of 2 live targets — naming the exact broken step (step 16 of 16, and step 3 of 3) and succeeding on the next call. By contrast, classic label-matching self-healing tools only repaired steps whose recording happened to keep readable notes; that is exactly the failure mode Clickwright's repair avoids.
Safety and governance
Clickwright is built to be safe by default, not safe after an incident:
- Host fence — a connector can only drive the host it was compiled against. Off-site navigation is refused during exploration and replay, and a deployment-wide ceiling caps all of them.
- Credentials stay out of the AI — sign-in details go to a secret store. The model is told to type literal placeholders; the browser swaps in the real values. Secrets never appear in prompts, screenshots, or recorded steps.
- Policy gateway — amounts above a threshold are held before the browser opens; instruction-style text ("ignore all previous instructions…") is blocked; held actions wait for a human decision in the console.
- PII redaction — emails, card numbers, phone numbers, and IBANs are redacted before anything is persisted or shown.
- Human pause — mid-run, the agent can stop and ask the operator a question. Sensitive answers come back as tokens the model never sees.
- Observability — one trace span per model turn, browser step, and connector call (Cloud Trace when deployed).
Architecture
Clickwright has two sides. On the left, an explorer agent drives a real browser once, a recorder fences the host and substitutes credentials, and a compiler produces a playbook. On the right, a control plane holds the connector registry (versioned OpenAPI + SKILL.md), a deterministic runtime, a policy gateway, and a secret store. Other agents load the OpenAPI document and call the site like an ordinary API. Canaries feed failures back into the explorer for repair.
The repository contains:
app/
agents/ explorer · compiler · healer · consumer
computer/ browser driver · selector recorder · host fence
connectors/ models · registry · deterministic runtime
governance/ secrets · policy gateway · redaction
server.py control-plane API + connector gateway + SSE
telemetry.py tracing
bench/ architecture benchmark suite + explore-vs-replay economics
portal/ a bundled target system (2 tenants, DRIFT flag)
frontend/ React + Vite console
tests/ end-to-end, scope, benchmark, and unit tests
docs/ calling a connector from another agent
portal/ is a real FastAPI app with sessions, validation, and an out-of-order
wizard, so anyone can reproduce a full run — including the healing path, which
needs a target whose UI can be changed on demand.
Benchmarks
The benchmark suite (bench/run_suite.py) answers three questions, none of
which need an AI model to run:
| Check | What it asks | Why it matters |
|---|---|---|
| Does it still work? | Replay the saved steps against the real website | This is the product: seconds per call, $0 |
| Does it find the break? | Secretly break one step's pointer; the health-check must name that exact step | A fixer that repairs the wrong step is worse than useless |
| Does it recover? | Put the good steps back — the next call must succeed | Proves breakage is temporary, not permanent damage |
Latest measured run: 2 of 2 targets still worked ($0.00 per call), both found the broken step, both recovered, at roughly 9 seconds per call (the AI exploration took minutes). Every number shown was measured, and nothing is mocked.
What to point it at
Clickwright works best on any website you already click through by hand: supplier portals, benefits administration, HR and payroll consoles, practice-management systems, and filing portals. Server-rendered forms and wizards — stable structure, real element ids, one page per step — compile into short, durable playbooks. Public sites work too: news, documentation, e-commerce. Anything you could describe to a new colleague in three sentences is a good candidate.
It handles single-page apps with care (generated class names make pointers
brittle; data-testid attributes compile best), pauses for multi-factor codes
during exploration (compile the post-login portion for unattended replay), and
deliberately does not attempt to defeat CAPTCHAs or other bot defences.
Limitations
- Connectors are single-operation by design: one compiled flow, one endpoint. Compose multiple connectors from the calling agent.
- Replay requires the page's controls to still exist. Content-only changes (a rotating headline) don't matter; structural changes trigger a repair cycle.
- Each repair cycle costs one short agent run. On rapidly-changing sites, expect more cycles than a stable internal form would need.
- Unattended replay cannot answer MFA challenges — compile post-login flows.
- Bot defences that block automation also block Clickwright. That is out of scope by design.
The one-line version
Clickwright records one live browser session, then compiles it into a connector any agent calls over OpenAPI in about a second for free, and self-heals whenever the site changes.
