Shiftwright

Agents that migrate a Selenium or Cypress suite to Playwright, run every test in a container to prove it, and heal what breaks.

Comment

Inspiration

Legacy Selenium suites rot: brittle XPath, arbitrary time.sleep(), CI flakiness. Teams want Playwright's auto-waiting and parallelism, and they don't migrate, because it's weeks of work that ships no features.

The numbers say how big that backlog is, and it is not only Selenium. Selenium ships ~57M PyPI installs a month, flat; Cypress ships 31.4M npm downloads a month. Public GitHub holds ~1.78M source files across the two — 927,744 Selenium Java, 453,632 Selenium Python, 401,664 Cypress (JS + TS). Of those Python files only 780 import Playwright as well, 0.2%, because migrating is a rewrite per test: 246,784 of them pair Selenium with time.sleep, and Playwright has no implicit wait to map it onto. Playwright ships official migration guides for Protractor, Puppeteer and testing-library. None for Selenium. (pypistats.org 2026-08-22; api.npmjs.org and GitHub code search 2026-08-26. Code-search counts are estimates over a live index; the co-existence figure read 109 on 22 Aug and 780 on 26 Aug.)

Migration is a runtime problem. The code has to run, the failure has to be read against live DOM, and the fix has to hold. Shiftwright executes what it writes: every migrated test runs in a container first, so what lands in your pull request is a suite that already passes.

Features and functionality

Point Shiftwright at a public end-to-end test repository — Selenium in Python, Selenium in Java, or Cypress in JavaScript or TypeScript — and a coordinated swarm of Google ADK agents migrates it. The repository picks the path. detect_language routes it to one of three scanners, prompt sets and patch guards, and the sandbox runs the result under the matching toolchain. A Java suite comes out as Playwright Java, a Cypress suite as Playwright TypeScript.

  1. Curriculum planning — the Planner parses the repository AST, builds the import graph, and orders files into topological phases: fixtures, then leaf page objects, then the journeys that use them. Early migrations seed a knowledge base later ones reuse.
  2. Idiomatic modernization — the Modernizer rewrites imperative driver logic into accessibility-first locators (get_by_role, get_by_label) and web-first assertions, with no sleeps and no raw XPath. A check enforces that; a file that still contains them gets one corrective rewrite and is then left for a human.
  3. Sandboxed execution — a private MCP server on Cloud Run runs each migrated file in an isolated container, under the runner its language needs: pytest-playwright, Maven/JUnit 5, or @playwright/test. It captures exit code, output, DOM, screenshot and trace.zip. All three toolchains ship in the image and are smoke-tested at build time.
  4. Multimodal self-healing — on failure the Diagnoser receives the screenshot and DOM at the failing step, as image plus text. It names the root cause and patches the locator or the wait. Three attempts, each briefed to try a different repair than the last.
  5. Compounding memory, two tiers — every resolved locator, healing recipe and flake heuristic persists to Firestore, scoped to the repository that learned it. Lookup is a hash hit first, then EmbeddingGemma cosine for near-identical selectors. The same facts also go to Vertex AI Agent Engine Memory Bank, which is asked the question Firestore cannot: what has ever worked for a widget like this, in any repository this project has migrated.
  6. Benchmark and PR — the Verifier times the same tests twice in the same container, under the legacy runner and under Playwright, and drafts the pull request from those numbers.

Two human gates bound it: approve the plan, approve the result. Approval opens a draft pull request on a new branch as a GitHub App.

What the workflow gets you

Closed-loop autonomy with a real oracle. The agent runs the code, reads the runtime failure, fixes its own mistake, and runs it again. The exit code decides when it is done.

A live view of the run. The dashboard subscribes to Firestore and renders the run as it happens. The curriculum graph lights up file by file. Each file shows its Selenium source beside its Playwright rewrite. The Diagnoser's screenshot appears with the DOM lines it read, the knowledge base counts up, and the agent strip shows who is working. Browser and worker never call each other. Both read and write the same documents, so the UI has no backend to lose.

13-of-16 on a repo we did not design for. An unmodified third-party suite migrates to 13 passed / 3 failed. Those three match the Selenium baseline measured in the same container beforehand: the migration changed the framework and left the assertions alone.

Memory that fires. On a run of the pytest corpus the Diagnoser recalled three prior heals for one file. Two were matched by EmbeddingGemma similarity rather than exact string.

Memory that crosses repositories. A run of the third-party corpus healed two files and wrote eight memories to Memory Bank — for each repair, the locator mapping and the diagnosis, to both the repository's own scope and the shared one. Asked afterwards how to fix that timeout as a repository the system has never seen, recall returns the diagnosis it had just learned: the dropdown's first option is disabled, so select_option waits for it forever, and evaluate("el => el.selectedIndex = 0") is what passed.

One agent tree, three source stacks. The same Planner, Modernizer, Diagnoser and Verifier migrate Selenium/Python, Selenium/Java and Cypress/JavaScript. Only the scanner, the prompt variant, the patch guard and the sandbox runner vary. The workflow itself never learned there was more than one kind of repository.

Technologies used

Control plane — Google ADK 2.7, Gemini 3.5 Flash on Vertex AI
  SequentialAgent "shiftwright"
    Planner    (LlmAgent)                  structured plan, pydantic
    Migration  (BaseAgent)
      Modernizer (LlmAgent)                one file per ADK session
      Diagnoser  (LlmAgent, multimodal)    output_schema=Diagnosis
    Verifier   (BaseAgent)                 benchmark + HITL gate
                          |
      +-------------------+--------------------+
      v                                        v
Data plane — Firestore + GCS          Execution plane — MCP server
  migrations/{id}   bus + run record    (Cloud Run, private, ID token)
  knowledge_base/locators               mcp_ast_parse
  knowledge_base/healing_recipes        mcp_playwright_sandbox_exec
  knowledge_base/flake_heuristics       mcp_selenium_baseline
  traces, DOM, screenshots (GCS)        mcp_trace_inspect
                                        mcp_kb_query / mcp_kb_upsert
                                        EmbeddingGemma-300M, int4, on CPU
  • Google ADK 2.7 manages workflow state, session lifecycles and the human gates. All four agents run Gemini 3.5 Flash on Vertex AI (location=global), text and vision.
  • MCP — a FastAPI JSON-RPC server on Cloud Run, private and ID-token authenticated, exposing six tools. Reasoning is decoupled from execution: untrusted test code runs in a container the agents never touch directly.
  • EmbeddingGemma-300M, int4 ONNX, ~200 MB, on the MCP server's own CPU — no GPU, no endpoint, no extra hop. Each hit reaches the model carrying its cosine score.
  • Vertex AI Agent Engine Memory Bank holds the cross-repository tier — durable, semantic, scoped by repository with a shared bucket for what outlives one suite. Firestore stays authoritative for the numbers, because it is exact and a locator this run already learned has to come back verbatim; Memory Bank generalises.
  • Firestore is both the memory service and the message bus. Browser and worker never call each other. Both read and write Firestore and the dashboard subscribes, so the UI has no backend to be down.
  • Cloud Storage holds every artifact a claim rests on: traces, DOM snapshots, failure screenshots and the per-run recap video.
  • Cloud Text-to-Speech narrates that recap, from a script built out of the run record itself.
  • Secret Manager holds the GitHub App private key; the worker never sees it on disk.
  • Cloud Build + Artifact Registry build and store the images for all three services.
  • Three Cloud Run services — dashboard (public), MCP server (private, ID token), orchestrator worker (private, always-on).

Agentic design patterns implemented

Prompt chaining — SequentialAgent[Planner → Migration[Modernizer, Diagnoser] → Verifier], with pydantic Plan and Diagnosis between stages, so every hand-off is typed rather than prose.

Multi-agent, hierarchical — a sequential coordinator over a migration stage that is itself a coordinator for its own two LlmAgents. Five ADK agents in one tree.

Planning — the Planner emits a structured curriculum built from the AST's import graph. Phases are topological layers; the model chooses order only within a phase.

Prioritization — that curriculum order is the priority: fixtures before the page objects that import them, page objects before the journeys, so knowledge flows forward.

Routing — classify_failure() derives the failure class from stderr and briefs the Diagnoser accordingly. A timeout is told to prefer a wait over a new selector. A strict-mode violation is told to narrow rather than index. A navigation failure is told not to touch locators at all.

Parallelization — the topological cut is the safety proof. A phase's files depend only on earlier phases, so their whole pipeline overlaps, each file in its own ADK session. The level is a setting: --concurrency, an environment default, or a field on the request document, clamped 1–8. Each concurrent file holds a sandbox browser and a Gemini call.

Reflection (generator–critic) with a real oracle — producer (Modernizer) and critic (Diagnoser) are separate agents. The critic's signal is a process exit code. The patch it proposes is then checked by non-model code before it is accepted.

Tool use — six tools, and the agents reach the repository, the browser and the knowledge base only through them.

Model Context Protocol — those tools are JSON-RPC on a private Cloud Run server. Running untrusted test code becomes a deployment concern rather than an agent concern.

Memory management — short-term is ADK session state. Long-term is two tiers: Firestore, scoped to the repository that learned each entry, because #login-button in two apps is one string, a near-identical embedding, and two different elements; and Vertex AI Memory Bank for what generalises past one suite. The exact tier answers during a migration, the semantic one answers across them.

Knowledge retrieval (RAG) — two-tier lookup: sha256 exact, then EmbeddingGemma cosine. Same-scope neighbours need 0.80. Cross-repository neighbours need 0.90 and arrive labelled. Every hit reaches the model carrying its score.

Learning and adaptation — successful heals write locators, healing recipes and flake heuristics back, and later files read them, so the system gets cheaper as a migration proceeds.

Reasoning techniques — diagnosis is multimodal: the screenshot and the DOM at the failing step, as image plus text, against the failing source and the error.

Exploration and discovery — each heal attempt is briefed to differ from the last. The smallest justified repair, then a different kind of locator, then widen. The third is told that a pre-existing-failure verdict beats another guess.

Guardrails — patch_guard.py rejects any patch that changed an expectation. The anti-pattern check gives one corrective rewrite and then fails the file. A per-installation quota bounds runs. Pull requests require ownership or a completed App install.

Human-in-the-loop — two blocking gates, and the second one gates the only outward-facing side effect.

Exception handling and recovery — the MCP transport retries 429/5xx with backoff and re-mints an expired ID token. It never retries a JSON-RPC error, which is the server's answer rather than a transport failure. Non-convergence degrades to fallback. A failed pull request cannot fail the run.

Goal setting and monitoring — a bounded heal budget, a per-file status taxonomy (passed, healed, preexisting, fallback, skipped), and every state change streamed to the run record.

Resource-aware optimization — EmbeddingGemma runs int4 at 256 dimensions on the MCP server's own CPU. Sandbox executions are time-boxed. The concurrency ceiling is set by what the sandbox can hold.

Evaluation and monitoring — the same tests, timed twice in the same container minutes apart, under Selenium and under Playwright.

Other data sources used

Four public MIT-licensed end-to-end suites, forked into shiftwright-dev so a finished migration can open its pull request against a repository this project owns. Upstream is credited in the UI:

  • CypherMorgan/saucedemo-selenium-pytest — two anti-patterns deliberately injected
  • jaquelineleite/qa-automation-selenium-java-saucedemo — unmodified
  • ibanezmartha9/Cypress-the-internet.herokuapp — unmodified
  • dbeniamin/Python_Selenium_automation — unmodified

They drive the public demo sites saucedemo.com and the-internet.herokuapp.com.

Market figures come from pypistats.org, api.npmjs.org and the GitHub code search API, captured 2026-08-22 and 2026-08-26.

The replay at /runs/demo is labelled as a replay. It is real agent output exported from live runs.

Findings and learnings

Measured results

Four public MIT-licensed corpora: one per source stack, plus an unmodified third-party suite we did not design for. Latest run of each.

Repository Source stack Size Outcome Benchmark (paired)
saucedemo-selenium-pytest (2 anti-patterns injected) Selenium · Python 4 tests, 4 page objects, 436 LOC 9 passed, 2 healed, 0 fallback 33.4s → 19.4s (1.7×), 2 sleeps removed (7.0s)
qa-automation-selenium-java-saucedemo Selenium · Java 8 tests, 7 page objects, 1298 LOC 15 passed, 0 healed, 0 fallback 81.3s → 64.1s (1.3×), 30 WebDriverWait polls removed
Cypress-the-internet.herokuapp Cypress · JavaScript 4 tests, 5 page objects, 177 LOC 9 passed, 0 healed, 0 fallback 78.5s → 28.0s (2.8×)
Python_Selenium_automation (unmodified third-party) Selenium · Python 16 flat unittest files, 656 LOC 12 passed, 1 healed, 3 already-failing, 0 fallback 82.8s → 56.8s (1.5×) over the 13 tests green on both sides, 8 sleeps removed (10.0s)

How to read the table. Outcome counts files — the Java row migrated 5 test classes and folded 10 page objects into them, so 15 is the file count, which is what every "green" number on the dashboard counts. The benchmark pairs only tests that passed under both runners, so a legacy timeout never reads as speedup. Per-test timings are in each run record.

Draft pull requests from real runs, opened by the GitHub App. One per source stack:

https://github.com/shiftwright-dev/saucedemo-selenium-pytest/pull/1 https://github.com/shiftwright-dev/qa-automation-selenium-java-saucedemo/pull/1 https://github.com/shiftwright-dev/Cypress-the-internet.herokuapp/pull/1 https://github.com/shiftwright-dev/Python_Selenium_automation/pull/3

An animated walk through the architecture, naming the real wire at every step: https://shiftwright.dev/architecture

What we ran into

A migration that edits assertions has destroyed the thing the suite was for. Our first run against a repo we had not designed for came out 16/16. It should have been 13/16. The Diagnoser had made three genuinely-failing tests pass by rewriting what they asserted. Each diagnosis was right that the test was wrong, and each was out of scope. patch_guard.py now extracts every test's expectation set before and after each patch, and refuses any change to a comparison operator, an expected literal, or the assertion count.

A verdict is only worth what the runner behind it did. Adding Java and Cypress created three ways to report an outcome nobody had measured. We found all three by running a full migration rather than trusting the unit tests.

The stripper that removes markdown fences matched ```python and nothing else, so ```java survived into the file handed to Maven. Every Java file failed to compile. The failure surfaced as Maven's [Help 1] documentation URL, because no branch of the error parser matched a javac line. The benchmark then timed those failures and reported the suite 5.1x faster — a build that falls over is fast.

Playwright's default testMatch collects only *.spec.ts. A migrated Cypress spec under its own login.cy.ts name went to the runner and was filtered back out. "No tests found", which the Diagnoser reasonably read as a pre-existing failure in the user's repository.

Both are now rules: pair only tests that passed on both sides, and a runner that collected nothing reports harnessError rather than a verdict.

A knowledge base can be dead and look alive. Ours had one writer storing Playwright expressions and one reader querying By.ID=… strings. Two vocabularies that could never match, so "reusing N locators" had never once fired — while counters incremented locally for documents that were never written.

Trace bundles are too big to reason over. The DOM at failure is truncated to a bounded window, and the screenshot is passed inline only under the document limit. The Diagnoser sees the failed action's context, not megabytes of mutation log.

Serial migration looked like a framework constraint and was not. The Modernizer is an LlmAgent driven through session state, and concurrent turns race on it. The constraint was the single session, not the agent. One session per file, and the same agent parallelizes untouched.

What we learned

Give an agent an oracle and generation stops being the hard part. Scope does. Knowing which failures are yours to fix is the real problem, and enforcing that boundary in code worked where prompting did not. The guard costs no legitimate heal, and it is why the third-party corpus lands at 13-of-16.

Systems drift where they assert instead of compute. One run exposed three defects of the same shape. The human approved a pre-heal draft rather than the code that passed. 18 unimported scripts were reported "folded into the migrated tests", with nothing checking. Scan totals counted files the migration never opens. Each was a sentence in the UI that no code backed.

Structure beats model size. A curriculum, a typed hand-off and a scoped memory made Gemini 3.5 Flash sufficient for every role, multimodal diagnosis included. The wins came from the shape of the workflow.

MCP made the sandbox modular. Standardising execution on JSON-RPC tools decoupled reasoning from the environment. Running untrusted test code in an isolated Cloud Run container became a deployment detail rather than an agent concern.

What's next

  • More source frameworks — Appium, and Selenium in C#.
  • Cycle-tolerant planning — legacy suites have circular imports between page objects; the planner's ordering should survive them rather than depend on a clean graph.
  • A self-healing CI action — the Diagnoser packaged as a GitHub Action that reads a flaky failure on a pull request and proposes the patch.

Built With

  • cloud-build
  • cloud-run
  • cloud-storage
  • cloud-text-to-speech
  • embeddinggemma
  • fastapi
  • firestore
  • gemini-3.5-flash
  • github-apps
  • google-adk
  • mcp
  • onnxruntime
  • playwright
  • pytest
  • python
  • secret-manager
  • selenium
  • sveltekit
  • tailwindcss
  • typescript
  • vertex-ai
Share this project:

Updates