WebPilot AI

Inspiration

Most "AI browser agents" fall into a costly and dangerous trap: they reason on every single run. This makes them slow, expensive, and constantly one bad model hallucination away from clicking the wrong thing or breaking a workflow. We were inspired to build a system that acts more like a human teaching a reliable worker: you show them how to do it once, and they repeat it perfectly every time. We wanted to prove that self-healing in AI doesn't just mean "retry and pray," but rather a robust, auditable pipeline of diagnose $\rightarrow$ patch $\rightarrow$ sandbox-verify $\rightarrow$ independently-verify $\rightarrow$ version $\rightarrow$ resume.

What it does

WebPilot AI is an Autonomous Web Operations Platform that understands a web task once, runs it deterministically, and heals itself when the web changes.

Teach Once, Replay Free: Give it a natural language goal. WebPilot uses Gemini 3.7 Flash and Google ADK to navigate and learn the task on a live browser. It then compiles an immutable, typed WorkflowSpec. Every subsequent run replays deterministically using Playwright with zero Gemini reasoning calls.
Self-Healing: If a website changes (e.g., a button moves or a DOM element is renamed), the run fails safely. WebPilot captures the failure context (DOM + screenshot) and invokes a bounded Recovery Agent to diagnose and propose a minimal patch.
Independent Verification: The patch is replayed in a sandbox and checked by a second, independent Gemini Verifier model that has no visibility into the first model's reasoning. If it passes, a new version is promoted automatically, and the original run resumes exactly where it left off!
Human-in-the-Loop Security: Nothing the model writes ever executes as code. We use strict Zod-validated data specs. High-risk actions, recovery approvals, or plan approvals pause the run for human review via our dashboard or integrations (Slack, Gmail, Chat).

How we built it

We built WebPilot AI using a robust, cloud-native architecture on Google Cloud:

Core Orchestration: We used a Next.js 16 frontend for the executive dashboard and a NestJS/Fastify API for the control plane.
Browser Execution: A private Playwright worker (running on Cloud Run) executes the workflows securely.
AI Integration: We leveraged Google ADK (LlmAgent + InMemoryRunner) and @google/genai (Vertex AI) with the gemini-3.7-flash model. We strictly separated concerns into four narrow, single-purpose agents: Planner, Navigator, Recovery, and Verifier.
State & Queuing: We utilized Cloud SQL (PostgreSQL via Prisma 7) for our relational domain model, Cloud Tasks for OIDC-authenticated run dispatch, Pub/Sub for operational events, and Cloud Storage for artifacts.
Infrastructure as Code: The entire topology is declared using Terraform, enforcing strict least-privilege IAM service accounts across the board.

Challenges we ran into

Stopping AI from hallucinating code: We had to strictly enforce that the AI only outputs structured data (WorkflowSpec) and never executable code (no eval()). Designing a deterministic engine that translates AI intent into Playwright actions without losing flexibility was a major hurdle.
Handling DOM Drift Seamlessly: Websites change constantly. Building the recovery pipeline required complex state management—specifically, pausing a failed Playwright execution, taking a snapshot, invoking the Recovery agent, running the patch in a sandboxed context, independently verifying it, and seamlessly resuming the original run without the end-user experiencing a failure.
Verifying the Verifier: We realized that an AI verifying its own patch is inherently flawed. We had to build an architecture that strictly isolates the Recovery Agent (diagnosing and patching) from the Verifier Agent, ensuring independent mathematical and logical validation.

Accomplishments that we're proud of

Achieving zero-reasoning fast paths. The fact that we can run complex web operations infinitely with zero LLM API costs and zero latency overhead after the initial discovery phase is a massive win.
Building a genuinely self-healing system. We successfully created a loop where a run can fail on step 2, auto-diagnose, verify a patch with 0.95 confidence, promote a new version in under 10 seconds, and finish the extraction—all in one continuous execution.
Delivering a fully deployed, production-ready system on Google Cloud with strict security boundaries, OIDC-authenticated dispatch, and immutable enterprise audit trails, rather than just a local script.

What we learned

AI in production is about bounding the model. Narrow, single-purpose agents with strict contracts (like our Planner, Navigator, Recovery, and Verifier) perform far better and are much safer than one monolithic "do-everything" agent.
Determinism is king. For enterprise adoption, users need to know exactly what the agent will do before it does it. Generating data instead of code is the key to safe AI web automation.
Testing LLM-driven UI flows requires completely new paradigms, leading us to build our own deterministic "DOM-drift" demo portal just to reliably test our self-healing capabilities.

What's next for WebPilot AI

Cross-Workflow Learning: If the Recovery Agent learns that a specific site's login button changed in one workflow, we want it to proactively propose patches for all other workflows interacting with that same site.
Expanding Integrations: Finalizing the Gmail API and Google Chat notification adapters to bring human-in-the-loop approvals directly to where teams already work.
Visual-only Fallbacks: Incorporating multimodal computer vision logic for the Navigator to fall back on when the DOM is completely obfuscated (e.g., <canvas> driven web apps).