M.I.N.D. - AI-powered autonomous web testing agent

M.I.N.D. is an AI QA agent that writes, runs, and self-heals Playwright tests from plain English. Use Gemini Live and visual AI to eliminate test maintenance and ship faster.

Comment

About The Journey

Inspiration

Software teams waste countless hours writing and maintaining brittle UI tests because websites constantly change. This pain point inspired M.I.N.D. (Machine Interface for Navigation & Diagnostics) — an AI-powered autonomous web testing agent capable of replacing manual QA testing. The goal was to build a system where you could just tell the agent your testing goals conversationally, and it would handle the rest: planning workflows, executing interacting with a real browser, self-healing during code changes, and visually verifying the results.

What it does

M.I.N.D. is an AI-powered QA engineer built with Next.js, Google Gemini, Playwright, Redis, and MongoDB. It features a conversational interface powered by Gemini Live AI to extract requirements from the user. It then plans test cases, executes them autonomously using Playwright, and uses Advanced Image Analysis via Gemini Flash for visual validation of the screen states. It tracks full execution runs with session replays, steps, network logs, and self-healing records, offering highly detailed interactive reports.

How we built it

The core architecture consists of a multi-agent pipeline:

  • Requirement Agent: Uses Gemini Live to converse and extract details.
  • Test Planner Agent: Uses Gemini Flash to translate requirements into actionable Playwright steps.
  • Browser Execution Engine: A headless Playwright instance that executes steps and captures the DOM/Screenshots.
  • Report Agent: Uses Gemini to synthesize the execution artifacts (screenshots, logs, timings) into an interactive report.

The entire project is deployed natively on Google Cloud. We leveraged Cloud Run for scalable container execution and Artifact Registry for our Docker images. The background execution runs via a BullMQ job processor managed with Redis.

We implemented a tiered AI strategy to optimize cost and performance:

  • The base application seen via the live URL runs on the blazingly fast and cost-effective gemini-3.1-flash-lite-preview model for all rapid reasoning tasks (navigation, conversation).
  • For very complex web UIs or deep reasoning (e.g., generating exhaustive test reports), the system architecture seamlessly supports upgrading to the Pro models (like gemini-3.1-pro-preview or future iterations) based on client needs and pricing tiers.

Challenges we ran into

Being relatively new to Google Cloud, the deployment and infrastructure side presented sharp learning curves. We wrestled with:

  • Container Registry Transitions: We hit roadblocks trying to push our initial Docker images before realizing we needed to configure proper IAM permissions (createOnPush) and switch over to Google Artifact Registry from the legacy gcr.io.
  • IPv6 Networking Quirks: Our worker instances on Cloud Run were throwing mysterious ENOTFOUND DNS errors when trying to connect to our Redis instance. After deep debugging, we found our IORedis configuration was forced IPv6 lookup on an environment that expected IPv4 default resolution.
  • Agent Orchestration: Ensuring deterministic outputs from LLMs required refining the way we passed state between the planning agent and the Playwright execution engine to prevent hallucinations or malformed JSON commands.
  • Hybrid Live Inputs: Capturing hybrid inputs during an active voice session with Gemini Live AI was tricky. Native voice often struggles with exact spelling of URLs or passwords, so we had to build a system that seamlessly accepts simultaneous text input without breaking the active audio connection.
  • Local GCS Testing: Testing Google Cloud Storage (GCS) asset uploads from our local development environment proved difficult due to service account authentication and restrictive IAM policies. We managed this by building an abstraction layer that temporarily stores execution artifacts (screenshots/videos) in an alternate local directory, seamlessly swapping to GCS only upon production deployment to Cloud Run.

Accomplishments that we're proud of

  • Streaming Execution State: We successfully built a realtime WebRTC-style Server-Sent Events (SSE) pipeline that streams live screenshots from the headless Playwright browser to the Next.js UI, so users can literally watch the AI "think" and click in real-time.
  • Self-Healing Selectors: We're proud of the DOM Analyzer module. When standard selectors break or web elements change classes, the system dynamically analyzes the surrounding DOM and finds the intended element, automatically recording the fix.
  • Cost-Effective Architecture: Getting an autonomous browser agent to perform reliably using the cheapest Gemini Flash models proves the efficiency of our prompt engineering and multi-agent design, making this a viable product to scale out to different pricing tiers.

What we learned

  • Agent Resilience Strategy: Building highly autonomous agents requires explicit fail-safes. You cannot just prompt an LLM; you must surround it with rigid data validation (we used Zod extensively), retry loops, and deterministic fallback functions to prevent infinite hallucination loops.
  • Dockerizing Headless Browsers: Deploying full-stack applications with headless browsers (Playwright) via Docker requires careful attention to system dependencies, fonts, and codecs, especially in serverless environments like Google Cloud Run where memory and port handling are strictly regulated.
  • Asynchronous Execution: Managing asynchronous, long-running processes (browser execution) across a microservices architecture is vastly more stable when delegating to a dedicated job queue (BullMQ/Redis) rather than trying to handle heavy tasks within standard HTTP API requests.
  • Mastering Google Cloud Services: Developing entirely on Google Cloud exposed us to a powerful ecosystem but also sharp learning curves. We learned how Cloud Build automates CI/CD, how Artifact Registry secures container images, and the nuances of configuring IAM permissions so that Cloud Run services can securely talk to other GCP resources.
  • Navigating Cloud Storage (GCS) Nuances: Implementing Google Cloud Storage for storing execution artifacts taught us valuable lessons about handling streams and service account authentication. We learned that while local filesystem testing is straightforward, mirroring cloud-native blob storage locally requires robust abstraction layers to ensure development parity.
  • Balancing Cost and Intelligence: Using the tiered Gemini models demonstrated that you don't always need the heaviest, most expensive model. We learned to strategically deploy gemini-3.1-flash-lite-preview for high-volume, low-latency tasks (like navigation decisions) while reserving advanced visual validation for the heavier lifting—teaching us that efficient AI engineering is about routing the right task to the right tool.

What's next for M.I.N.D. - AI-powered autonomous web testing agent

  • Enterprise Pro Features: Expanding our integrations to allow seamless switching to Gemini Pro models for enterprise clients who need the deepest possible visual analysis and exploratory testing capabilities on highly complex graphical interfaces.
  • CI/CD Pipeline Integration: Allowing engineering teams to trigger M.I.N.D. automatically on GitHub pull requests. The AI will read the PR diff, design a test to cover the code changes, run the test, and post a report back as a PR comment.
  • Persistent AI Memory: Implementing vector search over past test runs so the agent "learns" a company's specific application over time, making its selector healing and planning even faster on subsequent runs.
  • MCP Server Integration: We plan to expose M.I.N.D. as an MCP (Model Context Protocol) server. This will allow our entire autonomous testing and browser interaction framework to be utilized natively by other AI agents and seamlessly integrated into hundreds of existing AI tools.

Built With

Share this project:

Updates

Submission history