-
-
web-app-screenshot
-
Inspiration
What it does
Inspiration I work in QA. I know better than most how often end-to-end tests get skipped — not because developers don't care, but because writing them is genuinely painful. You need to know the DOM selectors, the API routes, the edge cases, the anti-flaky patterns. It takes hours for a single page. Most teams skip it entirely and hope nothing breaks in production. When I saw the Gemini Live Agent Challenge focused on Visual UI Understanding, I immediately thought — what if Gemini could look at a web app the same way a QA engineer does, and just write the tests? That became Spectre.
What it does Spectre is a Visual QA Agent with two modes: Web app — drop in a screenshot of any web app, Gemini 2.5 Flash analyses the UI and generates a complete Cypress test suite with cy.intercept() aliases, viewport tests, error states, and anti-flaky patterns built in. Spectre Lens (Chrome Extension) — a companion extension that captures the real DOM tree, network requests, and a full session recording of your user journey. Gemini receives screenshot + DOM + network log together and generates tests with real selectors and exact API routes — eliminating the two biggest sources of hallucination in AI-generated tests. The web app detects the extension automatically via a postMessage handshake and surfaces a Start Recording button inline. The developer records themselves using their app naturally — logging in, navigating, clicking — and Spectre reconstructs the complete user journey into a production-ready test suite.
How we built it
Gemini 2.5 Flash handles all visual analysis and test generation via the Google GenAI SDK. The multimodal input — screenshot + DOM snapshot + network log — gives Gemini everything a human QA engineer would need Node.js + Express backend exposes five routes: /api/analyze, /api/generate, /api/session, /api/generate-from-extension, and /api/generate-from-session React + Vite frontend walks users through a 4-step flow: Capture → Analyse → Configure → Generate Chrome Extension (Manifest V3) with a content script for DOM scraping and session recording, a DevTools page for network capture, and a background service worker that owns all API calls Google Cloud Run hosts both frontend and backend with continuous deployment from GitHub via Cloud Build
The extension prompt enforces a strict strategy — Gemini is explicitly instructed to only use selectors found in the captured DOM tree and only use routes found in the network log. This constraint is what separates Spectre from a simple screenshot-to-code tool.
Challenges we ran into Hallucination was the biggest challenge. Early versions generated plausible-looking tests with invented selectors and guessed API routes that would never pass. The fix was combining the DOM snapshot and network log with the screenshot, then enforcing strict prompt rules that Gemini could not deviate from. Extension architecture was harder than expected. Chrome's Manifest V3 service workers are ephemeral — they shut down between events. Moving all API calls to the background worker and persisting job state in chrome.storage.local was the key insight that made session recording reliable even when the popup closed mid-generation. Token limits required careful DOM minification — stripping scripts, styles, SVGs, and truncating large repeating lists to keep the DOM snapshot within Gemini's context window without losing the selectors that matter. CORS between the Chrome extension origin and the Cloud Run backend required explicit handling that standard web app CORS configs don't cover by default.
Accomplishments that we're proud of The quality of the generated tests genuinely surprised us. Using a real e-commerce checkout page as a test case, Spectre generated a 200+ line Cypress suite with the exact Firestore document IDs, real Firebase Auth endpoints, correct product names from the network log, and aria-label selectors pulled directly from the DOM — all without ever having access to the source code. The web app ↔ extension bridge via postMessage also works seamlessly — the developer never leaves the web app to start a recording session, and results persist in the extension popup even after it closes.
What we learned Gemini 2.5 Flash's multimodal capabilities are genuinely powerful — given enough context it reasons about UI the way a senior engineer would, not just pattern-matching on pixels. The key is giving it the right constraints so it cannot hallucinate its way to a plausible-but-wrong answer. The combination of screenshot + DOM + network log is what makes the output production-quality. Any one of those alone produces mediocre tests. All three together produces tests a senior QA engineer would be proud to write.
What's next for Spectre — Visual QA Agent
Browser extension on the Chrome Web Store — making Spectre Lens available to any developer without manual installation HAR file upload — allow developers to upload a recorded network session for even more accurate API route detection Playwright support — generate tests for Playwright in addition to Cypress CI/CD integration — a GitHub Action that automatically generates regression tests when a PR modifies UI components Multi-page session recording — extend beyond 10 interactions to cover complete multi-page user flows like onboarding or checkout funnels
Log in or sign up for Devpost to join the conversation.