Autopilot UI

Navigate. Automate. Guard. Any UI, any app.

Comment

Inspiration

The idea for Autopilot UI came from a real frustration every developer knows. You push a code change, all your tests pass, CI is green — and then a user messages you saying the login button has disappeared. Or your checkout page is broken on mobile. Or the navigation links are gone. We realized that every testing tool today checks the code — but nobody was checking the screen. Selenium checks if elements exist in the DOM. Percy compares pixels. Neither one understands what a user actually sees or experiences. That gap is where Autopilot UI was born. We asked: what if an AI could look at your screen the way your users do — and tell you exactly what's broken, where it is, and why it matters?

What it does

Autopilot UI is an AI agent powered by Gemini 2.0 Flash Vision that sees your UI the way your users do. It does three things: Navigate — Give it a screenshot of any website and describe what you want in plain English. "Click the Sign In button and fill in the email field." Autopilot UI analyzes the screen, identifies every interactive element with bounding box coordinates, and executes the action — showing an animated cursor moving to the exact position on screen. Automate — Run multi-step workflows across any application without writing selectors or test code. The agent observes, plans, acts, and verifies — step by step — using Gemini Vision to understand each screen state. Guard — Upload a baseline screenshot (approved UI) and a current screenshot (after your latest deploy). Autopilot UI compares them semantically and returns a structured regression report — severity level, affected element, plain English description of what broke, exact bounding box coordinates, user impact, and an overall UI health score from 0 to 100. Instead of "diff detected at coordinates (847, 203)", you get "Sign In button is missing from the navigation bar — users cannot log in."

How we built it

Backend — FastAPI + Google GenAI SDK Built a Python FastAPI server with three endpoints: /analyze for single-shot UI navigation, /run for autonomous multi-step agent loops using Server-Sent Events for real-time streaming, and /compare for visual regression detection. All Gemini calls use the Google GenAI SDK with gemini-2.0-flash multimodal — screenshots are sent as base64-encoded images alongside carefully engineered prompts that instruct Gemini to return structured JSON with action types, coordinates, severity levels, and reasoning. Frontend — Vanilla HTML/CSS/JS Built a zero-dependency single-file frontend featuring drag-and-drop screenshot upload, live screen capture via getDisplayMedia(), an animated agent cursor that moves to Gemini's exact coordinates, bounding box overlays drawn proportionally on screenshots, a real-time action log, reasoning panel, and execution timeline. The SSE stream from the backend drives every UI update in real time. Gemini Prompt Engineering The core of the project is the prompts. For navigation, we instruct Gemini to return percentage-based bounding boxes (0–100% of image dimensions) making coordinates resolution-independent. For QA, we engineered a senior QA engineer persona prompt that distinguishes critical regressions from minor noise — filtering out sub-pixel shifts and shadow changes while catching layout breaks and missing elements. Deployment — Google Cloud Run Containerized with Docker, deployed to Cloud Run with a single gcloud run deploy command. Fully serverless, auto-scaling, and publicly accessible with no infrastructure to manage.

Challenges we ran into

Getting Gemini to return consistent JSON — Gemini occasionally wraps responses in markdown code fences despite being explicitly told not to. We built a robust stripping layer using regex to clean the output before JSON parsing, with proper error handling that surfaces the raw response for debugging. Percentage-based coordinates — Early versions used pixel coordinates, which broke whenever the screenshot was displayed at a different size than it was captured. Switching to percentage-based bounding boxes (0–100% of image dimensions) made the system resolution-independent and the frontend overlay math simple. CORS on Google Cloud Shell — During development on Cloud Shell, requests between port 3000 (frontend) and port 8000 (backend) were blocked by CORS. The solution was serving the frontend as static files from the FastAPI backend itself, eliminating cross-origin issues entirely. SSE streaming on Windows — uvicorn with --reload on Windows PowerShell required using python -m uvicorn instead of the direct command, and the .env file needed explicit UTF-8 encoding to avoid a codec decode error. Semantic vs pixel diff balance — Tuning the Gemini QA prompt to ignore minor noise (1–2px shifts, anti-aliasing, shadows) while catching real regressions (missing elements, broken layouts, wrong colors) required multiple iterations to get the severity classification right.

Accomplishments that we're proud of

Real semantic understanding — Autopilot UI doesn't just find differences — it understands meaning. It knows that a gray button that should be blue "looks disabled to users" and that a missing nav link "prevents users from accessing their orders." This level of contextual understanding is genuinely novel in visual testing. Zero setup for any UI — The agent works on a screenshot of literally any application — web, mobile, desktop, any framework, any tech stack. No selectors, no configuration, no code changes to the target app. You just drop in a screenshot. Live agent cursor — Watching the AI cursor smoothly animate to the exact element Gemini identified — with a click ripple effect — makes the agent feel truly alive. Judges and users immediately understand what the agent is doing without reading a single line of output. Production-grade in 3 days — Built, deployed, and demo-ready in under 72 hours on Google Cloud Run with a real Gemini multimodal backend.

What we learned

Multimodal prompting is an art — Getting Gemini to return structured, actionable data from an image requires carefully designed prompts. The persona matters — "you are a senior QA engineer" produces dramatically better regression reports than a generic prompt. Percentage coordinates are the right abstraction — Any system that overlays AI-detected elements on displayed images must use relative coordinates, not pixels. This was a key architectural insight that made the frontend elegant. SSE is perfect for agentic loops — Server-Sent Events let us stream each step of the agent's reasoning — observe, plan, act, verify — to the frontend in real time. This transforms a slow API call into a live, engaging experience that clearly communicates what the agent is doing. The demo is the product — For an AI agent, the visual presentation of what the agent is doing matters as much as the underlying intelligence. The animated cursor, bounding boxes, and real-time log are what make the capability tangible and convincing.

What's next for Autopilot UI

CI/CD Integration — A GitHub Action that automatically runs Autopilot UI on every pull request, screenshots the staging environment, compares against the approved baseline, and blocks the merge if critical regressions are detected. Real browser execution — Integrate Playwright to execute the agent's actions in a real browser, enabling truly autonomous end-to-end testing across live websites. Multi-step QA flows — Allow users to define a sequence of navigation steps followed by visual assertions — "go to checkout, add item to cart, verify the order summary looks correct." Slack and Jira integration — Automatically post regression reports to Slack channels and create Jira tickets with screenshots, severity, and developer recommendations when a deploy introduces visual bugs. Support for video input — Use Gemini's video understanding to analyze screen recordings, catching transient UI bugs that only appear during animations or state transitions.

Built With

Share this project:

Updates

Submission history