ScreenPilot - See it. Plan it. Do it

Inspiration

Every developer knows the loop: run the tests, wait, find the log file, upload it somewhere, copy the deployment command, switch windows, paste it, wait again, check if it worked. Repeat.

We kept asking: why does none of this automation exist in a form that just works — across any app, on any screen, without writing integration code for each tool?

Existing solutions all have the same failure mode. Shell scripts break the moment you leave the terminal. Browser automation (Selenium, Playwright) only works in the browser and needs a custom script per website. RPA tools like UiPath require days of setup and break when the UI changes. CI pipelines can't see your local screen.

The insight that drove this project: a human doing these tasks uses only their eyes and a keyboard. If you give an AI model the same inputs — a screenshot and a goal — it should be able to do the same thing. No APIs, no integrations. Just vision and action.

That question became ScreenPilot.

What it does

ScreenPilot is a screen-native AI agent. You type a one-sentence goal. The agent does the rest.

Give it this:

Run pytest, upload the log to GCS, then deploy ui-navigator to Cloud Run and write a report.

And it will:

Open your terminal window by itself
Run pytest tests/ -v > test_out.log 2>&1
Read and parse the log — pass/fail counts, timing, warnings
Upload the log to Google Cloud Storage
Pause and ask for your confirmation before deploying
Run gcloud run deploy and capture the Cloud Run service URL
Generate a deployment report and copy it to your clipboard
Stop cleanly

How it works under the hood:

Every 2.5 seconds, the client takes a screenshot of your desktop
The frame is compressed and sent over HTTPS to the Cloud Run orchestrator
Gemini vision reads the screen — it sees UI elements, coordinates, modal dialogs, risk signals
Gemini text takes that perception and your goal, then returns a structured action list
The client executes those actions via OS-level inputs (mouse clicks, keystrokes, hotkeys)
After every major action, a VERIFY step re-captures the screen and checks the expected state occurred

Safety is built in, not bolted on:

Every piece of text is scanned for secrets (PEM keys, JWTs, API keys, passwords) before being typed or sent to the cloud — detected secrets are replaced with [REDACTED]
Terminal commands pass through a hard policy gate: allowlisted commands only, permanent blocklist for rm -rf, shutdown, gcloud * delete, curl | bash
Deploys always require explicit user confirmation via a dialog — the deploy never runs without it

When something goes wrong:

If a VERIFY check fails, a bounded recovery playbook fires automatically: try fallback UI elements → scroll → zoom out → switch tab → open in-page search → hand off to the user
After a configurable maximum number of retries, the agent always hands control back with a full summary of what it tried — it never loops forever

It works on any app:

Give it "Open Chrome, go to google.com, and search for Gemini AI 2026" — it will do it, using the same OS-level controls, with no browser automation library

How we built it

The system is split into two parts that talk over HTTPS.

Desktop Client (Python + PyQt6):

mss captures the screen every 2.5 seconds; a frame-diff hash check skips unchanged frames to reduce API cost and latency
pyautogui and pygetwindow execute the physical actions — mouse clicks, keystrokes, window focus — on the real desktop
A two-tier command policy gate (command_policy.py) runs before any subprocess is spawned: an explicit allowlist, a hard blocklist, and default-deny for everything else
redaction.py scans text with 8 compiled regex patterns before any TYPE action fires and before any frame leaves the machine
The Gemini API key is stored in the OS credential store (Windows Credential Manager) via keyring — it is forwarded per-request and never written to disk in plaintext
PyQt6 provides the control panel: live screen feed, scrollable action log, CONFIRM dialog for destructive actions

Cloud Run Orchestrator (FastAPI + Python):

Three endpoints: POST /session/frame (the main loop), POST /session/upload (GCS artifact store), POST /session/plan (task decomposition)
GeminiPerceptionClient runs two stages: perceive() sends the JPEG to Gemini vision and gets back a structured screen description with UI element coordinates; plan() sends that description to Gemini text and gets back an ordered action list as validated Pydantic JSON
FirestoreSessionStore writes an append-only audit log of every action, every VERIFY result, and every recovery attempt — timestamped to the millisecond
GCSArtifactStore handles file uploads with both gs:// and HTTPS public URLs

The server is stateless and deployed live on Google Cloud Run. It never needs the Gemini API key baked in — the client forwards it as an X-Gemini-Api-Key header so rotating the key never requires a redeploy.

The test suite has 548 tests, all offline — no real Gemini API calls, no GCP credentials required. It covers screen capture, JPEG compression, secret redaction, log parsing, command policy, server endpoints, cloud integration stubs, and the full verification loop.

Challenges we ran into

Getting Gemini to return consistent structured JSON from vision Gemini vision is powerful but free-form by default. We needed it to return coordinates, element types, and risk flags in a strict schema every time. The solution was a two-stage pipeline: one call for raw perception (vision only), a second call for planning (text only, using the structured perception output as context). Separating the two responsibilities made both outputs dramatically more reliable than asking one prompt to do both.

Frame-diff without missing critical transitions Skipping unchanged frames is essential for keeping API costs low. But if the threshold is too aggressive, the agent misses a dialog that appeared and disappeared quickly. We tuned the frame-diff hash comparison to err on the side of sending more frames near action boundaries and fewer during idle waiting periods.

Secret redaction that doesn't break legitimate text The redaction patterns need to catch real secrets without over-firing on normal text. An overly aggressive JWT pattern would redact base64-encoded image data. We used pattern anchoring (length gates, structural prefix checks) and tested every pattern against a corpus of normal developer text to keep the false-positive rate at zero in the test suite.

Bounded recovery that never loops forever Early versions of the recovery loop would retry indefinitely. The fix was straightforward in concept but required careful state management: a hard counter per session, a defined ordered playbook, and a guaranteed exit to HAND_OFF_TO_USER with a full attempt summary. The agent always has a way out.

Command policy that's secure without being unusable An allowlist-only policy sounds simple until you realize that pytest, make test, and gcloud run deploy each have dozens of valid argument variants. We handled this with prefix matching on the core command name plus a separate check for blocklisted substrings anywhere in the command string — covering both the "allow known good" and "block known bad" cases independently.

Accomplishments that we're proud of

The server is live on Google Cloud Run — this is a real deployed system, not a localhost demo
548 offline tests covering every component — most hackathon projects have zero tests
Secret redaction as a Day 1 requirement — it was in the architecture before the first line of feature code was written
The confirmation dialog is mandatory for deploys — there is no code path to gcloud run deploy that bypasses the user confirmation. The safety gate is structural, not advisory.
The recovery playbook always has a defined exit — the agent cannot loop forever by design, not by hope
It genuinely works on any app — the same session loop that runs terminal commands can open Chrome, search Google, or interact with any GUI application on the desktop without any code changes

What we learned

Multimodal AI is better at reading UI than we expected. Gemini vision correctly identifies button labels, input field states, and terminal prompt positions without any OCR preprocessing. The hard part was not perception — it was making the output of perception consistently structured enough to act on.

Two-stage pipelines beat one-shot prompts for agentic tasks. Asking one prompt to simultaneously understand the screen AND decide what to do next produced inconsistent results. Splitting into a perceive() stage and a plan() stage — where planning receives a clean structured description rather than a raw image — made the action plans reliably parseable.

Safety features need to be designed into the architecture, not added later. The redaction layer, the command policy gate, and the confirmation dialog were all specified before any feature code was written. Retrofitting security onto an existing action executor is much harder than building the executor around the security contracts from the start.

The unhappy path is where real systems live. Every impressive demo shows the happy path. The recovery playbook — VERIFY fails, fallback fires, scroll, zoom, hand-off — is the part that makes this feel like a real agent rather than a scripted macro. Building it required more careful state design than the entire perception pipeline.

What's next for ScreenPilot

Multi-monitor support — the frame capturer currently targets the primary monitor; extending it to detect and switch between monitors is the next milestone
Screen overlay mode — a transparent PyQt6 overlay showing the agent's detected UI elements and current intent in real time, without blocking the applications underneath
Workflow templates — pre-built one-sentence goals for common developer patterns (PR review cycle, log analysis, service health check) that users can adapt and share
Voice input — replace the typed goal with a Gemini Live audio stream so the user can speak the goal hands-free while continuing to work
Pub/Sub event triggers — start a session automatically when a CI event fires (test failure, deployment alert) rather than requiring the user to manually type a goal