Squint

Inspiration

Browser agents work by looking at your screen. That means a checkout page, an inbox or a banking tab goes straight to someone else's server, including your face, your card number and your ID. We wanted to know whether a vision model really needs to see any of that to do the job. A model only needs to know where the "Place order" button is and that a card field exists, not what the card number is.

What it does

Squint puts a privacy layer between your screen and the vision model.

1. It captures the page and finds sensitive regions on your device: faces, emails, phone numbers, card numbers, Aadhaar numbers and addresses.
2. It overwrites those pixels with solid fills (not blur, so nothing can be recovered) and numbers each region, like [3:card].
3. Only the redacted image goes to the model, with region labels and positions and no values.
4. The model decides the next action: click this element, type into that field. When it needs your data, it writes a placeholder such as {{card}}. Your machine fills in the real value locally, so the agent can complete a form it has never seen the contents of.

The demo UI shows three views side by side: the original (private), the redacted version, and exactly what the server received.

How we built it

• On-device pipeline (Python): Playwright captures the page. Tesseract reads text at 2x upscale. Regexes run on joined lines, because OCR splits 4111 1111 1111 1111 into four separate boxes. MediaPipe BlazeFace finds faces on overlapping tiles. Non-sensitive digits such as expiry dates and order numbers are deliberately left alone.
• DOM hints: when the page is inspectable, element geometry and input attributes (type=email, autocomplete, field names) catch names and other fields that pattern matching can't. Pixel mode remains the fallback for canvases, images and remote desktops.
• Server: a FastAPI endpoint sends the redacted screenshot to a Groq-hosted vision model with a forced tool call, so the output is a schema-valid action. A prompt explains the redaction scheme to the model.
• Benchmark: a generator renders fake pages (checkout and receipt templates) with Faker and Jinja2, randomizing font, theme, zoom and layout. Playwright reads the position of every tagged PII element, so the ground truth is free. We scored 200 screenshots at IoU ≥ 0.5.

Results (200 synthetic screenshots, pixel-only, CPU laptop)

Recall is matched at IoU ≥ 0.5 (faces: detector box inside the photo frame). "Area covered" is the share of ground-truth boxes with at least 90% of their pixels redacted, whatever the label, so it measures what actually got hidden.

The five structured types average 97.7% recall at 99.4% precision, in about 0.53 s per screenshot with no cloud calls for detection. Faces are the weak spot in pixel-only mode. BlazeFace is built for large faces, so small avatars (under about 96px) are often missed, and in a breakdown on a 20-image subset the found faces were mostly the larger photos.

Ablations:

• OCR scale: 1x is 44% faster (295 ms) but loses about 6 points of phone recall. 3x costs 52% more time (798 ms) and is no more accurate. We chose 2x.
• Face tile size: in a 20-image test, 192 px tiles found every photo of 96px and up, 256 px was worse, and 128 px was much worse.
• Confidence threshold: recall stays flat up to 0.7 and then collapses at 0.8 (72%). We redact at no threshold because a missed secret costs more than an extra black box. Junk boxes come to 2.4%.

Challenges

• OCR misread our own card number. The fix was a 2x upscale plus a fuzzy fallback pattern for number groups that don't parse cleanly.
• The vision model gave wrong pixel coordinates: it said the right thing to click but pointed at the wrong place. We replaced coordinates with numbered elements (E1, E2, ...) drawn on the screenshot, and the model returns a number.
• Faces: BlazeFace is built for large faces, and small avatars were missed. Tiling helped for larger photos. We also found our own metric was unfair, since the detector boxes facial features and our ground truth boxes the whole photo, so face matching is by containment and we also report area covered.

Limitations (we'd rather say them)

• Faces: only about 48% of faces are redacted correctly from pixels alone, and only 37% are fully covered. Small avatars are the main failure. A stronger small-face detector is the first thing we'd improve.
• Names can't be found by pattern, so in pixel-only mode they stay visible. The DOM-hint mode covers them, and a learned classifier is the obvious next step.
• The benchmark is synthetic and cleaner than real sites, and our address pattern is tuned to its format, so expect lower numbers in the wild.

What we learned

Measuring changed our decisions more than guessing did. The OCR-scale and threshold sweeps picked our defaults, the face breakdown by photo size showed where the detector breaks, and the metrics found a flaw in our own evaluation.

What's next

A browser extension that talks to the local service, so the agent works in your own tabs. A small learned classifier for names. A stronger small-face detector.

Built With

• css
• faker
• fastapi
• groq
• html
• httpx
• javascript
• jinja
• matplotlib
• mediapipe
• numpy
• opencv
• pillow
• playwright
• pydantic
• python
• tesseract
• uv
• uvicorn