Glance

Inspiration

Every AI assistant makes you start from zero. You take a screenshot, copy and paste, and explain what you were just looking at, even though it's right there on your screen. We pictured Aoife, a first-year student with ten laptop tabs open, a budget in her Notes and a course app she's never used. Her Mac already shows everything she needs. It just can't see it. We wanted help that starts from what you're looking at, from an app you can trust to watch your screen.

What it does

Glance is a Mac menu-bar app. Point at anything on your Mac, ask out loud, and get an answer.

Point and ask. Press ⌥Space, drag a box over anything in any app, then hold ⌥Space and speak. The full answer streams into a chat panel, and a short version is read aloud.
Memory across apps. Glance keeps a rolling 10-minute memory of the windows you've used, read with on-device text recognition and stored only on your Mac. On a third laptop page, ask "How's this different from the earlier ones?" and it compares all three against the budget you typed into Notes.
Guide. Say "Show me how to export this as a PDF" and Pip, Glance's 8-bit penguin, flies to the right menu item and points at it. As you open each menu, Pip moves on to the next item. Glance never clicks for you.
Private by design.
- Only the box you select is captured.
- Card numbers, IBANs, PPS numbers, passwords, keys, emails, addresses and more are hidden before anything is sent, and blacked out in the image.
- You see exactly what will be sent. If something high-risk was hidden, Glance asks before sending.
- Password managers, bank, payment and login pages, private browser windows and password fields are never recorded.
- You can pause memory or forget the last 10 minutes from the menu bar.
- Local-only mode keeps everything on the Mac (a local model or Apple's on-device model, Apple speech recognition and the Mac voice) and blocks every other network request.
Your choice of AI: Claude, OpenAI, DeepSeek or a local model, with ElevenLabs for voice.

How we built it

A native Swift app built with AppKit and SwiftUI.
Seeing the screen: ScreenCaptureKit captures only the selected region (or, for memory, only the focused window). Apple Vision reads the text on-device, and the Accessibility API supplies the app, window, URL and menus.
Memory: SQLite with full-text search, secure delete and rolling deletion. Its folder is excluded from Time Machine.
Privacy: the redactor validates checksums (Luhn for cards, mod-97 for IBANs, the PPSN check letter) and uses Apple's NaturalLanguage framework to spot names. A single function is the only way screen content can leave the Mac, and every network request passes through one gate.
Voice: ElevenLabs speech-to-text and streamed text-to-speech, falling back to Apple's on-device speech and the Mac voice. Each answer opens with one or two spoken sentences, and the full answer appears on screen.
AI: one provider interface with a Claude implementation and an OpenAI-compatible one (OpenAI, DeepSeek, Ollama, LM Studio). Text-only models get the recognized text instead of the image.
Guide: the model receives a list of the app's menus and controls plus a redacted screenshot, and replies with one step. Glance matches that step to a real on-screen element and follows menu paths locally, with no extra model call per menu.
We built in gated phases so there was always a working demo, using Claude Code with parallel chats on separate branches and an audit pass that found and fixed 22 issues. A built-in self-test mode runs over 300 automated checks.

Challenges we ran into

No Xcode at the start. Apple's Command Line Tools can't compile SwiftUI's macros, so the whole UI uses the older ObservableObject pattern.
Permissions reset on every build. Each unsigned rebuild changed the app's signature, so macOS revoked screen recording and accessibility access. A self-signed certificate fixed it.
An OCR flake. If Apple Vision's text recognition loads after a NaturalLanguage call, it fails and stays broken until the app is relaunched. We now warm it up at launch.
Memory that went missing. Spoken questions without a selection got no memory at all, and keyword matching dropped the budget note because it shared no words with the laptop specs. Every question now carries the recent activity.
CPU cost. Reading the focused window's text every second costs a real share of a CPU core. We capped the capture size, stop while the screen is locked or asleep, time out unresponsive apps, and slow down in Low Power Mode.
Three coordinate systems. Cocoa, Accessibility and Vision each measure the screen differently, and Pip has to point at the exact button, even across several displays.
Knowing what's private. Detecting private browser windows and password fields reliably is hard. When in doubt, Glance skips the window.

Accomplishments that we're proud of

An answer that combines three web pages and a note from another app, with no copying or pasting.
Privacy enforced in code rather than promised. A self-test fails if any networking code appears outside the network gate, and local-only mode blocks other hosts before a connection opens.
The mock bank page demo: the card, IBAN, PPS number, email and password are all blacked out in the preview, and Glance asks before sending.
Pip, an 8-bit penguin drawn pixel by pixel in code, who reacts while listening, thinking and talking, and flies over to whatever it's explaining.
Over 300 automated self-tests, including one that renders the mock bank page and checks that every sensitive field is hidden.

What we learned

Privacy has to be part of the design. Exclusions run before anything is read or stored, not afterwards.
Speak the short version and show the long one. Two spoken sentences feel fast, and nobody wants a paragraph read aloud.
On macOS, permissions are tied to the code signature, and each system framework uses its own screen coordinates.
Build a thin end-to-end version first. With point, ask and answer working early, every later phase added to it instead of rebuilding it.
Redaction will never be perfect, so it needs layers behind it: exclusions, a preview, a confirm step and local-only mode.

What's next for Glance

Guide v2: watch the screen after each step, catch a wrong click and correct it ("That opened Share. Close it and click File instead"), with Why? and Skip buttons.
Do mode: with your approval, save a comparison as a table with links to the sources.
"Where was I?" recaps of what you were working on.
Better redaction: test it on much more real-world data. Single first names and unusual address formats are known gaps.
Real users: put Glance in front of students and new employees who are learning unfamiliar software.