Lookie

Inspiration

Working with AI still means taking screenshots, switching apps and typing long explanations of what we mean. Between people it's simpler: you point at something and say "fix this". We wanted AI to work like a colleague sitting next to you.

What it does

Lookie is a desktop assistant for Windows that knows what you are looking at.

Look and ask: a regular webcam tracks your gaze (or use the cursor). Hold one key, ask out loud or type, and Lookie answers about the exact thing you were looking at: a chart in a PDF, a line of code, a file, a web page.
It acts, with your permission: Lookie can move, copy, rename and create files and folders, type text, click and edit files. Every action appears in a small card with Allow / Deny and can be undone.
Code: with the VS Code extension it reads the code you're looking at, highlights the exact lines, explains errors and proposes fixes.
Personal: an "About you" profile (role, field, preferred answer length) shapes every answer. Answers come in your language, including Lithuanian.
Context: you can attach files to a question, and Lookie remembers recent activity ("where did I just move that file?").

How we built it

Desktop app: Electron + React + TypeScript, a transparent click-through overlay, global hold-to-ask key.
Eye tracking: webcam-only gaze tracker (MediaPipe face landmarks + a neural/iris model, built on WebEyeTrack), a 9-dot quick calibration (~35 s) that is saved and re-checked with one dot. We tuned the gaze bubble (One Euro filter + spring motion) by replaying 34 of our own recorded calibration sessions instead of guessing.
AI: the overlay sends the region you looked at plus screen context to Claude (with Gemini/OpenRouter/custom models as automatic backup), which returns an answer, the target it understood and, if needed, an action plan that the app validates before asking you. Voice via ElevenLabs.
Integrations: a VS Code extension and an MCP server so Lookie also works inside connected AI chats.
Process: we built with AI coding agents (Claude Code, Codex) running in parallel git worktrees, with written specs, independent reviewer agents, ~3,900 automated tests and local pre-push checks.

Challenges we ran into

Webcam gaze is imprecise (often 100–200 px off). We stopped treating it as a point: Lookie treats it as a region, combines it with the screen and your question, and always shows what it understood.
Jitter vs. responsiveness: the raw tracker wanders even when your eyes are still. Getting a bubble that is calm but still follows you took several measured iterations.
Safe computer control: an AI that can click and type is risky. Every step needs confirmation, nothing is deleted without undo, passwords are never typed, and text on the screen is treated as data, never as instructions.
Windows details: overlay focus and click-through, display scaling and coordinate spaces, and keyboard layouts (Ctrl+Alt is AltGr on a Lithuanian keyboard).
Team coordination: three people and several AI agents in one repo. One silent default change cost us hours of testing, so we introduced owned areas, announced default changes and automatic checks before every push.

Accomplishments that we're proud of

Real gaze-driven questions with just a laptop webcam: in one recorded run Lookie picked the intended target 11 out of 11 times across diagrams, formulas, code, a terminal and a browser.
A full "look, ask, approve, done, undo" loop for actions on your own computer.
A product that feels calm: one floating island, a soft gaze bubble, small permission cards instead of chat walls and clutter.

What we learned

Treat uncertain input as uncertain and show it to the user; that builds more trust than pretending to be precise.
Measure on real data before tuning anything.
AI agents are fast when tasks are small, specified and independently reviewed, and chaotic when they're not.
Clear communication about who changes what matters as much as the code.

What's next for Lookie

Multi-step tasks that look at the screen again after every step.
macOS release, easier install, and testing with many more people (glasses, lighting, different cameras).
A browser extension for exact page text, and app connectors (e.g. email) instead of clicking through interfaces.