P76W26 - Shay P - VisualEyes

Inspiration

Dedicated eye-tracking communication devices cost thousands of dollars, and they still make people type one letter at a time. For someone living with ALS, cerebral palsy or locked-in syndrome, a conversation at that speed loses its jokes, its timing and its warmth. Friends and family slowly drift away, and that isolation is a health risk of its own. We wanted to build something that runs on a laptop people already own, needs no special hardware, and puts the conversation first, with typing as the fallback.

What it does

VisualEyes lets people who can't speak or use their hands hold a conversation using only a laptop webcam. A small head movement steers a pointer, a short dwell selects, and VisualEyes speaks the message aloud, in the user's own voice if they set one up.

Head pointer from any webcam: MediaPipe tracks the face on-device at about 30 fps. A 20-dot calibration takes about 55 seconds and maps head movement to the screen.
Built for imperfect tracking: the app opens on 6 large ready-to-speak phrases. Users select by holding the pointer (1.5 s by default) or a deliberate blink, and targets snap and hold so small wobbles don't cause wrong picks.
Prediction-first typing: a 4×4 keyboard of letter groups with word completion and next-word prediction. A personal phrase store learns every sentence the user speaks, and abbreviation expansion turns one key per word into a full sentence.
AI reply suggestions: Claude suggests whole sentences that match the user's profile and writing style. Only text is sent, never video.
Hears the other side: with "Listen to the other person" turned on, the partner's speech is transcribed and Claude offers replies on the home screen, so answering takes one selection.
Your own voice: "My voice" clones the user's voice from their recordings with ElevenLabs. Without it, the browser's system voice speaks.
Profiles and sign-in: each user has their own words, phrases and calibration. Caregivers can add names of people, an about note and writing samples. Auth0 sign-in syncs settings and the profile between devices.
Works over any app: an Electron overlay opens with a global shortcut, an offline voice command ("gaze type") or by aiming at the top-right corner. It then pastes the sentence into whatever app you were using.
Mouse mode: no camera? The same interface works with hover-to-dwell.

How we built it

Front end: Vite and plain TypeScript, with no UI framework
Face tracking: MediaPipe Tasks Vision (Face Landmarker), running in the browser
Pointer model: ridge regression from head yaw and pitch to screen position. Smoothed based on data of great matches and degrees of error. We wrote both from scratch.
Desktop overlay: Electron
Offline voice commands: Vosk (browser WebAssembly build)
Partner listening: the Web Speech API
AI suggestions: the Claude API (Anthropic SDK), called through a small Node.js proxy
Voice: ElevenLabs voice cloning, with browser speech synthesis as the fallback
Voice and account service: Python, FastAPI and boto3
Sign-in and data: Auth0 and Amazon DynamoDB
Cloud: AWS EC2 for hosting and AWS Secrets Manager for API keys
CI/CD: GitHub Actions deploys to EC2 on every merge to main
Testing: Vitest and pytest

We kept a strict privacy line in the design. Video is processed on the device and never uploaded. Claude and ElevenLabs receive only text, and voice recordings go to ElevenLabs only after the user agrees on a consent screen.

Challenges we ran into

Webcam tracking is noisy. With no pointer on screen, our open-loop pointing error averaged about 2.9° across 7 calibrations. On a 6-button layout that left only 62% of frames on target, and on a 16-key grid only 36%. Instead of fighting that, we designed around it. We show the user a pointer they can steer, use large targets that snap and hold, offer dwell or blink selection, and lead with predictions so the user rarely has to hit small keys.

Partner listening was tricky to get right. The app had to avoid transcribing its own voice, so we mute the microphone while it speaks. We also had to be upfront that Chrome's speech recognition sends microphone audio to Google, which is why the feature is off by default and VisualEyes stores no audio.

Safety mattered too. Anything spoken aloud comes out in the user's own voice, so we added a slur filter that runs before speech.

Accomplishments that we're proud of

100% correct selections in our target tests (with the pointer visible), on both the 6-button home screen and a 4×4 keyboard that included all four corners. That was 2 of 2 testers, with 0 wrong selections and 0 timeouts.
A median of 2.3 to 2.9 seconds to select a button at a 1.5 s dwell
All of this on a standard Windows or Mac laptop with its built-in webcam. No special hardware.
Users can speak in their own cloned voice, not a generic robot voice
An overlay that lets VisualEyes type into any app, not just its own window
A full deployment pipeline: Auth0, DynamoDB and EC2, with GitHub Actions deploying on every merge
Measured, not guessed: sessions can be recorded and replayed offline (npm run replay), and a built-in benchmark reports words per minute, selections per character and errors

What we learned

The gap between blind aiming accuracy and real usability is huge. Once users can see a pointer and correct it themselves, imperfect tracking becomes workable. We also learned that the biggest speed gain comes from reducing how much the user has to type, not from typing faster. Building the replay tool early let us compare tracking changes with data instead of gut feeling. And we learned to be honest about our evidence: all of our numbers come from two able-bodied testers, so they're a starting point, not proof.

What's next for VisualEyes

Testing with people living with ALS and other motor impairments, together with their clinicians
Dwell-free swipe typing
More languages (English only today)
A caregiver mode and emergency phrases
Measuring messages per minute with suggestions versus letter-by-letter typing
Tool calling so they can perform actions such as making a call