EyeTalk

The home page
Calibrating your eyes & head
Making sure your browser is compatible
Selecting the phrase associated with the concept(s) you chose
Selecting a concept

Inspiration

People with conditions like ALS or cerebral palsy often communicate at under 15 words a minute. That's a fraction of normal conversation speed. Real eye-gaze AAC devices exist and work well, but they cost thousands of dollars and require insurance approval and months of waiting. We wanted the same core interaction, speaking with your eyes, using nothing but a laptop webcam and a browser tab.

What it does

EyeTalk tracks your face through the webcam. It learns how your eyes and head move through a quick calibration, then lets you select word tiles just by looking at them and holding your gaze. Pick a couple of concepts and a small language model, running entirely on your own GPU, assembles them into a full sentence and speaks it out loud.

No account. No install. No backend server. No data ever leaves your device. The app makes one network request total: a one-time download of the AI model weights, which your browser then caches for every future visit.

How we built it

Face tracking: MediaPipe FaceLandmarker running client-side. It extracts iris position, head pose, and blink state from 478 facial landmarks in real time.
Calibration: a hand-written ridge regression model, no ML library, that learns a per-user mapping from gaze and head features to screen coordinates.
Smoothing: a One Euro filter removes tracking jitter without adding noticeable lag.
Selection: a dwell-based state machine with blink-cancel gestures, so every control is reachable by gaze alone.
Sentence generation: Transformers.js runs a quantized language model directly in a Web Worker via WebGPU, falling back to WASM on unsupported browsers.
Speech output: the Web Speech API.

The full pipeline: webcam feeds MediaPipe, MediaPipe feeds our ridge regression and One Euro filter, that feeds dwell selection, dwell selection feeds the on-device LLM, and the LLM feeds speech output. No backend anywhere in the loop.

Challenges we ran into

The hardest bug lived in the calibration math, not the tracking itself. The cursor worked fine sitting still. The moment you turned your head, it flew off screen.

Here's why. Calibration tells the user to keep their head still, so head-pose features have almost no variance during capture, just landmark jitter. Standardizing each feature by its own observed standard deviation scaled that jitter up to look like real signal. A real head turn then showed up as tens of standard deviations from anything the model had seen, and a quadratic term amplified that into a wild prediction.

The fix: add a physical scale floor to the standardization instead of relying only on observed variance, and clamp extreme values. A frozen head now produces a near-zero signal the model correctly ignores. A real head movement stays bounded and sane.

We also had to be honest about the AI layer's limits. The on-device model is small, and roughly one generation in four comes out clumsy or off-topic. Instead of hiding that, the app labels generated sentences and falls back to simpler templates when the model can't produce something good. We wanted the demo to stay honest about what actually works.

Accomplishments that we're proud of

A fully client-side AI pipeline. Face tracking, gaze estimation, and sentence generation all run on-device, with zero backend and zero user data transmitted anywhere.
Hand-writing the calibration math from scratch, including diagnosing and fixing a subtle statistical bug under time pressure.
Building a fallback system so the product degrades honestly instead of breaking silently.

What we learned

How appearance-based gaze estimation actually works, and why it has a real, published accuracy floor even in research systems.
Why visible honesty matters in an AI product. Silent failures erode trust fast.
How to run meaningful ML inference entirely in-browser using WebGPU.