Glasses
CAD Glasses Prototype
Branding
Software

Inspiration## Inspiration

We kept coming back to one question: what happens when someone can't use their hands anymore? People with ALS, spinal cord injuries, or MS often lose hand control long before they lose control of their eyes. Eye-tracking setups for them do exist, but the ones we looked at were expensive and came with their own special hardware. Every laptop already has a camera pointed at your face, so we wanted to see how far that camera could get on its own.

What it does

You look at the screen and the cursor follows your eyes. Double blink to click. Wink your right eye for a right click. Hold a left wink to drag something, then open your eye to drop it. The app can also turn your speech into text.

How we built it

It's Python. MediaPipe finds the face and irises in each webcam frame. From that we pull out where each iris sits inside the eye, plus which way the head is turned and how far away it is (we estimate distance from the gap between the eye corners, which is about 9 cm on most people).

Calibration is ridge regression from those features to screen position:

$$W = (X^\top X + \lambda I)^{-1} X^\top Y$$

Raw gaze shakes a lot, so we smooth it with a One Euro filter. It smooths heavily when your eyes are mostly still and backs off when they jump, so the cursor doesn't lag behind big movements. Clicks go through macOS Quartz events, and the UI is PySide6.

We're also building glasses with cameras pointed at each eye and a motion sensor for head tracking. They weren't done in time, so we wrote a fake headset in software whose "eyes" look at the calibration dots. That let us test the whole glasses pipeline with no glasses.

Challenges

Blinks were the first real problem. People blink all the time, and we didn't want every blink to be a click. Short blinks get ignored now. There was a second issue: while your eyes are closing, the gaze reading goes all over the place. So we lock the cursor position the moment the blink starts and click where you were actually looking.

Head tilt is still our biggest weakness. Tilt your head and the iris moves in the camera image even though you're looking at the same spot. Feeding head pose into the model helped a lot, but it isn't solved.

We also lost time to calibrations that silently stopped working after someone changed a camera setting. Now every saved calibration is tied to the exact camera config, and if anything changes, it tells you what changed and asks you to recalibrate.

Then there were setup problems. MediaPipe needed Python 3.11, and the eye-tracking libraries we planned to use don't run on Apple Silicon, so we wrote our own pupil detection.

What we learned

Accuracy matters less than we expected. The webcam is off by a couple hundred pixels, which sounds bad, but with big targets and snapping to the nearest key, typing actually works. How you design the clicks and targets mattered about as much as the tracking.

We also learned to build debugging tools early. When the cursor stopped moving, a report showing which stage broke saved us a lot of guessing.

What's next

Finishing the glasses is first, since cameras right next to the eyes should be far more accurate than a webcam across the desk. After that we want a communication board for people who can't speak, and a proper benchmark so we can measure ourselves against other systems with real numbers.