Inspiration
Commercial eye trackers for people with ALS, spinal cord injuries, cerebral palsy, and locked-in syndrome cost anywhere from $300 to $15,000 — yet the core hardware (an infrared camera and an LED ring) is built from about $40–50 in commodity parts, and the underlying algorithm (pupil–corneal reflection tracking) has been public, patent-free academic literature since 1990. Insurance covers fewer than 30% of eligible patients in the US, and in much of the developing world, fewer than 1% of paralyzed people will ever get access to any eye tracker.
That gap isn't a technology problem — it's a business model problem. We wanted to find out how far we could get building the real thing: not a demo, not a toy, but a complete, working Augmentative and Alternative Communication (AAC) system, using only free/open tools and off-the-shelf parts, and give it away.
What It Does
NeuroGaze is a full eye-tracking communication and control platform made of two parts:
NeuroSketch, a browser-based AAC application (no install, works offline) with five selectable control modes — head movement, an "eye joystick," row/column scanning, WebGazer.js-based gaze estimation, and true infrared gaze tracking — so it can meet a user wherever their remaining motor ability sits. Optional IR hardware (a Raspberry Pi Zero 2W, a NoIR camera, and an 850nm LED ring) that tracks the corneal reflection relative to the pupil for head-movement-independent gaze accuracy around 10–25px, competitive with trackers costing 100x more.
On top of tracking, the app provides a phrase board and predictive AAC keyboard for communication, a dwell-to-draw canvas, a gamified rehab/training mode for stroke and TBI patients, and a one-glance emergency SOS screen — all switchable from a single interface, all running locally in the browser with no data leaving the device.
How We Built It
We layered the system from the most accessible tracking method up to the most precise:
MediaPipe FaceMesh (WebAssembly, runs client-side) gives 478 facial landmarks at 30fps, from which we derive iris position, eye-aspect-ratio for blink detection, and head pose (yaw/pitch). On top of that we built Eye Aspect Ratio blink detection, a 1D adaptive Kalman filter (switching between "saccade" and "fixation" parameter sets in real time), magnetic target-assist snapping, and dwell/row-column scanning for click emulation — all handwritten, no black boxes. For higher accuracy we trained a MobileNetV2-based gaze regression model on the MPIIFaceGaze dataset (45,000 images, leave-one-subject-out validation), exported through ONNX to TensorFlow.js so it runs in-browser via WebGL. For the highest accuracy tier, we built a Raspberry Pi IR server in Python (OpenCV + MediaPipe + asyncio WebSockets) that performs true pupil–corneal-reflection tracking and streams calibrated gaze coordinates to the browser at 60fps. The whole gaze pipeline is arbitrated by a priority selector — IR hardware first, falling back through the ML model, WebGazer, eye-joystick, and finally pure blink-scanning — so the app degrades gracefully on whatever hardware the user has.
Everything ships as a single self-contained HTML file for the app and a single Python file for the server, deliberately avoiding build tooling so a caregiver with no technical background can just open it.
Challenges We Ran Into
Most of our hardest bugs weren't tracking-accuracy problems — they were interaction-logic problems that silently broke all input for a paralyzed user, which is a uniquely bad failure mode for this kind of tool:
A const declared inside a conditional block caused a silent ReferenceError that froze the entire app on the calibration screen. An early return in the main input loop for the Draw view accidentally disabled every button and gesture across the whole app whenever that view was active. The drawing canvas measured its own size while hidden (offsetWidth returns 0 for hidden elements), so every draw coordinate check evaluated against a canvas of width zero — nothing could ever be drawn. An overly broad CSS selector cleared active-state styling on unrelated UI tabs, making a fully working feature look completely broken.
The deeper challenge was designing for involuntary eye movement. Eyes never stop moving — microsaccades, scanning, blinking — so any control scheme that treats "gaze on target" as "intent to act" produces constant false positives. Our answer was the dwell-to-press model for drawing (a visible 0.6-second fill-ring before the "pen" presses down) and adjustable dwell/tremor thresholds throughout the app, so intent has to be sustained, not just glanced.
Accomplishments We're Proud Of A complete, five-mode control system that works for users across a wide range of remaining motor ability — not just "eye tracking," but a full fallback ladder from blink-only scanning up to true IR gaze tracking. A $53 hardware bill of materials achieving ~10–25px accuracy, in the same range as trackers costing $2,000–$15,000. A trained gaze-estimation deep learning model, from dataset to browser-deployed TF.js artifact, running entirely client-side. Software that is genuinely usable today, free, and MIT-licensed — not a locked demo. What We Learned Silent JavaScript failures are the most dangerous kind for accessibility tools. A dropped return or scoping bug doesn't just break a feature — for a user with no other way to interact with a computer, it can mean total loss of communication with no error message to explain why. Corneal reflection tracking is a genuinely elegant piece of physics — the fact that the P–CR vector is invariant to head translation, using nothing but an LED and a camera, made clear why this approach has dominated professional eye tracking for decades, and why it doesn't need to be expensive to replicate. Accessibility constraints make you a better engineer generally. Designing so that no accidental action can occur without sustained, deliberate intent is a discipline that would improve almost any interface, not just an assistive one. What's Next Finish building and field-testing the first batch of IR hardware units. Run a small clinical validation study (ALS, spinal cord injury, and other motor-impairment participants) comparing communication speed and usability against participants' current devices. Pursue non-dilutive funding (grants, Kickstarter) to get initial units into the hands of real users, and eventually pursue insurance reimbursement pathways so the device can reach patients at $0 out-of-pocket cost.
Log in or sign up for Devpost to join the conversation.