Inspiration
Computers are starting to point with your eyes. Apple Vision Pro selects whatever you look at, and smart glasses are heading the same way. That works when your eyes hold still. For people with nystagmus, whose eyes move on their own, it often doesn't: about 24 in 10,000 people have it, in one UK survey.
Zoe Knox, who has nystagmus, wrote that Vision Pro's eye pointing ran into trouble with "many selection targets close together," and that its eye-mode pointer "definitely shows the jitter that the eye tracking sees in my eyes." The head pointer was "easily the most stable" option, but "more fatiguing."
I have nystagmus too, and it's one of the reasons my eye exams take hours. This project began as eye tracking and didn't work on me, so I switched to the head. That points at the real problem: head pointers are steady, but tiring. Head pointers aren't new: Camera Mouse has been downloaded over three million times, and Google built Project Gameface, where you click by raising your eyebrows. Noddle asks a narrower question: can small, gentle head movements, with no install and no special hardware, handle something as dense as a 225-square Scrabble board?
What it does
Noddle lets you play Uno and Scrabble with your head through a laptop webcam. Turn your head to move the pointer, nod to pick, and shake to put a tile back. You don't need your hands, an install or a headset. Face tracking runs in the browser, and video never leaves your device.
Dense boards zoom in around where you're aiming, and the pointer snaps to legal targets, so aim doesn't have to be exact. Noddle adapts to how each person nods, big or subtle, and adjusts as they relax or tire. A short lesson teaches the gestures with animated pictures, because testers didn't read written instructions. You can play against bots, or in multiplayer rooms where each player uses their own laptop.
How I built it
MediaPipe's Face Landmarker runs in the browser, and Noddle takes head turn and tilt from its transformation matrix, then smooths and maps it to a pointer. The nod and shake detector is custom, tuned on recordings of real play. Recordings keep head angles, the pointer and every pick, never video.
The Uno and Scrabble rules are seeded engines that run the same way in the browser and on the server. The Scrabble bot uses Appel-Jacobson move generation over a 172,823-word list. The app is SvelteKit, multiplayer runs on Cloudflare Workers and Durable Objects, and it was built solo with AI coding agents.
Challenges I ran into
A nod moves your head, and your head is the pointer, so every nod tries to throw the pointer off its target. Noddle holds the pointer through the nod and uses what was selected when the nod began. That caused its own bug: nodding while still sweeping across the rack picked the tile just passed. Reaching for G got I.
Replay scores misled. One smoothing change scored far better in replay (33 ms of catch-up against 1200+) and made real play much worse, so it was reverted. After that, nothing counted as fixed until someone had played it.
Calibration didn't help. Fitting the pointer to each person's picks did nothing, because people steer the visible pointer onto the target before nodding, so the picks reveal nothing about the mapping. Hiding the pointer to get clean data made the game look broken, so that went too.
Accomplishments that I'm proud of
A full game of Scrabble, played with small head movements on a normal laptop, by someone whose eyes won't hold still.
The results are measured, and reported honestly. Across 7 hand-labeled sessions, 159 of 163 picks were right: a best case, on the scripted demo, by people who knew it. On Scrabble rack picks, the pointer sat a median of 0.6 degrees of head turn from the tile's center.
What I learned
Head input can be gentle. You don't have to bob your head like a horse: about 3 in 10 recorded nods were under 4 degrees and still worked. Most recordings so far come from one person, so this needs more testers.
Changing the screen usually beat tuning a threshold. Moving Play to a corner did more than rounds of tweaking. And people don't read, so the lesson teaches with pictures.
What's next for Noddle
Testing with more people who have nystagmus, and with people who can't rely on their hands. Measuring whether subtle nodding really is less tiring, since today that's only how it feels.
Then taking head input beyond board games, to everyday web apps, smart glasses and AR headsets, so people whose eyes don't hold still aren't left out of the devices that point with them.
Built With
- computer-vision
- onnxruntime
- svelte
- sveltekit
- typescript
