Inspiration
Imagine needing a glass of water but having difficulty speaking or using a keyboard. Expressing a simple everyday need may require significant effort or assistance from another person. We wanted to explore a way for people to communicate using head movements, with nothing but the camera already in a laptop, and make it easier to express everyday needs.
What it does
Four Looks lets someone type and speak with only a laptop camera. Head movement moves a cursor across an on-screen keyboard. Holding still on a key, or smiling, presses it. The app guesses the word you meant, so most words take two or three presses. The speak key reads the sentence out loud.
The keyboard sits along the bottom of the screen and never takes focus, so text goes into whatever Mac app is in front, such as a Gmail draft. In mouse mode, head movement moves the real Mac cursor, and a quick open of the mouth clicks.
Grok can take a task from the keyboard. Simple commands like "open gmail" run with no model at all. For anything that sends, buys or deletes, Grok stops and waits for a nod.
How we built it
Seeing the face. We use MediaPipe Face Landmarker in the browser. It gives the head angle and face scores such as smile and jaw open, about 30 times a second. The camera picture never leaves the computer.
Turning the head into a cursor. A six dot calibration maps your own comfortable head range to the screen. A filter smooths small tremor and still follows fast moves. The cursor is less sensitive near the screen edges, where head tracking is noisier.
Pressing a key. You press with a smile or by holding still on a key. Each key fills only while the cursor is on it, and it holds on through small wobbles once it is a third full.
Guessing the word. A two-word language model built from a public set of messages written for people who use communication devices (Vertanen and Kristensson, 2011). It corrects near misses when a word ends, finishes words, and suggests the next one. With presses scattered by a third of a key width, it still gets 97.5% of words right on 1,048 test words.
Fixing spelling. A misspelled word used to bring up nothing useful. We added dictionary suggestions one or two edits away, ranked by the word before. On In A's eleven test words, the right word is now the top guess for all eleven. An optional setting lets Grok pick the word that fits the whole sentence.
The Mac app. An Electron window that cannot take focus, so typing goes to the app in front. A small Swift helper moves the real cursor and reads the names of buttons on screen.
Grok. Simple commands run with no model. For harder tasks, Grok gets the names of the buttons in the front app and looks at a screenshot only when it has to. It asks before anything that sends, buys or deletes, refuses passwords and card numbers, and the app stops spending at one dollar.
Testing without a face. We wrote a simulated face that shakes and jumps like a bad camera, and a suite of 14 scenarios that types whole sentences with it. It caught bugs we could not see by hand, such as a diagonal move to the space bar being read as a nod.
Tools. Plain JavaScript, no framework. We used AI coding help from Claude Code and Codex.
Testing and results
All of these come from one participant in an initial test, so they are a first look, not a conclusion.
- Completion. The participant completed all 12 target sentences.
- Speed. Across the 12 first trials the mean displayed speed was 6.0 words a minute, ranging from 3.3 to 7.8. Repeating "I need water" reached 12.7 words a minute, against 3.3 on the first attempt. Practice or familiarity with the sentence may explain that.
- Word suggestions. Speed was higher with word suggestions for all 12 sentences. The average was 11.7 words a minute with suggestions and 4.9 without.
- Spelling fixes. After dictionary suggestions were added, suggestion accuracy on misspelled words rose from 0% to 60% in the participant's test, and average speed rose from 12.6 to 14.8 words a minute. Getting used to the app may explain part of that.
More participants and repeated trials under the same conditions are needed.
Challenges we ran into
- Small head moves are noisy. Smoothing enough to steady the cursor made it lag, and people overshoot a lagging cursor. We now smooth only small moves and hold a key once it starts to fill.
- Gestures look like ordinary movement. Three blinks woke Grok by accident, and moving down to the space bar looked like a nod. We replaced the blinks with a frown the app learns from your own face, and a nod now needs the head to be at rest first.
- The word list had junk in it. It knew "dor" and "bok" as words, so no spelling fix was offered. In A's test words exposed this.
- Giving an AI the keyboard is risky. In one test it pressed Allow on a system prompt. It now stops when a system window is in front and waits for a nod before anything that cannot be undone.
Accomplishments that we're proud of
- A word guesser that gets 97.5% of 1,048 test words right when presses are scattered.
- In a simulated typing test, wrong letters in a 55 key sentence dropped from one to three down to none.
- The right spelling is the top guess for all eleven of In A's misspelled test words.
- A 14 scenario test suite that types whole sentences through a simulated bad camera.
- Grok runs with a one dollar spending cap and refuses passwords and card numbers.
What we learned
- Testing with a simulated face finds bugs that hand testing misses.
- A word list is only useful if it is clean. Junk entries quietly block fixes.
- An AI agent that controls the keyboard needs hard stops on system windows and on anything that cannot be undone.
What's next
- Test with people who cannot speak or use their hands, not only with our own tests.
- Compare selection counts and completion times with and without suggestions. The app now counts selections, so this can be measured.
- Try eye-corner input for people who cannot move their head. One early test with one person was not clean.
- Add the Ability Scan, so each person sets up their own signals.
- With enough data, fine tune a mistake fixing model and evaluate it on a separate test set.
