Inspiration
Hoot started with a simple question: what if using a computer did not require a mouse, keyboard, or voice?
Eye tracking already exists, but most systems try to turn your eyes into a mouse cursor. That creates a precision problem: clicking small buttons and navigating a dense desktop UI requires very accurate tracking, which usually means specialized and expensive hardware.
I wanted to approach the problem differently.
Instead of making eye tracking more precise, I asked: can I redesign the interface so precision matters less?
That led me to Hoot — a hands-free macOS assistant where a user can communicate intent through a few large gaze targets, while an AI agent handles the low-level desktop interaction.
What it does
Hoot uses a normal webcam to estimate where the user is looking.
Rather than asking the user to point at tiny UI elements, Hoot presents four large semantic choices based on the current screen and what the user may want to do. The user selects an option by looking at it, and the choices become more specific over several rounds until Hoot understands the intended action.
I think of this almost like Akinator for computer intent: instead of trying every possible action, the system keeps asking the most useful next question until it can confidently infer what the user wants.
Once the intent is clear, the user confirms it and a computer-use agent carries out the task using normal mouse and keyboard actions.
For potentially consequential actions such as sending, deleting, or purchasing something, Hoot requires an additional confirmation before execution.
How I built it
Hoot is built as an Electron application for macOS.
The system has three main pieces:
- Gaze input — webcam-based eye tracking estimates which part of the screen the user is looking at.
- Semantic decoding — an LLM uses screen context and the user's previous selections to generate the next four useful choices.
- Computer execution — once the user confirms an intent, a computer-use agent performs the desktop task.
One idea that influenced my design was Claude Shannon's work on communication theory. A gaze estimate is a noisy, relatively low-bandwidth input channel, but that does not mean it is useless. If each selection only needs to distinguish between four large regions, every choice still communicates meaningful information.
Rather than requiring pixel-perfect gaze, I use multiple simple selections to progressively narrow a large space of possible intents.
Challenges I faced
The biggest challenge was gaze accuracy.
Webcam eye tracking is much noisier than dedicated eye-tracking hardware. Early on, it became clear that trying to use gaze as a traditional mouse would be frustrating and unreliable.
That limitation ended up shaping the core idea behind Hoot. I moved away from precise pointing and toward large semantic targets that tolerate noisy gaze.
Calibration was another major challenge. Small differences in camera position, screen size, display scaling, and where the user is sitting can noticeably affect predictions, so I built a calibration flow and coordinate correction system around the tracker.
I also had to think carefully about safety. An AI agent that can control the desktop should not interpret an accidental glance as permission to send a message, delete a file, or make a purchase. That is why consequential actions require explicit confirmation.
What I learned
The biggest thing I learned is that accessibility does not always require making an input device more accurate.
Sometimes the better solution is to redesign the interface around the signal you already have.
I also learned that LLMs can be useful for more than generating text. In Hoot, the model acts as a semantic decoder between a very low-bandwidth human input and a huge space of possible computer actions.
Combining gaze tracking, language models, and computer-use agents gave me a way to turn a simple glance into a much richer instruction.
What's next
I would like to improve gaze calibration and test Hoot with more users, especially people who rely on alternative computer-access methods.
I also want the semantic decoder to become more personalized over time so that common actions can be reached with fewer selections.
Long term, the same interaction model does not have to be limited to eye tracking. A blink, switch, head movement, or other low-bandwidth signal could potentially use the same semantic interface.
