Wink Wink Fruit

Inspiration

We wanted to focus on accessibility for users who do not have full use of their physical mouse and keyboard.

What it does

The system tracks your pupil movements in real-time to seamlessly map your gaze to precise pixel coordinates on the screen. It also translates facial gestures into physical actions, allowing users to perform left or double-left clicks using deliberate winks or simple head tilts.

How we built it

We engineered a custom pipeline that uses computer vision to extract eye-relative gaze coordinates, applying piecewise affine mapping and Kalman filtering for smooth, accurate cursor movement. We also implemented a robust gesture state machine that relies on Eye Aspect Ratio (EAR) calculations and configurable safety lockouts to process clicks without triggering false positives.

Challenges we ran into

Achieving smooth, jitter-free cursor movement was difficult and required extensive tuning of our signal smoothing algorithms and geometric projections. We also had to design strict validation schemas to reliably differentiate between intentional command gestures—like a deliberate wink—and natural, involuntary blinking.

Accomplishments that we're proud of

We successfully built a rigorous, independent validation pass (Stage 3) that guarantees high precision and measures median pixel error before securely saving a user's calibration profile. We are also incredibly proud of our comprehensive test suite, which boasts over 140 automated tests ensuring total reliability across gaze geometry, symmetric covariance, and UI flows.

What we learned

We discovered that accommodating the nuances of human facial geometry requires a delicate balance between strict safety thresholds and continuous, dynamic signal filtering. Additionally, we learned the critical importance of robust hardware state management, particularly when handling camera compatibility checks and persistent calibration profiles.

What's next for Wink Wink Fruit

We plan to introduce full yaw and pitch compensation to further refine gaze accuracy across a wider range of natural head positions. Future updates will also explore expanded gesture vocabularies, potentially enabling seamless click-and-drag functionality or intuitive scroll controls.