myEye

Screen Detection
Minecraft Movement Left
Gesture - Fist
Gesture - Open

Inspiration

There's a growing need for more intuitive and accessible web interfaces. Traditional methods like keyboards and mouses are limiting and cumbersome for individuals with disabilities or in situations where hands-free interactions is desired.

What it does

myEye leverages computer vision and AdHawk's eye-tracking software to create glasses that assist you with your digital needs. From basic scrolling to selecting YouTube videos, to playing whole video games, myEye customizations automate and enhance your digital experiences by providing an "invisible" mouse, and keyboard. myEye simulates the mouse through your gaze, and keyboard commands through hand gestures.

How we built it

myEye leverages AdHawk's Mindlink glasses for tracking eye movements (gaze, imu, etc.), and the actual software was built in Python. We are using Adhawk's Python SDK to interface with the device and stream data. Mediapipe and TensorFlow, for the computer vision interface and creating commands from just your hand movements. Numpy for data manipulation and calculations. Pyautogui to map gestures with keybinds, and many more packages to bring this platform together.

Challenges we ran into

The biggest challenge we ran into was transforming the vectors created from the eye-tracking software, into coordinates onto the screen that the viewer is looking at. Since the camera is on the right side of the glasses, and the eye tracking is in the middle, we would have to shift all of our vectors so that it matches the camera, as well as convert the vectors into coordinates. Generating these coordinates was really difficult and involved complex linear algebra calculations to scale, and orient mappings.

Accomplishments that we're proud of

Bypassing Adhawk's secure camera encryption Connecting all components together into a fast and feasible replacement of your digital accessories. We experimented with various API's to implement gaze tracking, screen detection, gesture controls and more. However, we found classic overlays, contouring, and self-serve, on-device ML provided faster processing. In the end, the long math equations, and model setups were so worth it.

What we learned

We increased our knowledge in game development, by aplpying concepts like ray tracing in order to provide accurate gazing for eye tracker. We learned about AdHawk's eye tracking systems, various python libraries and optimization techniques.