Visionary Navigation: An Eye-Mouse Control System

Inspiration

There exist many people with physical limitations, neurological conditions, arthritis, and other repetitive strain injuries (RSIs) that prevent them from using their hands or wrists, and by extension, prevent them from navigating the web on a computer or laptop. Our proposed control system has far-ranging applications and has potential to help such disabled individuals. By using their eyes to control the mouse, our control system enables them to use technology more easily and independently, ultimately allowing for greater access.

What it does

The eye-mouse control system allows users to move their cursor in four directions (left, right, up, and down) by simply moving their eyes to the desired position. Additionally, the system enables users to perform mouse clicks by tilting their head, eliminating the need for hands or wrists.

How we built it

The eye-mouse control system was built in two main steps. First, we developed three predictive models using Convolutional Neural Networks (CNNs) and Deep Learning in Python, specifically utilizing TensorFlow with the Keras API. The first model predicts the user's head position (ie. left tilt, right tilt) in degrees, the second model predicts the user's vertical pupil gaze (ie. looking up, looking down) in degrees, and the third model predicts the user's horizontal pupil gaze (ie. looking right, looking left) in degrees. These models were trained on a vast dataset of images showcasing ethnically diverse subjects in different head and eye gaze positions, all derived from the Columbia Gaze Dataset. Secondly, we coded a script in JavaScript to capture user images via a web camera at regular intervals of 0.5 seconds. The script then fed each image into the three models to predict the user's gaze, updating the mouse's coordinates accordingly and performing clicks when the model detects a head tilt.

Challenges we ran into

Designing and training the predictive models required meticulous fine-tuning to achieve high accuracy levels, and training a NN on thousands of images is very computationally expensive, so we had to walk the fine line of computational cost and accuracy given hardware limitations.

Additionally, in order to obtain accurate gaze predictions, we had to look at the subject’s pupils, so we had to crop the input image to only a picture of the user’s face (so pupil features can be more readily extracted) using Haar Cascades/Viola-Jones Algorithm for object detection.

Additionally, integrating the models with the web camera feed and ensuring real-time performance demanded careful optimization. Overcoming technical complexities and aligning the system's responses with the user's intentions were among the primary challenges faced.

Accomplishments that we're proud of

Achieving a validation accuracy of 90% for head position prediction, 89% for vertical gaze prediction, and 85% for horizontal gaze prediction stands as a significant accomplishment.

Additionally, in our demo video, we were able to successfully navigate Amazon.com and add our favorite item ‘to cart,’ exemplifying the practical uses of the eye-mouse control system!

What we learned

We gained valuable insights and experience in the field of computer vision and machine learning. We honed our skills at data pre-processing methods, data augmentation to create more training images, and learned how to deal with models’ overfitting of data through Dropout regularization, BatchNormalization, etc.

We also learned the theory and implementation behind Haar cascades, as well as the theory and implementation of learning rate schedules (ReduceLROnPlateau) to increase model accuracy.

Additionally, we learned the importance of considering accessibility in technology and how innovations like their eye-mouse control system can make a positive impact on the lives of individuals facing physical limitations.

What's next for Visionary Navigation: An Eye-Mouse Control System

Refinement and Optimization: We may continue refining and optimizing the predictive models to improve accuracy and responsiveness. By collecting more diverse data and fine-tuning the algorithms, we can enhance the system's performance and make it even more efficient and user-friendly.

Expansion of Accessibility Features: The eye-mouse control system has the potential to serve a broader range of users with various disabilities. Future iterations of the system could incorporate additional accessibility features, such as voice control integration, to provide an even more inclusive experience for users with different needs.

User Feedback and Iterative Development: Collecting feedback from users, especially individuals with disabilities, will be crucial for identifying areas of improvement and understanding specific user needs. Continuous iterations based on user feedback will be essential to make the system more effective and user-centric.

Security and Privacy Enhancements: As with any technology involving user data, ensuring robust security measures and safeguarding user privacy will be paramount. Continuously enhancing these aspects will build trust among users and stakeholders.

Built With

• cnn
• cv2
• jupyter
• keras
• pyautogui
• python
• tensorflow