Mobile Gaze

Inspiration

The inspiration for the "MOBILE GAZE" project stemmed from the need to make eye-tracking technology more accessible and affordable. Traditional eye-tracking systems rely on expensive specialized hardware, limiting their use to specific fields and professionals. With the widespread availability of smartphones equipped with high-quality cameras and powerful processors, we saw an opportunity to democratize this technology by leveraging machine learning techniques to achieve similar levels of precision without additional equipment.

What it does

The Mobile Gaze project provides a software-based solution for eye-tracking using the selfie camera of a smartphone. It accurately estimates where the user is looking on their device screen in real-time by analyzing the user's eye movements and facial features. This technology enables hands-free control and interaction with the device, enhancing user experience and accessibility. It also offers applications in areas such as user behavior analysis, medical diagnostics, and immersive gaming experiences.

How we built it

We built Mobile Gaze using a combination of advanced machine learning techniques and mobile computing technologies:

• Data Collection: Utilized the MIT Gaze Capture dataset for training the model, supplemented by real-time data collection through a mobile app.
• Model Development: Employed TensorFlow to develop a convolutional neural network (CNN) for gaze estimation. The model processes images of the user's eyes and face to predict gaze direction.
• Preprocessing: Implemented preprocessing steps to extract and normalize features such as eye images, face position, and orientation metrics.
• Real-Time Analysis: Integrated Google MLKit to analyze frames captured by the smartphone camera in real-time.
• Calibration: Developed a two-phase calibration process to fine-tune the model for individual users and adjust predictions based on device-specific characteristics.
• Deployment: Converted the trained model to TensorFlow Lite (TFLite) format for efficient execution on mobile devices.

Challenges we ran into

• Hardware Limitations: Smartphones have limited processing power and lower-quality sensors compared to dedicated eye-tracking hardware, posing challenges in achieving real-time performance and accuracy.
• Calibration: Ensuring accurate gaze tracking required a robust calibration process to account for individual user variations and device-specific characteristics.
• Data Variability: Handling variations in lighting conditions, device orientations, and user interactions required sophisticated preprocessing and model tuning.
• Model Optimization: Balancing model complexity and performance to fit the constraints of mobile hardware while maintaining high accuracy was challenging.

Accomplishments that we're proud of

• Successfully developed a software-based eye-tracking system that achieves accuracy comparable to specialized hardware without the need for additional equipment.
• Implemented real-time gaze estimation on mobile devices using advanced machine learning models and preprocessing techniques.
• Developed a robust calibration process that enhances the precision of gaze tracking for individual users.
• Achieved significant optimization of the model to run efficiently on mobile hardware using techniques such as post-training quantization.

What we learned

• The importance of robust preprocessing and feature extraction in achieving accurate gaze estimation.
• Techniques for optimizing deep learning models to run efficiently on resource-constrained devices like smartphones.
• The value of real-time data analysis and calibration in improving the performance of machine learning models.
• Insights into user behavior and interaction patterns that can inform the design of more intuitive and responsive interfaces.

What's next for Mobile Gaze

• Enhanced Calibration: Further refine the calibration process to make it more user-friendly and improve accuracy.
• Broader Applications: Explore additional applications in areas such as virtual reality, augmented reality, and accessibility tools for individuals with disabilities.
• User Studies: Conduct extensive user studies to gather feedback and improve the system's usability and performance.
• Integration with Other Platforms: Expand the compatibility of Mobile Gaze to work with a wider range of devices and operating systems.
• Advanced Features: Incorporate additional features such as head position tracking and emotion recognition to provide deeper insights into user behavior and interaction.
• Open Source: Consider making the project open source to encourage collaboration and further innovation in the field of mobile eye-tracking technology.

Built With

• android-studio
• androidlibsvm
• java
• mlkit
• python
• tensorflow