Reflections
*Introduction * With the rise of portable, accessible technology the use of augmented reality has increased. The idea of virtual assistants and generated images that react to the user are now more possible than ever before. However, we still cannot simulate natural eye gaze, instead creating virtual assistants that give off a feeling of uncanny valley. We hope to start moving towards simulating natural eye gaze by first predicting the rest of the eye trajectory given some starting trajectory.
*Challenges * One of the hardest aspects of this project was gathering the data. The code to extract Webgazer dataset has been a bit unwieldy because of software compatibility issues. Additionally, extracting eye information for blinks has been difficult as we had to figure out what the corresponding coordinates matched the coordinates with MediaPipe’s face mesh.
*Insights * Yes, the current MSE was about 2500-3000. We also have the pipeline to visualize and generalize eye data from facemesh. It does surprisingly okay for a dataset with low resolution data in a semi-structured setting.
Plan We have started training a model. We have three architectures with variations of LSTM and CNN. Our current goal is to meet a low MSE threshold. We plan to integrate the eye aspect ratio and tune our model. We plan to focus on comparing the MSEs of our architectures to understand which model may work better. Our focus is more on interpretability than accuracy. We may change the integration of the blinks if it doesn’t improve our accuracy, but we also think this is a part of creating an eventually generative model.
Log in or sign up for Devpost to join the conversation.