JustLook

• 

  chart for business analysis 1

• 

  chart for business analysis 2

• 

  AI model for feature selection

• 

  mathmatical model for gazing estimation

• 

  Google Trend for 'touchless' keyword

• 

  virus transmission occurs in modern order machines using touchpad

• 

  scientific report about risk of touching surfaces

• 

  gaze_estimation_example_1

• 

  gaze_estimation_example_4

• 

  gaze_estimation_example_3

• 

  gaze_estimation_example_2

• 

  usage scenario 1

• 

  usage scenario 2

Inspiration

Humans are almost always in a battle against a deadly epidemic or virus throughout history, and the most recent COVID-19 pandemic raised the demand for public hygiene to a higher level. Under the current circumstances, the safest technology will be one that does not require physical contact. Our eye gaze tracking software provides a touch-free solution for vending machines, restaurants, apartment complexes, and more.

What it does

JustLook is a machine-learning-based technology that replaces a mouse or touch screen with eye gaze tracking. It collects eye movements and predicts a user behavior or selection. In this project, we implements this technology on an electronic menu, and the user completes their order touch-free.

How we built it

First, we used a combination of deep learning models and mathematical formula calculations to reduce the model's requirements for image clarity and to improve the accuracy of detection. We extracted 18 feature points of the eye through a deep learning model supervised by the upper and lower layers. The upper layer of our model uses an hourglass model to extract multi-dimensional eye information. Our lower model supervises and updates the weight of the upper model. Then we use a mathematical formula to converge the detected landmarks to the final gaze direction vector. As for the interface part, we used frontend HTML and Bootstrap to simulate a menu from a general restaurant interface, upon which we demonstrate how we use our touchless algorithm to help customers purchase items without physical contact.

Challenges we ran into

1) The accuracy of detecting the annotation direction is not high enough 2) There is a network delay in the connection with the model in the server

Accomplishments that we're proud of

Each team member of ours truly used and dedicated each of their own best talents and efforts for this Hackethon. So it is a truly great learning experience for us.

What we learned

We learned to work with each other and delicate tasks that suit each team member's skillsets. We also learned to use deep learning model to solve a practical problem.

What's next for JustLook

We would definitely want to extend our project scope by polishing it in our free times and bring it to a scale that is more accurate and smoother.

Built With

• css
• dlib
• excel
• html
• javascript
• opencv
• python
• tableau
• tensorflow