CookieJar.vision

Take an at-home eye exam if you can't make it to the optometrist during the pandemic or just want to see how good your vision is.We have a mode for audio-response users and for nonverbal/deaf users.

Comment 1

Inspiration

Unable to schedule an appointment to get eyes checked due to the pandemic, CookieJar.vision found a solution!

What it does

Administers an at-home eye exam:

When shown a line of letters in the alphabet:

  • Audio response users will speak the letter which they see on the screen
  • Non-verbal/deaf users can show their webcam the American Sign Language hand gesture that corresponds to the letter they see on screen The user will receive their acuity score without having to have visited the optometrist (disclaimer: we are not a substitute for a licensed optemetrist)

How we built it

  • We implemented a machine learning convolution neural network (CNN) to classify American Sign Language hand signals into their corresponding letter of the alphabet
  • We used a web speech API to recognize user speech and compare it against the Snellen Chart to output a visual acuity score

Challenges we ran into

  • Getting the computer camera to take in hand gesture as input and feed that into the model
  • Uploading the ML model to website from Keras to TensorFlow.js

Accomplishments that we're proud of

  • Our tool is inclusive to people who are nonverbal and deaf
  • Getting the webcam set up
  • Learning html, css, javascript, web development, and glitch, from scratch
  • Converting from TensorFlow Keras to TensorFlow.js
  • Implementing the web speech API
  • Implementing machine learning algorithms and methods to get an accurate model from over 27,000 images
  • Accounting for biases and reducing noise in the data so that we can differentiate between different hand signals

What we learned

  • Html, css, javascript, web development, and how to use glitch
  • How to successfully work together and communicate well in a team of individuals with very different backgrounds
  • American Sign Language (kind of)

What's next for CookieJar.vision

  • Enhancing UI/UX for a greater user experience
  • Making the website more secure
  • Connecting the TensorFlow model to video input
  • Data analytics of user information: for what reason they are using the tool, what age groups, etc.
  • Further research and more features
  • Getting certification from official body of science
Share this project:

Updates