Inspired by Jean-Dominique Bauby, a PLS stroke patient with full body paralysis (except for eye movements) and unable to produce speech, he documented his story and experiences through a book written using eye movements, an alphabet board, and the help of his speech therapist and assistant. They worked with him to transcribe his thoughts blink by blink, letter by letter. We wanted to build an application that would similar patients to have the freedom to express their thoughts - giving a voice to those without the ability to speak or sign.

What it does

  • Tracks eye movement through a webcam and translates movements into keyboard actions
  • Uses a combination of horizontal and vertical eye movement to navigate keyboard using cursor
  • Blink to enter the selected character

How we built it

We went through multiple eye tracking application/package online and found a tutorial that has the basic function of what we need written in c++. We adapted some pre-existing algorithms from a project utilizing Haar Cascade Classifiers (Haar Cascade Algorithm) for face and eye detection along with OpenCV for video capturing, image processing, and feature detection (to locate the pupil). Then, we tracked eye movements in four directions and blink actions by comparing the current position of the pupil to ones in the previous frames. The movements and the blink signals (selection) are then used to control the keyboard displayed on the screen where users would get to enter text.

Challenges we ran into

  • c++ runs a lot faster than python, so after adapting into python, our image processing is slower and we had to trade off some accuracy in detecting the pupil in order for the application to run a bit smoother
  • finding the right parameters for best face/eye detection is really hard and gets affected by a lot of outside factors (lighting, angel, position of webcam, type of computer, etc.)

Accomplishments that we're proud of / What we learned

  • Translating a c++ algorithms into python
  • Learning how to use OpenCV to for image detection
  • Making use of what we learned in CPSC425: IMAGE PROCESSING

What's next for eyeWriter

  • better pupil detection
  • more efficient image processing process
  • stable direction/blink detection with more robustness to outliers & noise
  • connecting text output stream to other running processes for interoperability with other programs (Web browser, search, etc.)

Built With

Share this project: