Wavi: Highlighted Accessibility

Point at any line of text: your webcam highlights it in high contrast, reads it aloud, and zooms in. This extension is easy, interactive reading for low vision and dyslexia.

Comment

Inspiration

What if you control your screen with just a series of gestures? That's what we wanted to consider for our project. Some people may not be able to use a keyboard, but gestures create a more inclusive way to interact with the web. Largely composed of students highly eager and interested in human-computer interaction (HCI), we wanted to create a computer vision model that aids in accessibility; something that can be used on a daily basis, that's truly helpful for a wide audience.

What it does

Wavi is a gesture-controlled reading aid, where a webcam tracks your hand and gestures replace the mouse.

  • Point with your index finger: Move the mouse
  • Point with both index & middle finger: The line your finger rests on is highlighted with high contrast for accessibility
  • Thumbs up: Reads the line your finger last rested on, out loud
  • Four fingers: Scroll
  • Quick pinch (thumb + index): Click
  • Pinch and hold, then move hand up/down: Zoom in/out (50-300%)
  • Wave full hand twice: Pause program

How we built it

Running on MediaPipe Hand Landmarker and combined with OpenCV for webcam capture and preview, we returned 21 landmarks per frame for each hand. All gestures are just geometry on the landmarks, meaning no training must be done on the model. A StableLine tracker remembers the last line your finger rested on for at least 0.4 seconds, so that pinching or thumbs-up won't read a line your finger just drifted to.

Challenges we ran into

We ran into difficulties turning the Python code that allowed the hand tracking to run into a browser extension. The highlighting feature was particularly difficult as the text would not be highlighted properly or would not read the text.

Accomplishments that we're proud of

We are proud that we managed to implement a UI and the hand tracking feature together as they were not molding well initially.

What we learned

Many of us never used Python before so having to create extensions with Python, pivoting to creating an application and back to an extension, and going back and forth with our personal terminals was a new prototyping experience for us. We learned to try something completely outside of our comfort zone and expertise.

What's next for Wavi - Accessibility Gesture Extension

We wanted to implement voice commands that allowed for certain actions to happen if the user said a certain phrase. We would want to add that next.

Built With

Share this project:

Updates

Submission history