The Guiding Hand

Are things becoming difficult to understand? Why not use The Guiding Hand?

The Problem

Ever since the onset of covid, the world has heavily been depending on various forms of online communication in order to keep the flow of work going. For the first time ever, we experienced teaching via zoom, Microsoft teams, and various other platforms.
Understanding concepts without visual representation has never been easy and not all teachers or institutions can afford Edupen, iPads, or other annotation software.

What it does

Our product aims at building a richer bridge between machines and humans than primitive text user interfaces or even GUIs (graphical user interfaces). It is an alternative user interface for providing real-time data to a computer instead of typing with keys thereby reducing the amount of effort required to communicate over platforms.
When a user opens the Guiding Hand's website they see a button. On clicking the button they are led to our application.
The application uses hand gestures to perform various functions.

  • 1 finger to draw on the screen.
  • 2 fingers to change pen color or select eraser.
  • 3 fingers to take a screenshot.
  • 4 fingers to clear the screen.

How we built it

We used React to build the site, and Flask and Python libraries such as mediapipe, opencv, numpy, and pyschreenshot to build the interface.

Challenges we ran into

Our biggest challenge was integrating our Python application with our Reactjs website. To overcome this issue we created a webserver using Flask.

Accomplishments that we're proud of

We are very proud of the fact that we came up with a fully functional site and application in such a short duration of time.

What we learned

We learned how to create a site in ReactJS, link it to a Flask server that opens up our application that allows a user to draw on the screen using just hand gestures.

What's next for The Guiding Hand

  • Adding more hand gestures for more features.
  • Allowing multiple people to draw on the same screen simultaneously.
  • Annotating over a shared screen.
  • Converting hand-drawn text on-screen to text and saving it in a document.

Built With

+ 27 more
Share this project:

Updates