Inspiration
Our team wanted to create a program to enhance the convenience of the computer operations for everyone. People with hand tremor, poor eyesight, and other disabilities have difficulty operating detailed computer/window actions, so we thought of this idea to help them control programs without mouse or mousepad.
How to use Air-Pad
Air pad currently performs 7 different actions; each action corresponds to some real-time human hand gestures read through computer webcam. When first running the program, stage your palm to be inside the square and press "a" to capture histogram of the hand. Now you can perform actions below and program will run actions correspondingly.
- Move window - one finger (all directions)
- Close window - slide down with 4 windows.
- Restore window - slide up with 4 windows
- Resize window - slide left(-) and right(+) to change size
- Minimize window - 3 fingers
- Maximize window - 2 fingers
- Lock Screen - Dab
How we built Air-Pad
We had three challenging goals throughout the project. First, was detecting the hand and detecting specific hand motions. Second, was determining how to control window and os operations, and last but not least, was finding specific criterion to connect hand motions with different operations.
For hand detection, we used the python OpenCV library that is commonly used for motion recognition. OpenCV generates contour lines that replicate the boundary of the hand, but there were problems distinguishing between face and hand. Through attempting various models, the problem was consequently solved by using a algorithm that defines the largest contour area as the hand representation.
For window and os operation, the python built in os library and a outer-source library named PyGetWindow was used to make different computer operations. We were able to execute operations such as shut down, lock screens, and move, close, activate, minimize (single and multiple), maximize, close, minimize windows.
Lastly, given the output data of coordinates, area, and number of fingers, we linked different hand gestures to separate operations. The operation - action pairs we have defined are as follows.
What's next for Air Pad
For the future, our team wants to develop our program for people to customize their own hand gesture for a computer operation, and people to operate computers without mouse. We also expect to make improvements in hand gesture recognition using machine-learning assisted complex models.
Log in or sign up for Devpost to join the conversation.