The inspiration for building this project came from the passion to help people who are affected by the speicifc physical conditions of carpal tunnel or any immobilizing factors with the wrist or fingers. We wanted to design a keyboard and mouse application that would be a substitute for a mouse and commonly used shortcuts on the keyboard.
What it does
The program recognizes the hand gestures through foreground and background segmentation. As well, the ultrasonic moition sensor will measure the distance from the hand to the sensor, in order to predict the gesture or movement. This way the mouse and keyboard functions will work when specific hand gestures are performed.
How we built it
We used hypersonic sensors to track the distance of the hand from the sensor using Arduino. Then we programmed using python that uses a keyboard/mouse GUI(library) to control certain keys and mouse movements when certain values are received from the Arduino.
Challenges we ran into
A lot of files had to be updated and installed in python in order to control certain aspects or recognize the hand gestures in our program. As well, we ran into issues with file corruption, which led to hours of trial and error to find the source of the problem. The program that recognizes the hand is sensitive to background colors and movement which made it difficult to follow the hand. The foreground and background segmentation works effectively with a plain white background.
Accomplishments that we're proud of
We managed to get the program working with the hand gestures, as well as we incorporated a finger tracking software that tracks the finger and moves the mouse accordingly.
What we learned
We learned how to use the OpenCV library and apply more python functions to our coding. The finger recognition software required us to learn about the concept of foreground and background segmentation.
What's next for Hand Recognition Software For Keyboard Controls And Mouse
We would like to develop a full API software that is capable of handling more features and hand gestures to further ease the navigation of a computer.