Open hand gesture
Closed fist gesture + swipe down gesture
Anyone who has seen a Leap Motion before knows how cool the devices are. Yet, not many people seem to actually own these devices. What if there was a way for us to bring the LEAP Motion experience to everyone with a mobile phone?
LEAP (or Leap Motion Emulating Android Project) is an Android app that allows you to use your front facing camera as a sensor for gestures, which can be interpreted by a computer and customized however you'd like.
What it does
LEAP uses the front facing camera in order to recognize users' hands. The app tracks the position of the hand, and uses this to track gestures such as waving your hand to the right. The app also allows users to clench and unclench their fists as a form of gesture. From these data points, a Python script runs custom keyboard commands, allowing you to actually control your computer without even touching anything!
How we built it
The image processing is done with the aid of OpenCV. We create a color filter based on the user's skin tone and after applying this to the frame, we get a contour of a hand. From here, we can take the convex hull in order to determine whether or not the hand is currently clenched. We calculate the center of the contour, which we can use to track movement. All of the data that we collect about the contour, and its movement are then sent to a Firebase database, which is read from in real-time by a Python script. The Python script then determines if any of the data constitutes a gesture, and then executes the keyboard command corresponding to that gesture.
Challenges we ran into
Where can we start?! Firstly, although OpenCV is widely used, the Java library is not as well documented. This is even worse for Android, so many of our problems and questions had to be solved with trial and error. Even once OpenCV was working, we still had to worry about false-positives. Moving your hand off of the screen often resulted in false hand-clenching reports. We also had to deal with the case of moving your hand back after completing an operation, which would often trigger the opposite keyboard action.
Accomplishments that we're proud of
Basically, the whole app! We have never used OpenCV before, and we're all pretty proud with how this turned out. We also ended up getting an MVP about 3/5 of the way through, which allowed us to fine-tune things.
What we learned
- WE STILL HATE THE ANDROID CAMERA!!!
- OpenCV for Android is under-documented and you have to get lucky with StackOverflow solutions
- Everyone loves Python
What's next for LEAP
We're looking to sell our app to Leap Motion for $1 000 000 000. If you are a Leap Motion executive, and are looking to acquire our application, please contact us.