While we were checking out the cool hardware at TAMUhack we stumbled upon the Leap Motion controller. After looking it up and checking out the documentation, we realized there was a lot of potential for it with its relatively precise hand tracking and finger tracking.
What it does
In the example game, a player can use his or her hand to control a car. To start and stop the engine, the player makes a fist with their thumb extended as if they were holding an imaginary key, and rotates their fist to the right approximately 90 degrees. By adjusting the angle of his hand, he can control the speed of the car. By rotating or swiping his hand he can rotate the car in the direction of their hand. The player can also point their index finger down quickly to shoot projectiles to eliminate boxes.
How we built it
Using the Leap Motion documentation and SDK for C#, we were able to use Monogame, a framework based on Microsoft XNA, to create a gesture and hand interpreter from scratch using vector math and trigonometry.
Challenges we ran into
The Leap Motion controller is not 100% accurate, and often times during testing there were many blips or glitches that occurred at very small timeframes, meaning we had to adjust thresholds and tolerance levels for a lot of our functions. The controller also required constant calibration because of smudges and lighting changes, since the controller is mainly based on using a high quality camera to do precise hand tracking.
Accomplishments that we're proud of
We were pretty proud of being able to build an interface for the motion controller that was easy to work with and easy to operate in C#. We were also proud of being able to interpret the C# documentation and SDK documents provided to us by Leap, and that we were able to implement many of their functions with relative ease.
What we learned
We learned about how to better manage our time and divide up work. We also learned about the challenges of working with such sensitive and finicky sensors and how much time and testing would be needed for getting the right result we wanted.
What's next for Hand Nav
The Leap Motion controller has provided us with a lot of potential to work with and we're really excited to see what it can do in the future. The next step for Hand Nav is to migrate into a 3D environment such as Unity or Unreal (which the Leap has excellent support for). We think that a lot of businesses could benefit from a precise gesture/hand control system like this.