Hand Gesture Interface

Problem

The mouse was invented in 1964 and the keyboard was invented almost a century before that. These systems have gone through several form factor changes, but they retain largely their original functionalities. This creates an inflexible relation between the user and the computer. Hardware is difficult to fix, difficult to update, and expensive to manufacture.

Solution

Hand gestures are easily recognizable, and as humans, we innately understand a multitude of universal gestures. They're easy to learn, easy to understand, but not quite easy to implement in software. Traditionally, approaches towards hand gestures recognition rely on external hardware such as IR cameras or distance sensors. We took a more economic approach by using solely a laptop webcam running at less than 10 FPS and a resolution of 192x108. This imposes a very low hardware requirement for anyone to begin using this software.

By training a neural network over hundreds of thousands of samples across 27 different classes, we're able to achieve sufficient accuracy, and by using specific types of convolutional layers, we achieve low enough latency to run real time.

Implementation

We used a 117 layer neural network that processes 2D images in the first 112 layers, and across temporal slices in the last 5 layers. This distinction between spatial dimensions and temporal dimensions allows us to dynamically scale the framerate of the system by throttling frame rates and process frames in parallel, giving us both low latency and low power usage. By using separable convolutions and Inception-ResNet blocks, we're able to achieve both sufficiently high accuracy as well as deep networks with high efficiency.

End Goal

We hope that a future of hardware-less computer interfaces can emerge, as IoT devices proliferate our daily lives. TV remotes, for example, can become a thing of the past. Smart monitors can operate seamlessly alongside users throughout their day, regardless of their location and what they're doing. We hope to redefine the "home computer" as more of a network of devices spread throughout a user's home, rather than a fixed terminal dictated by the location of a keyboard and mouse.

Built With

• numpy
• opencv
• pyautogui
• python
• tensorflow