This project acts as the first step to an entirely gesture based computer experience. When paired with a mouse, using Mid Air means there is no need to ever touch a keyboard. It's not hard to imagine how a mouse could be replaced by a second Leap Motion, or even an extended version of Mid Air on a single device. This could be built upon even further for more benefits, such as using the movement mapping capabilities of Mid Air for security in the form of in-air signatures. Finally, the simple hand gestures involved make this product a good alternative for people with limited digit mobility, and means those with few fingers don't have to rely on a 'hunt-and-peck' method.
What it does
Mid Air uses the Leap Motion's precise hand tracking and Azure Cognitive Services' text recognition to enable in-air handwriting as a text input to a computer. Simply write a word with your finger over the sensor, and watch as it is analyzed and extracted to your copy clipboard.
How we built it
The Leap Motion tracks the tip of the index finger of a hand held above it, the up, down, left, and right movements of this tracking point are used like the movements of a pen on paper to generate an image of what was written. This image goes through some pre-processing before being submitted to Azure Cognitive Services for text extraction. Finally, any text which is found is pushed into the copy clipboard.
Challenges we ran into
There was quite a bit of challenge in setting up the pre-processing so that Azure could (relatively) reliably understand that it was being shown some form of text.
Accomplishments that we're proud of
The product produces semi-reliable results despite how difficult it can be to be accurate when drawing in the air.
What we learned
A lot more about python data and image manipulation. Also how to make a GUI in Python.
What's next for mid air
Ideally it wouldn't need any GUI controls whatsoever. Through Leap Motion's gesture recognition, we would like to enable starting/stopping recording, as well as adding spaces and periods, and pasting the generated text.