We were inspired to create something that would accelerate the extent of technology in the future. We noticed how often we utilized the finger-gliding method on our phone keyboards in order to type words at a fast pace by gliding across characters rather than press down on them individually. We decided to implement this method with a tangible keyboard connected to a laptop with the use of different types of hardware and software.
What it does
Glyde traces the movement of a user's finger while wearing a special glove over the intended keyboard. The program is first initiated by a pressure sensor that detects when the keyboard is placed in the position in the stand. Then, we track the user's finger while it moves over the letters in order to interpret the words intended to be typed. This output can then be used in order applications that require text input. For example, a user can 'swipe' the word 'read' by tracing their finger over the letters 'r','e','a' and 'd'. In some instances, the user's movements are harder to read and can lead to misspelled or incorrect words. Our program than utilizes a spell-checker that is trained via large text files to suggest a word based on the inputted characters, hamming distances, and frequency in the English language. However, no matter the situation there is always a guarantee that there will be an output based on the selected characters glided over.
How we built it
In order to recognize which characters the user glided over, we built a stand out of metal beams, plexiglass and duct tape. This stand creates an arch on which we mounted a Logitech 720p camera that points down. On the bottom of the stand, we placed a pressure sensor that detects when the keyboard is placed into position. Next, the camera begins tracking the finger as it moves across the board and stores the characters. Then, we analyze the character by using our spell-checking and string manipulation software to determine the best choice word to match the user input.
Challenges we ran into
Our main challenges were building a way to detect the individual keys and to track the finger as it moves across the board. We had to find an optimal design for our stand in order to mount the camera in a reliable angle that is able to recognize each individual key. Other issues included finding a way to connect the laptop data to the Arduino, and processing the data in order to display it on a LCD display.
Accomplishments that we're proud of
We are proud that we were able to make such a smooth and reliable keyboard system that does not require any physical pressing of the button. We simply just require the placing of a keyboard, and gliding of the finger over the intended keys in order to determine the word to be typed.
What we learned
We learned how to process images and video through openCV. We focused mainly on contour detection, object tracking, and string manipulation. We also focused on serial communication with the Arduino in order to send data back and forth between sensors and the computer.
What's next for Glyde
We intend to make Glyde a completely software based system that only requires a laptop webcam. As of now, we require certain placement for the keyboard in the stand in order to correctly determine corresponding keys, characters, and location. However, in the future, we intend to detect the characteristics of each key simply by using the laptop webcam and thus requiring no additional software.