A Hacktech Hardware Hack
We wanted to do something cool and new with a Synaptics Touchpad. First idea that came up was play music like strumming a virtual guitar in the air (because the Touchpad can even sense hovering!) But as we were making our program learn gestures, the brilliant idea of learning gestures for handwritten text just came to us. It was an ambitious but doable project and then the benefits of such a novel technology became apparent and motivated us to create new and optimize old algorithms to detect English capital letters.
What it does
Many people whose communication abilities are impaired must use Braille to comprehend the world around them, and talk to the world around them. But with Writing2Learn, the options to communicate with the classical languages of our world open up to these people. Through hand motions on a Synaptics TouchPad, Writing2Learn voices out the letter being drawn and also records the letter virtually.
How we built it
Challenges we ran into
Perhaps the crux of this product was being able to recognize which letter is being written at a given time. We are taking the image and flattening it into a vector and then pushing it into an automated averaging algorithm which takes many, many image samples of say the letter "O" and determines the characteristics of a handwritten "O".
Our symbol recognition has been our toughest challenge yet. By taking the touch points that we recognize in the gesture, we created a map of the letters that we have pressed based on is recognized. We transform this data into long vectors of size length squared. We are taking the vectors that we created and then computing the inner product of the x and y coordinates of every pixel that is affected by the touch on the Synaptics TouchPad. We take the highest inner product and correlate it to the best represented letter or symbol in our dataset JSON file.
Playing Audio on a HTML page is simple, but playing audio on a local host's index page is a different story. We had to learn server-side http requests and just being able to play the .wav sound of a particular letter after it is written was difficult.
Accomplishments that We're proud of
We are proud of our algorithm to categorize image data, applying concepts we learn at school in real life, and creating a functional product in less than 36 hours. Each team member had unique perspectives, experiences, and expertise, yet we were able to use those differences to enhance and optimize the ultimate product. We overcame many technical challenges as well, which are discussed above.
What We learned
What's next for Write2Learn
Write2Learn currently has letter recognition and audio files for English characters. We expect to create a training set for Chinese and Arabic characters as well as audio files for the most common words of these languages to provide people the chance to learn how to write foreign characters. There are several feature to be implemented like being able to record the sound of phonetics/letters for other languages. Also, given more time we could create some algorithms that gave tips on how to improve calligraphy and style of writing which would increae the educational value of Writing2Learn.