Recording a new gesture through the web interface!
Jesture Web Interface
Jesture drawing application
We were inspired by the increasing prevalence of hardware in the world of software. As we were browsing through the sponsors list for Calhacks this year, Synaptics caught our attention, and we decided to use their touchpad as the base for our project. From there, we wanted to add convenience to our lives, and the idea of an augmented mousepad was born.
What it does
Jesture extends the functionality of a touchpad mouse with customizable gestures that activate desktop applications and websites. Users can view their personalized gestures on a website and create their own unique gestures by using the Synaptics mousepad.
How we built it
We used a Python script to read data from the touch pad, filter out some noise, detect the location of the touch(es) using flood fill and k-means, link multiple readings over time to curves, and then detect similarities between two curves by calculating the magnitude of changes in the slope of the tangent line over the curves (after using derivatives at various points to generate some more points and obtain equally-sized sets of points for the two curves being compared). We also used Python to control the mouse position and let the touch pad be usable as a trackpad for the computer, and to delegate actions when specific motions were detected on the touch pad, such as specific curves (gestures) and taps with the tip of the finger. Actions were handled with a mixture of Python and AppleScript, and the settings/gesture managing interface was written with a Python backend and HTML/CSS/JS frontend.
Challenges we ran into
One challenge we faced was processing the raw Synaptics touchpad images into accurate touch points. It was important to reduce noise until we could reliably recognize sequences of continuous touch locations as gestures. Furthermore, in order to smoothly integrate a variety of applications, we had to adapt to different requirements for each piece of software we wanted to include.
Accomplishments that we're proud of
As beginners, we’re very proud that we were able to learn so much in such a short period of time. Armed with knowledge from computer science classes and little practical knowledge, one of our biggest obstacles was certainly learning to apply our knowledge to the web platform. In addition, we’re proud that we were able to tackle integrating the Synaptics touchpad into our project, since we have never used anything like it before.
What we learned
What's next for Jesture (ver. DDOSKI)
We’re working on integrating other apps with Jesture to reach more users. Along those lines, another area of expansion would be in the Internet of Things - integrating household and real-life applications.