Gestur

Inspiration

Inspired by family members of our own suffering from arthritis, as well as the intuitive controls found on Nintendo's Wiimote, we decided to use Jacky's newly acquired gyroscope to develop a tool that brings hardware and software together.

What it does

We developed Gestur, a microcontroller-based accessibility tool that maps specific motion-based inputs to a wheel, which then translates these inputs to a keyboard input. We included auto-complete word suggestions for a streamlined user experience.

How we built it

The microcontroller is programmed in C, using the ESP IDF SDK. The gyroscope produces values of pitch, roll, and magnitude. Communication between our web application (written in TypeScript, React, and Tailwind) and the microcontroller is done via Bluetooth, where the value is extracted and decoded. Based on different thresholds, inputs are replicated on screen in real-time.

Challenges we ran into

One of the most significant challenges we had was creating a reliable user experience with the hardware. Often the input would feel slippery when spinning around the wheel, so we would set a timer so that after a constant threshold (.5 second), the user would clamp to that section, rather than being slipped into the next section on accident.

We also iterated over several different wheel designs as we needed to balance the ability to access our chosen letter quickly combined with the ability to have a margin of error with the device. Initially having a wheel of 26 characters, we eventually settled on a multi-tiered wheel design for choosing each character within 3 gestures while maintaining usability.

Accomplishments that we're proud of

We are proud of the user experience, and final wheel design we settled upon. After just 10 minutes with the device, we were typing characters and full words quickly with the autocomplete functionality.

We are also proud of the way we programmed the gestures from the raw inputs in our microcontroller, and the trial and error process we used with different threshold values resulted in a reliable user experience, combined with "clamping" so that the user's selection doesn't slip around on minor pitch changes.

What we learned

For some members in our group, this was our first experience programming a microcontroller and hardware project in general, so hardcoding specific thresholds for reliable motion controls required significant trial and error.

Also for some of us, it was our first experience writing and JavaScript and TypeScript, so getting familiar those languages to develop webpages will be valuable going forward.

What's next for Gestur

We would eventually want to migrate Gestur's logic into a standalone application, able to produce keyboard input across an entire system, rather than just being restricted to this singular webpage.

There are several keyboard integrations that still have to be implemented for the tool to be fully functional, such as numeric and special character input, as well as switching between cases.

We would also ideally like to develop the capability to switch between keyboard and mouse input.