FingerSpellr

A glove that detects signed letters in American Sign Language.

Comment

Inspiration

Fingerspelling, or the act of signing individual letters to spell out words, is one of the first lessons that non-native learners of American Sign Language encounter. However, it can be quite difficult to remember all the corresponding signs and be able to spell out words fluently. I would like to facilitate the learning experience by designing a glove that can sense the movements and postures of the hand in order to detect the hand signs of ASL letters. This glove can be used for practicing fingerspelling efficiently, as well as increasing the ease of communication between ASL learners and speakers.

Accessibility considerations

It should be noted that this project is not meant to serve as a direct translator for ASL, as it is against the spirit of accessibility to promote devices that need to be used by deaf people for the benefit of hearing people. Instead, I hope to introduce more hearing people to ASL by improving the learning experience through FingerSpellr.

What it does

FingerSpellr is a glove that can detect signed letters in American Sign Language. Through flex sensors and accelerometers on the glove, FingerSpellr would be able to recognize the hand shapes corresponding to letters and numbers in ASL. With this ability, FingerSpellr can help ASL learners with the basics of sign language through a flashcard-like learning program that gives feedback on their fingerspelling.

How we built it

Milestone 1

Goals:

  • Sensors and peripherals should be securely mounted to the glove.
  • The glove prototype should be able to parameterize different hand gestures, including whether or not fingers are bent/outstretched, whether or not fingers are touching each other, orientation of the hand, and acceleration of the hand.

By milestone 1, I was able to have a prototype of the glove with two flex sensors, and two contact sensors mounted on the glove, and the accelerometer wired to the MCU. On the software side, I was able to poll the ADC port by port and print the readouts directly over serial.

Final Demo

Goals:

  • Sensors and wiring should be securely mounted, compact and allow for mobility.
  • Each of the letter and number signs of ASL should be accurately recognized when signed by different users.
  • Create an ASL learning program which goes through letters in a flashcard-like manner and gives feedback on the signed letters from the user.

For the Final Demo, I was able to reach almost all my previously set goals. The glove had all the sensors securely attached, including all five flex sensors, four contact sensors in the form of copper tape, and the accelerometer. I had collected data on all the ASL letters, and hardcoded parameters for identifying different letters using finger position, contact, and acceleration. Then, with the current sensor data, the program would calculate the "distance" between the five finger readings to the expected readings for each of the 26 letters, and determine three of the most likely letters. It also accounted for contact between fingers, using the digital readouts of the 3 contact pads as boolean filters, i.e. if the thumb and the middle finger were in contact, then only the letters where the two fingers are touching are considered for the guess. For letters that were similar in shape but different in orientation, as well as letters that relied upon hand movement, the accelerometer data was used to determine the most likely letter. After using the above algorithm, the top three most likely letters are printed on the LCD screen, and updated every cycle. Furthermore, there is a "quiz mode", where one letter is selected at random at a time and displayed on the screen, and a timer counts down for around 10 seconds. If the user of the glove is able to produce the correct letter in this time, i.e. the chosen letter is within the top three guesses at any given time, then a green screen is displayed on the LCD to confirm that they are correct. The finished product was able to recognize almost all letters accurately, especially the ones with relatively unique hand shapes, and could do this in a short amount of time that is not noticeable to the user. However, the letters that relied on accelerometer data were not as accurately recognized. Letters like 'g' which relied on the accelerometer to provide orientation data were recognized around 80% of the time, while letters like 'j' and 'z', which have the exact same shape as other letters but rely on movement of the hand to establish the letter, were recognized correctly around 50% of the time.

Challenges I ran into

There were several different challenging aspects during the completion of this project. The first unforseen challenge was related to project management, as I was sick for half a week and in isolation for another half. Therefore, I was unable to access materials and equipment for a while, and had to re-order my priorities. I had previously planned on finishing all of the hardware installation before Milestone 1, and focusing on the software afterwards, but I ended up working on both at the same time, in order to be on schedule during my covid week. Another challenge I had was related to the ADC. First was the fact that the Arduino only had 6 analog ports, while I had 5 flex sensors and 3 axes for the accelerometer. My solution was to determine the one accelerometer axis that changed the most significantly (in this case, the y axis), and to only use one analog port for that. Another challenge was regarding ADC noise, as I discovered that the flex sensor readings would have irregular peaks and a lot of noise. I ended up calculating a rolling average for the last 3 data points for each ADC input, and using that as the analog reading. This was able to somewhat lessen the amount of noise, while still keeping the cycles fast enough. I also added calibration constants in the header to record the baseline values for each sensor, in case one of them changed over time or I had to switch sensors. For the flex sensors, the constants included an averaged reading over many data points at both minimum and maximum curvature, while for the accelerometer, it recorded the value at level (1g).

Accomplishments

I am glad to have completed all of the planned milestones in the desired timeframe, especially with the one-week holdup of covid isolation. I am proud of the flex sensor mounting mechanism that I came up with, which uses a certain type of hair-tie that forms a loop on itself, and is just the right width for the flex sensors. I am also excited about the speed and accuracy of the finished product, and the way that it reliably tells apart similar letters such as 'e' and 's', which assures me of the feasibility of the algorithm that I used to produce the most likely letters, as well as the reliability of the parameters data.

What I learned

One thing I learned a lot about is how to work with ADC input, especially with sensors that require calibration and have considerable noise. Another takeaway is how to design products with ease of access in mind. When making the prototype, I had to rearrange the components and wiring in order to allow for better hand movement when wearing the glove. Improving user experience is something that I would not have taken into consideration for a regular lab, so it was challenging but useful to have that be a consideration from the start.

Improvements & What's next for FingerSpellr

  • I had used 5V for the flex sensors, so I set the ADC reference voltage to 5V. However, the accelerometer output turned out to be 3.3V, and due to polling the inputs one by one, I couldn't just use a 3.3V external reference voltage for the ADC. Thus, I had used 5 V as the reference voltage for the accelerometer as well, which led to decreased resolution. A better solution would be to also use 3.3V for the flex sensors, since they are only variable resistors, and to set that as the reference.
  • The wiring on the glove is still somewhat bulky, as there are two wires for each flex sensor, and 6 wires for the accelerometer. This can be improved by using a small breadboard attached to the glove to handle the sensor connections, and use less and longer wires to connect that to the arduino.
  • The glove could be made more accessible, as it currently only fits medium to small hand sized. Instead of using gloves at my own size, it would be better to order unisex medium gloves that can be used by more people, or make multiple models for different hand sizes.
  • The "quiz mode" currently only gives feedback on whether the user got the letter correct or not, and does not provide any extra incentive. It would be nice to create a game-like mode where users can earn points based on how many letters they signed correctly.

Built With

Share this project:

Updates