ASL Translation

• 

  Glove sensor

• 

  Robotic hand

• 

  Robotic hand

• 

  Glove sensor

Inspiration

Inspiration came from seeing this technology 20 years ago and wanting to approach it using today's hardware and algorithms.

What it does

Two parts:

1. ASL to Letter glove: Using a dedicated glove that tracks hand gestures, we train a machine learning algorithm to learn the pattern. After training, we let the algorithm figure out what letters we are signing with our hands in real time.
2. Text to hand: Robotic hand that gestures the letters that's being translated from typed texts.

How we built it

Utilizing PyTorch for machine learning. MQTT and Python (with Blockchain authentication) for communication. FPGA for motor control for hand. ESP32 for glove sensor data transferring. 3D printing for the hands.

Challenges we ran into

Our sensors could not cover the full motion of our hand. In fact, some sensors could not describe the movement of the thumb.

Accomplishments that we're proud of

We are proud of integrating hardware and software together as a team.

What we learned

We learned how to communicate with different IoT devices using MQTT, trying a different range of resistors and see how they affect the output of sensors, and how to use FPGA to control servos to move the hand.

What's next for ASL Translation

A different set of sensors that are more sensitive to thumb and palm movements.

Built With

• ai
• arts-and-crafts
• blockchain
• c++
• esp32
• fpga
• python
• solderingiron
• vhdl