American Sign Language Recognizer

• 

  A

• 

  B

• 

  C

• 

  D

• 

  E

• 

  F

• 

  G

• 

  H

• 

  I

• 

  K

• 

  N

• 

  M

• 

  O

• 

  L

• 

  P

• 

  Q

• 

  R

• 

  T

• 

  U

• 

  S

• 

  W

• 

  Y

• 

  V

• 

  X

• 

  A snapshot of what the application looks like! Classifying the letter C

Inspiration

I've always was kind of interested in learning american sign language but never really got around to it. I saw this as a suggested project and thought it would be a great way introduce myself to it!

What it does

It takes a stream of images and feeds it into the AI machine. This machine then spits out the corresponding finger sign. While it's primary goal was to act as a "translator" from american sign language to text, I believe it could work instead as an effective and interactive learning tool. People who are interested in learning the basics can use this recognizer to get real time feedback on which letter they are signing (right now it only works on the letters a-y excluding i).

How I built it

I built it using pytorch and flask. I started by training the model and testing it on my own webcam before bringing it to flask where I can view my results in a brower.

Challenges I ran into

My original dataset contained a lot of similar images that all had the same background. When I tried testing my model on real images using my webcam, I was getting very unreliable results. In the end, I chose a different dataset which contained images with a variety of different backgrounds and found better results. However, it still was not performing well on real data that didn't have a plain background. While it seems to work well on plain backgrounds, finding a plain background could be inconvenient and so I would have to find a solution in the future.

Accomplishments that I'm proud of

It works! Its not perfect, could be much better but it works and I'm very proud of that.

What I learned

The hard stuff isn't as hard as you might think it is and the easy stuff isn't as easy either. The most important step is to not assume or think too much about it and just jump right into it. You'll get a grip and understanding of what you need to do much quicker and can give yourself more realistic expectations.

What's next for American Sign Language Recognizer

step 1 would be to improve performance on images with a busy background. A possible solution right now could be to apply some sort of background subtraction algorithm so that the model can perform similarly between plain and busy backgrounds.

step 2 would be to deploy it on a public server so that anybody can try it out!

Built With

• flask
• opencv
• python
• pytorch