I was inspired to make Sign Speak when I read about a barista at Star Bucks who learned sign language to make a frequent customer feel even more welcome at Star Bucks. While I may not have the determination to learn ASL, I can sure as hell engineer a way for people who use sign language to be heard!

What it does

Sign Speak is comprised of two main components. 1: The Arduino based glove that feeds data through bluetooth low energy while a person is signing. 2: The iPhone app is collecting the data send over bluetooth and proceeds to feed it into a special machine learning algorithm (SVM) and helps to determine the sign that the person is making. The iPhone then uses it's speaker to speak up what the person signed.

How I built it

I first breadboarded my entire circuit to ensure that everything was in tip top shape. From there, I went on to establish the connection between my Arduino and iPhone application, and limit the flex sensors to a readable value of 0-100. After that I learned how to solder and created the circuits that I needed. Unfortunately, it got fried, and I had to restart... So I re-soldered everything, fit it into an NES Power Glove, and went on to work on the machine learning algorithm. Unfortunately due to the nature of machine learning, a lot of data was needed, and I did not have the hours to train the glove to learn every ASL letter. So I decided on 4 important letters to me. H-A-C-K.

Challenges I ran into

Machine learning was definitely a challenge for me having used it only a few times before. I was given help by a friend not attending PennApps, Nathan Flurry aka McFlurry. Another challenge was losing all of my hardware mid way through. Luckily a PennApps volunteer/organizer found out that they had more Arduino Micros, that had not been listed before and saved my project.

Accomplishments that I'm proud of

I am very proud to have made Sign Speak over the past 36 hours on a 1 man team. It was a daunting task but I think I was able to execute a successful hack!

What I learned

I learned a great deal about Arduino hardware and how to solder, as well as Arduino and iOS interfacing.

What's next for Sign Speak

I hope to create a more refined version of Sign Speak, one with an accelerometer/gyroscope that will allow me to not only identify letters, but also words/gestures.

Share this project: