Hand Signals

Example of the handsignal ai running. Here it is recognizing the "rock" signal and reading it out.
Another example of the handsignal ai running. This is our program recognizing the "open" hand sign.

Inspiration

AI/ML has always been something that we wanted to incorporate into a project and we though this would be the perfect opportunity to experiment with this new tool in Computer Science.

What it does

Hand Signal uses an image recognition AI to read different signals and gestures created with your right hand. With its image recognition, it then labels the hand in the frame to an assigned label and is then put into a text to speech to speak out what the hand sign is.

How we built it

We implemented the TensorFlow library and the MediaPipe google api to create the image recognition tool. Then we fed it data for each of the hand signals we wanted to have in our project into a TensorFlow model we trained and built from scratch. To start the actual text to speech and reading of hand signs, we assigned the motion of palming open your hand from a fist. We then use the recognition built in and implement it into the text to speech to assign what the voice says.

Challenges we ran into

Original implementation of the data and the use of TensorFlow was a bit tricky at first because of the lack of experience in our group. We had to watch a lot of different videos and google up tutorials to familiarize ourselves with the database and its uses. The main challenge was creating the image recognition AI itself. Because of our previously mentioned experience level and lack of familiarity, we didn't know where to start exactly with the image recognition. However, after looking into other models and similar AIs, we were able to push through it.

Accomplishments that we're proud of

We are proud that we were able to create an hand recognition AI and use it with the text to speech editor. That was the original concept for the project and our proudest accomplishment is the fact that we accomplished our mission statement.

What we learned

How to use TensorFlow and MediaPipe. We also began scratching the surface into developing a neural network and its applications into real life uses.

What's next for Hand Signals

We would like to develop our image recognition into other uses, such as developing an app for remote LED control. Our future project involves using the hand signal AI, but instead of a text to speech, it will send signals over to the Raspberry Pi and take control of the LEDs. The goal is to use hand signals to remotely change the colors and intensity of LED strips.