handibee

Inspiration

Google's Teachable Machine itself was the main inspiration for this project. I (Fiona) have wanted to integrate Teachable Machine into one of my projects for a while now, but never thought of how or had the chance. At TreeHacks, I finally had that chance and brainstormed ways to incorporate trained data (either image, audio, or body pose) into an application.

In addition, ever since I was young, I had learned the ASL alphabet (and only the alphabet), a skill I also never had the chance to incorporate in any other part of my life. Thus, this project idea merged these two areas that I've wanted to explore together.

What it does

The application gives the user a picture of an ASL sign. The user must then finger-spell the word that the sign represents by fingerspelling to the camera. The camera will take an image every 1.5 seconds. The application sorts each image into a letter to decipher each letter the user was spelling. Finally, the application puts all the letters together and tells the user if the word was spelled properly.

How we built it

First, the model was trained using Teachable Machine. Images of individual ASL letters and their category were given as the training data. Images were of Fiona's hand at various angles. Teachable Machine trained the model and provided us with code to be exported and integrated into our project.

We created some user flows and prototypes on Figjam and Figma. Both the frontend and backend were created using the React framework, where we integrated JavaScript, HTML, and CSS to style and set up our app.

Challenges we ran into

Training the model took much longer than expected and there were five whole attempts, _ 5 times we trained the model _. Because we were training the model with images, things in the background of the image would get unintentionally picked up as well. For example, if training data had a bright light source in the background for some of the images, then the model might sort letters based on the background and not just the ASL letter symbol, which would lead to inaccurate sorting.

In terms of code, we had some trouble processing the captured image from the webcam to classify the letter that the user is signing (I.e. converting the base64 string to an actual file object for the Teachable Machine algorithm to process), so we weren't able to flesh out that particular feature yet. We also faced some issues with npm packages (some modules had trouble installing), so we had to work around those challenges as well.

Accomplishments that we're proud of

This was our first time training data altogether and using Teachable Machine. Although features of the project had to be cut down and we didn't get to implement everything we wanted, this was a great learning experience for us.

What we learned

Fiona - I learned how to use Figma's pen tool, which was something I used a lot to create the background images for our application. I also learned how to use Figma's inspect feature. I created a prototype on Figma before doing the frontend, so when it came time to doing the frontend, it took me some time to set things up, but once I realized I could use the inspect feature, frontend development went by a lot more quickly. Lastly, I learned how to use Teachable Machine to train data.

Ashley - I learned how to import and use various npm packages such as react-webcam and had the chance to review/hone my React skills. It was also my first time using and integrating the Teachable Machine algorithm into our code, so I was able to learn how to do that as well.

What's next for handibee

Unfortunately, our application is not functioning the way we wish it would. If we had more time, we would like to get to the point where our application could read multiple images being passed from webcam to app, which is our biggest hurdle. We have predicted that once this hurdle is overcome, the rest of the development process would be smooth sailing.

We also want to incorporate difficulty levels into the project in the future. Different words would be classified as easy, medium, or hard based on how long or unknown they are. Another function we want to add is some sort of point system. Lastly, as the original vision for this was a spelling bee, we would like to incorporate some sort of multiplayer functionality into the project.

Built With

• css
• figjam
• figma
• html
• javascript
• machine-learning
• react
• teachable-machine