Inspiration

For this hackathon, we wanted to build something that could have some sort of social benefit. As engineers, our loyalty is first and foremost to humanity. However, the goals of engineering are often centered around increasing efficiency and decreasing cost. This narrow mindset has led to everything from exploitative labor practices to rampant online data collection. However, given the time constraints of the hackathon, we found it difficult to settle upon something feasible yet impactful. While taking a break to play our daily Wordle game, we were inspired by its rapid and immense success to develop a variation that would promote the learning of American Sign Language (ASL). Something entertaining, relevant, yet also had some meaning to it.

What it does

"Handle" is a spin-off of the hit word-guessing game "Wordle". Whereas in the traditional Wordle, users guess letters by inputting five-letter words through the keyboard, Handle takes input through American Sign Language. OpenCV and Tensorflow have let us use the device's camera to read visual input from users and generate a string from individual ASL characters. The rest of the game works exactly as Wordle does. Only valid five-character words may be inputted, used words are highlighted based on their inclusion and position, and users only receive six tries.

How we built it

The front end is built with Tkinter and Python, complete with game logic and win conditions. The completely original code is reflected on the attached Github in the 'handle.py' file. Socket, and again, Python, are used to achieve communication between the front and backend. Namely, it sends a detected character to the front-end for input and processing. On the backend, we re-trained a pre-existing model taken from Github. Because some numbers and phrases were unnecessary to us and would throw off the accuracy, we decided to remove those images from the dataset and re-train the model. The original model can be found here link.

Challenges we ran into

There were many challenges with the machine-learning aspect of the project, as well as the integration. On the backend, settling upon a framework and tech stack proved to be time-consuming. Furthermore, due to slow hardware, training an accurate model took a significant amount of time. In terms of integration, we found that front and backends are not as easy to integrate as it seems. For example, computer-generated input from the backend has some difficulties being inputted into the Tkinter input bar.

Accomplishments that we're proud of

We're super happy with the re-trained model! It had close to 98% accuracy on most characters, so at least on the backend and frontend separately, we knew that everything was great.

What we learned

Computer vision, full-stack development, socket, Tkinter, and Tensorflow. However, those are all technical skills. More importantly, we learned a lot about projects and coding as a whole. For example, we learned that things are always going to take longer than you expect them to. We also got a taste of what group software development looked like, learning how to build functions asynchronously.

What's next for Handle

We hope that we can get this software as popular as Wordle! Our product is very close to being able to be shipped, as it's already slightly past the MVP stage. With some marketing help and further development, Handle could become a popular Wordle spin-off, reaching millions of users and raising awareness for ASL beyond what any individual person can.

Royalty-free sign language illustrations designed by Freepik.

Built With

• opencv
• pillow
• python
• tensorflow
• tkinter