-
-
-
Home page
-
Image recognition recognizing the C letter in ASL
-
Image recognition recognizing the T letter in ASL
Inspiration:
During the pandemic, many of the previous communication-related struggles that deaf and hard-of-hearing people faced were amplified as the rise of virtual platforms revealed the continuation of the dinner table phenomenon in which they are side-lined during conversations (Murray). Many web-conferencing tools lacked features like real-time captioning or sign language integration. Since virtual social spaces are here to stay, it is crucial that these gaps in accessibility are bridged.
What it does:
SignScribe is a web-based tool that allows users to have their signs transcribed in real-time. It uses image classification to capture hand poses and convert them to the appropriate expression. SignScribe is a web-based tool that allows users to have their signs transcribed in real-time. It uses image classification to capture hand poses and convert them to the appropriate expression.
How we built it:
To build SignScribe, we first trained a machine-learning model using a data set sourced from Kaggle. We then used Python to build the backend, which detects signs from user input every five frames and picks the most likely sign from the model's predictions. For the front end, we designed the layout in Figma and coded it using HTML and CSS. Lastly, we linked the front and back end together using Flask to complete our comprehensive application.
What we learned:
From this project, we gained significant knowledge. We obtained hands-on experience with training machine learning models, understanding issues like overfitting and the importance of picking viable datasets. We also became familiar with using Flask, a web framework, to create a functional tool.
Future growth:
SignScribe has a high potential for future growth. A logical progression would be its evolution into an extension that can be integrated into mainstream web conferencing platforms. Currently, its scope is also restricted to the ASL alphabet. Moving forward, by training the model on a more exhaustive dataset so that it can identify a greater variety of signs.
Citations:
https://www.today.com/health/deaf-people-are-left-out-conversation-during-covid-19-t193570
Log in or sign up for Devpost to join the conversation.