Sign With Me

Math Teacher, Mother of Four, and Grad Student - Kasi Edwards

Inspiration

I am a high school math teacher in a district with a significant population of Deaf and/or Hard of Hearing students. Integrating group activities can be challenging, but I am dedicated to enhancing their social, emotional, and academic learning. I am developing a GenAI Retrieval Augmented Generation tool to help increase their vocabulary and improve their communication with peers in school, work, and life.

What it does

Sign With Me prompts for a word and returns a video of the sign, along with synonyms. It includes a student login area for small group chats. Deaf and/or Hard of Hearing students get visual cues for communication, can read chat messages, look up words for interpretation, and respond. The homework section allows students to upload text and look up underlined words. The database currently has 2,000 words.

How I built it

I found open-source sign language videos on WLASL GitHub with 2,000 vocabulary words. After unpacking 10,000 videos into 2,000 categories and cleaning the data with Google Sheets, I sorted the videos with queries. I explored UI toolkits like Flutter, React Native, Joomla, and WordPress, and chose WordPress. For backend services, I opted for phpMyAdmin over Firebase and Heroku. For Generative AI tools, I used WordPress Search engine and a plugin glossary. I initially created a Joomla website on cloudaccess.net and later imported the dataset into a WordPress site on the same host.

Sign With Me is accessible on both standalone computers and mobile devices.

Challenges I ran into

Data - Video Material: There aren't many videos for the Deaf and Hard of Hearing online yet. I would love for our community to help develop more videos to expand the dataset.

Money: I'm a 3rd-year high school math teacher, mother of four, and a grad student. Except for data storage, SignWithMe was built using free resources. I frequently encountered costs for services, products, plugins, and platforms while working on the project, forcing me to change directions often.

Space: Space was limited with free services. I ran out of space on cloudaccess.net and had to purchase Dropbox for data storage, which presented its own challenges. After generating a link to each file in dropbox I was able to connect the videos to each web page in WordPress with an SQL query.

Time - Platform: I tried GenAI platforms like MS Azure, Flutter, React Native, Firebase, and Heroku, but they required payment. With my Joomla experience, I created a site on cloudaccess.net but faced backend challenges. To save time, I switched to WordPress, finding it faster and easier for database management despite data size limits. I ran SQL queries to load the data and am working on improving the front end.

Accomplishments that I'm proud of

I am proud to have a better understanding of American Sign Language, and that I am able to interact with the deaf and hard of hearing community on a deeper level. I am also proud of my 6th grader who said immediately after watching my video pitch that he could see using Sign With Me for group projects. I am also proud that he spent the next few days learning how to say "Hi, my name is ... " in sign.

I am also proud of the technical skills I learned along the way - using sheets and SQL to clean datasets and load databases, using GitHub resources to find datasets, using WordPress, and figuring out how to mass link items in dropbox to connect to databases.

What I learned

I learned that I want to learn more about how to create projects like these more efficiently. I learned that I have a lot of perseverance to obtain the results I desire. I also am learning more about how to use GitHub, and would love to learn more about GenAi creation, specifically a sign to text app.

What's next for Sign With Me

I would love to see Sign With Me have more data to train, so that a deaf and/or hard of hearing person can sign a word or phrase, and it will return the results into writing. I would love to develop a sign to text GenAI application for my students to use at school, work, home, and life.

I would love to involve the deaf and hard of hearing community at my school to help develop data to continue in enhancing their communication skills.