SignWave by The She-Coders With A He

Thumbnail
Your lovely skeleton teacher
Dashboard page
Learning module
Hosting the game and enjoy it with your friends!

Inspiration

Inspiration for this project began with an interest in medical technology and how it can make healthcare and communication more inclusive. We realized that communication barriers affect many groups, not only those who are Deaf but also those who may have lost the ability to speak, and non-verbal children in neurodivergent communities.

What it does

We created a web platform that helps users learn ASL interactively through real-time gesture tracking. Using a webcam and machine-learning-based hand and face tracking, users can practice signing letters, numbers, and words while receiving live feedback. Our platform also includes mini-games for practicing ASL, including a multiplayer option. Users can join together to practice signing in a fun and competitive setting. Our website takes ASL learning from static textbook lessons into an engaging, movement-based experience that is more aligned to how the language is actually used.

How we built it

We built our website using Next.js for the front end and Django for the backend. The frontend provides a responsive UI for live camera input and lessons. The backend processes the hand placement data using MediaPipe Hands. This was used to determine which hand signs corresponded to specific numbers by checking specific finger placements. We also use two external pre-trained models with our customized improvement for letter and word recognition. The pre-trained letter model was taken from https://huggingface.co/prithivMLmods/Alphabet-Sign-Language-Detection, and the word recognition model was taken from https://github.com/209sontung/sign-language?tab=readme-ov-file.

Challenges we ran into

Some of the challenges we ran into included the AI model's accuracy, as getting a consistent ASL recognition across different lighting, backgrounds, and hand angles was tricky. Also, because we used multiple frameworks, coordinating between them was challenging.

Accomplishments that we're proud of

We successfully built a working live-tracking ASL recognition system for letters, numbers, and words. This creates an educational platform that promotes accessibility and inclusion while building a bridge between neuroscience, linguistics, and technology. Finally, we are proud that our project started from empathy with a desire to help people communicate, and it turned into a tangible, functioning prototype.

What we learned

We learned how to integrate machine learning, web frameworks, and ASL into one product.

What's next for The She-Coders With A He

We plan to expand beyond letters, numbers, and simple words to full sentence recognition, integrating deep learning models for greater accuracy. Add more interactive and fun options for users to gamify the learning experience. Partner with ASL educators and schools to allow our platform to become a universal accessibility tool, bridging communication gaps and helping more people connect through sign language.