SignQuest

Inspiration

Over 70 million deaf individuals communicate through sign language globally. Witnessing the communication barriers faced by the deaf community, we envisioned a world where anyone could learn sign language through an engaging, game-based web app, utilizing computer vision to make learning not just effective but truly accessible. That's why we created SignQuest. This project is our contribution towards a more inclusive world, where language is no longer a barrier but a bridge connecting us all.

What it does

SignQuest is a web-based game that utilizes computer vision technology to teach sign language in an interactive and engaging manner. Users are guided through a series of games and challenges, where their gestures are recognized and evaluated in real-time, providing immediate feedback. This innovative approach not only makes learning sign language fun and accessible but also enables users to practice and improve their signing skills effectively, fostering better communication with the deaf community.

How we built it

We combined cutting-edge technologies and programming frameworks to create an interactive and user-friendly web-based game for learning sign language. At the heart of our application is TensorFlow, which powers the motion detection model, allowing us to accurately track and analyze users' hand gestures in real-time. To ensure the precision of sign detection, we integrated Fingerpose, a library designed to recognize specific hand poses, enhancing the app's ability to distinguish between different signs accurately.

For the web app's development, we utilized a robust stack that includes Three.js and Spline for creating immersive 3D environments and interactive visuals, making the learning experience more engaging. The front-end of the app was developed with React, offering a dynamic and responsive user interface that adapts to the user's interactions seamlessly.

On the server side, we chose Flask—a lightweight web framework written in Python—due to its simplicity and flexibility, allowing for rapid development and easy integration with TensorFlow and other components of our tech stack.

Challenges we ran into

One of the primary challenges we faced was ensuring high accuracy in motion detection and sign language recognition. This task was particularly complex due to the significant variability in hand sizes, shapes, and movements among different users. Developing a system that could accurately interpret a wide range of gestures without significant errors required extensive testing and refinement. We had to balance the sensitivity of the motion detection algorithms to accommodate subtle sign language nuances while preventing false positives that could frustrate users. In addition to this, our interactive 3d modeled background was a challenge to implement while keeping full user functionality.

Accomplishments that we're proud of

We are especially proud of overcoming the technical challenges associated with real-time gesture recognition and creating an engaging, educational platform that has the potential to make a real difference in people's lives. The development of an intuitive and interactive user interface that seamlessly integrates advanced computer vision technology to teach sign language is a significant achievement.

What we learned

Throughout the development of SignQuest, we learned a great deal about the complexities of sign language, the importance of accessibility in educational technologies, and the challenges of implementing computer vision in real-world applications. We gained valuable experience in integrating various technologies and frameworks to create a cohesive and functional application. This project also deepened our understanding of the user experience, teaching us the importance of designing with empathy and inclusivity in mind.

What's next for SignQuest

Looking forward, we plan to expand SignQuest's capabilities and reach. Our next steps include refining the motion detection algorithms to improve accuracy and reduce latency, ensuring the platform can accommodate a broader range of users, including those with disabilities. We also aim to incorporate more sign languages beyond American Sign Language (ASL), making the platform accessible to a global audience. Additionally, we plan to develop more advanced games and learning modules, incorporating user feedback to make the learning experience even more engaging and effective. Ultimately, we envision SignQuest becoming a comprehensive platform for learning various sign languages, contributing significantly to breaking down communication barriers between the deaf and hearing communities worldwide.