Msign

• 

  the site

Inspiration

The inspiration behind Msign comes from the growing need for inclusive technology that helps bridge the communication gap for individuals who are deaf or hard of hearing. With more than 70 million people worldwide using sign language, there is a pressing need for tools that can help others learn and translate sign language into text seamlessly. We wanted to create an intuitive and accessible platform to help both learners and non-signers communicate more effectively using modern technologies like machine learning and computer vision.

What it does

Msign is a web-based sign language learning and translation platform that allows users to:

Learn American Sign Language (ASL) through interactive exercises and gesture recognition.
Translate real-time gestures into text using a webcam-based detection system.
Receive feedback on their gesture accuracy and improve their signing skills.
Provide quick and accurate translation from sign language to text, helping with communication for non-signers.

The app utilizes machine learning models to detect hand gestures from the webcam and translate them into readable text, bridging the gap between sign language users and non-signers. How we built it

Msign was built using:

Frontend: HTML, CSS, and JavaScript for the user interface to capture webcam input and display gesture translation results in real time.
Backend: Flask (Python) for handling the API that processes gestures and sends them for translation.
Machine Learning: We integrated machine learning models that process the webcam feed, recognize ASL gestures, and translate them into corresponding text. We used libraries like MediaPipe for hand tracking and PyTorch to power the translation model.
Deployment: We deployed the project using Heroku to ensure it can be accessed via a public URL, allowing users to learn and translate gestures online from any device.

Challenges we ran into

Some of the major challenges we faced while developing Msign were:

Real-time gesture recognition: Ensuring that the webcam feed was processed accurately and efficiently in real time was tricky. We needed to optimize the model to work quickly without compromising on accuracy.
Model accuracy: Training the model to distinguish between subtle differences in hand gestures required fine-tuning. In particular, differentiating between similar-looking signs was challenging.
User interface: Creating an intuitive and responsive UI that worked well across different devices and screen sizes posed some challenges. We needed to ensure the learning experience remained smooth on mobile, tablet, and desktop.
Latency in translation: Reducing the delay between when a gesture is made and when the translation appears was a major technical hurdle, especially for real-time use cases.

Accomplishments that we're proud of

Real-time gesture translation: One of the most significant achievements is successfully implementing real-time sign language gesture translation using a standard webcam.
User-friendly design: We’re proud of how intuitive and accessible the platform turned out. Both sign language learners and non-signers can benefit from the platform’s easy-to-use interface.
ML integration: Incorporating machine learning into the project and ensuring the model performs well across different users and hand gestures was a significant technical accomplishment.
Webcam integration: We managed to create a seamless experience where users can interact with the platform using their webcam, offering a real-time learning experience.

What we learned

Throughout the development of Msign, we learned:

The importance of model optimization: Achieving real-time performance with accurate gesture detection required a deep understanding of model optimization, including fine-tuning and reducing the computational load.
User-centered design: Creating an accessible and easy-to-use interface requires constant feedback from potential users to ensure that the platform meets their needs.
Team collaboration: Building a project that integrates frontend, backend, and machine learning components taught us a lot about cross-functional collaboration. Each part of the team had to work closely to ensure a smooth, integrated experience for users.
Machine learning in web applications: We gained a lot of practical experience in deploying machine learning models in a web environment and handling live data processing.

What's next for Msign

We have big plans for Msign in the future:

Expanded sign language support: We aim to support other sign languages beyond ASL, starting with British Sign Language (BSL) and International Sign.
Mobile app: We plan to develop a mobile app to make learning and translating sign language even more accessible on the go.
Advanced learning modes: Incorporating quizzes, progress tracking, and a more extensive set of signs to provide a comprehensive learning experience.
Community engagement: We plan to build a community where users can practice signing with others and receive feedback from sign language experts.
Voice integration: Eventually, we want to support voice-to-sign and sign-to-voice translation to make communication even more seamless.

Built With

• css
• flask
• html
• javascript
• python
• roboflow