The Scenario

When Mr. XX, a friend of my mother, reached his 50s, he discovered he had a hearing problem. Concerned about communicating with his wife and son as his condition worsened, he asked me during a visit in June if I knew an affordable and simple way to learn basic sign language. He mentioned that commercial apps often require ongoing subscriptions, are too complex, or don't focus on the basics—making them less practical for his needs and others like him (Appendix 1). Understanding his concerns, I offered to develop a custom app tailored to his preferences: offline accessibility, simplicity, and functionality for learning basic sign language in a fun and engaging way. He saw great potential in the idea, even suggesting it could later become a commercial project and keeping it exclusive (restricted access, all user have to be verified) (appendix 1). We decided to collaborate, focusing on creating a practical solution for him, his friends, and the community.

Rationale for solution

The application helps beginners explore and learn American Sign Language (ASL) intuitively and accessibly. Its features include a graphical interface for searching ASL alphabet gestures, a Word Bank for common words, and an exercise feature for creating, practicing, and reviewing activities. An AI Detection feature was implemented to recognize ASL gestures. To build the app, I used Next.js and HTML for a fast, flexible, and offine-compatible front-end. Radix UI and Lucid React ensured a responsive, intuitive interface, while Python provided effcient programming and debugging for the back-end. Data is stored in a secure local database for cost-effectiveness and privacy. TensorFlow powers scalable and effective AI Detection, leveraging its compatibility and strong community support.

Suggestion

To begin, it was noted in particular during the last interview that there is room for improvement in the exercise feature by increasing the types and the number of quizzes available. Incorporating time challenges, matching gestures and even having comprehension activities will enhance the experience of the ASL application. Competing with other users for best scores as your uncle and aunt correctly proposed would also aid motivation and retention. Next, the AI detection tool feature, interesting as it is, could be modified to have less variation in accuracy when used in a different environment. The AI works well, but only under controlled conditions; AI improvement would ensure a user experience that would be equal across different lightings and settings. I also think about incorporating a gesture correction feature that identifies common errors and offers suggestions for improvement. This feedback mechanism would serve as a virtual coach, helping users refine their ASL skills more effectively.

To make the Word Bank more interactive, it was also suggested to use ASL hand gesture clips or images when users store words so that they can connect words with gestures. This connection would enhance learning and enable better recall of and memory for words. Incorporating a feature that provides users with real-time feedback during their ASL practice sessions would be highly beneficial. This could include motivational notifications, indicators to track progress, or even a reward system to encourage regular practice and reinforce positive results. We also add an informative section that highlights the significance of ASL, outlines best practices for learning, and discusses the research behind gesture detection would greatly enhance the app. Users could access this information through a dedicated button on the main screen, giving them valuable insights into the purpose and advantages of using the app. Furthermore, integrating a social learning component where users can collaborate or communicate via shared challenges or forums would promote a sense of community. This interaction could enhance motivation and provide peer support during the learning journey. These enhancements are designed to make the application more functional, user-friendly, and engaging, ensuring it caters to the diverse needs of its users and supports effective ASL learning.

Built With

• html
• javascript
• machine-learning
• next.js
• python
• tensorflow