Inspiration
Learning American Sign Language (ASL) takes practice, patience, and feedback. Beginners often struggle to know whether they are forming a sign correctly, especially when practising alone. We wanted to make that learning process more interactive and accessible.
What it does
SignCoach is a webcam-based ASL learning platform that acts as a personal signing tutor. It uses MediaPipe hand tracking, computer vision, and machine learning to recognize ASL alphabet handshapes and provide real-time feedback. Learners can explore guided lessons covering the alphabet, numbers, and everyday expressions, practise with quizzes, and track their progress.
How we built it
We combined MediaPipe hand landmarks with a TensorFlow.js neural network for alphabet classification and a rule-based correction engine for instructional feedback. The platform also includes interactive lessons, progress tracking, and a browser-based camera pipeline designed to keep raw video on the user's device. OpenCV 5 supports additional image-processing and vision functionality in the backend.
Challenges we faced
Reliable hand recognition is difficult because of finger occlusion, similar-looking handshapes, lighting, and camera positioning. We also had to balance real-time performance with useful feedback, and clearly distinguish recognized static handshapes from signs that involve movement.
What we learned
We learned how to combine computer vision, machine learning, and interactive web development into an educational application. We also learned that recognition accuracy alone is not enough: useful feedback, honest evaluation, privacy, and clearly documented limitations matter just as much.
What's next
Future improvements could include more robust recognition of visually similar signs, evaluation with a broader range of signers, feedback reviewed by fluent Deaf ASL users, and support for more dynamic signs and conversational practice.
Built With
- amazon-web-services
- fastapi
- mediapipe
- opencv-5
- python
- react
- tensorflow.js
- typescript
- vite
Log in or sign up for Devpost to join the conversation.