Speak-Sign

Game Start Screen
Camera Setup Page
Lesson - Milk
Lesson - Eat
Lesson - Love

About Our Project

90% of deaf-born kids have hearing parents who often don't know ASL. In their formative years, parents and their children can face communication issues, making it harder for parents to understand what they are saying. Speak Sign is an interactive learning platform that allows kids and their parents to learn the basics of ASL early through basic storytelling exercises.

Technical Implementation

Our project consists of a frontend developed with ReactJS, a backend developed with Python and FastAPI, and a Mediapipe & OpenCV gesture estimation model to track hand signs.

Child-Centric Design: The frontend includes larger tactile elements, a cartoonish style, and a vibrant color palette to captivate younger children.

Real-time Processing: Each level contains a live camera feed sent to the backend via websockets.

Gesture Recognition: The backend runs the Mediapipe model to determine the respective word based on the user's hand sign.

Feedback Loop: Results are sent back to the frontend, allowing the user to proceed to the next level.

Security and Privacy

Security is a primary concern given the target audience of parents and children.

Current State: Due to the hackathon timeframe, the video feed is transmitted over websockets to a backend for processing.

Production Goal: Ideally, Mediapipe would run directly on the frontend so no sensitive video data is transmitted.

Data Protection: Future iterations would replace unencrypted websockets with the wss:// method to better protect all transmitted data.

Challenges Faced

Building this platform within a hackathon environment presented several obstacles:

Team Dynamics: Integrating a team member who joined the project late.

UI/UX Logic: Organizing the progression of levels on the front page for optimal user flow.

Model Optimization: Training our model to recognize words from hand signs efficiently without the high computational cost of a convolutional neural network (CNN).

Work Distribution: Balancing hackathon work divisibility with privacy concerns; a frontend-heavy project would have led to a contribution imbalance among team members.

Conclusion

Overall, we are very proud of what we have accomplished. We hope that Speak Sign brings necessary attention to accessibility and creates a more inclusive learning environment for families.