CoSign

CoSign

Inspiration

We wanted to help people learn ASL so they can connect with the Deaf people in their lives. Many learners can’t attend in-person classes, and passive videos don’t give them feedback or motivation to keep practicing.

What it does

CoSign teaches ASL through short, interactive lessons. Learners watch a sign demonstration, practice in front of their webcam, and get feedback on whether the sign was recognized. XP and lesson progress encourage them to keep learning.

How we built it

The web app uses React, TypeScript, and Vite for the learning experience, with MediaPipe to track body and hand landmarks through the webcam. A Python FastAPI backend preprocesses those landmarks and runs a pretrained TensorFlow Lite ASL recognition model. Lesson content is stored in JSON, and user performance and XP is tracked using SpacetimeDB to display a leaderboard of the users who learn fastest.

Challenges we ran into

Recognizing signs reliably means handling differences in lighting, camera angle, framing, and how people perform a sign. We also needed to connect real-time camera tracking to the model’s expected input and turn recognition results into clear lesson feedback.

Accomplishments that we're proud of

We brought webcam-based ASL practice, recognition, and lesson progression together in one learning flow. The design turns watching a demonstration into active practice, with retries and progress tracking built in.

What we learned

We learned how to connect browser-based landmark tracking to a pretrained recognition model, and how much the quality of a learning experience depends on clear feedback and thoughtful handling of real-world camera conditions.

What's next for CoSign

We want to expand the lesson library, improve feedback for signs that are close but not quite right, and make practice more resilient when camera access or tracking conditions aren’t ideal. We also plan to involve Deaf educators and reviewers in shaping and validating the learning content.