ASLingo: The Smart Learning Platform

Jump into ASLingo, the gamified way to learn American Sign Language! Get instant feedback, crush mini-games, and level up your skills. No matter your level, this is your all-in-one ASL platform.

Comment 1

Inspiration

Everyone deserves a voice; However, learning American Sign Language later in life can prove to be a challenge. Especially when so many words look so similar. You wanted to buy an apple, but instead you end up with... an onion?!

We're here to revolutionize how individuals learn ASL. No more looping videos trying to get that one word correct; instead, we walk the user through exact modifications they can make to learn their target word!

What it does

ASLingo offers an interface that allows users to learn at their own pace. They go through our customized lesson plans, watch videos of signs, and try to recreate them! We analyze the users' gestures through a custom-trained AI model on ASL gestures and provide real-time feedback. Learners get step-by-step, real-time guidance on exactly how to adjust their hands, palm orientation, or motion to reach the target sign. That's not all! We also provide opportunities for users to challenge themselves with levels and minigames, giving individuals an incentive to keep learning. -Guided lessons with short, focused sign videos. -Practice mode, where you perform a sign on camera. -Real-time feedback from our custom gesture model -Progress tracking so your wins and XP are saved to your profile. -Friendly competition with streaks, leaderboards, and challenges to keep motivation high.

How we built it

We provide a user-friendly frontend interface with React.js, where users can then create accounts and log in with Auth0. Their progress and data are saved to MongoDB. We connected to our backend using Express.js and deployed our website with Vercel.

Backend & API: -Node/Express (on Vercel) for routing, auth/session handling, and a thin API layer to the model service. -MongoDB stores user profiles, lesson state, and attempt history.

AI Model Service (Python): -MediaPipe + OpenCV extract 3D hand landmarks from the webcam stream. -A PyTorch sequence model (LSTM/GRU) classifies the sign and computes feature-level deltas (e.g., finger curl, wrist rotation, relative joint distances). -We convert those deltas into human-readable coaching tips (e.g., “curl middle finger slightly,” “move hand 3 cm right”). -FastAPI serves low-latency inference endpoints; responses include both label + confidence and actionable corrections.

DevOps & Deployment: -Vercel for the frontend + Node API. -Containerized Python model (FastAPI) running on a separate service. -Environment-based config for Auth0/MongoDB.

Challenges we ran into

-Vercel: understanding databases, Auth0, connectin the backend to the vercel. -Landmark drift & jitter: stabilizing MediaPipe output so micro-tremors don’t trigger noisy feedback. -Training AI model: acquire a large enough dataset to train an accurate model, and find the hyperparameters (e.g. learning rate) that fit our dataset. -Learning: We have used algorithms and concepts that we have never used before, like Vercel, MongoDB, or training an AI model. Getting them to work together—securely and with low latency—was one of our toughest hurdles. -Docker Middleware: Connecting our Python ML backend to our React frontend came with many challenges. This resulted in us exploring Docker, hosting our ASL Gesture Recognition on Render, a Web Service, which could then be called by our frontend.

Accomplishments that we're proud of

  • Building an AI model from scratch to recognize ASL by providing training data and running supervised learning to classify gestures
  • Implementing Verecl and MongoDB to build an authentication system
  • Designed a user-friendly UI with a welcoming and distraction-free layout that keeps learners focused.
  • Built a UX that creates an accessible environment, such as our bilingual(EN/FR) system
  • Incorporating Gemini to provide real-time feedback to users on their accuracy with certain ASL gestures

What we learned

  • How to use different Python libraries such as numpy, pandas, and TensorFlow
  • How to obtain the dataset, process the data, and train our data to build a custom BiLSTM (PyTorch) sequence model for real-time ASL sign classification.
  • How to build the frontend with React.js and CSS to implement things like animated components, webcam capture.

What's next for ASLingo: The Smart Learning Platform

Further training the model, using open-source data on gestures Create more interaction between users; gamify it more; include challenges (PvP?!) Sentences, grammar, etc.

Share this project:

Updates