Leepi

Our app helps people learn American Sign Language. The app gives instant feedback for the signs we make in front of the camera, the only app that does it.

Comment 1

Pitch

Leepi wants to make learning sign language accessible and free for everyone! Learning happens with the help of an avatar who performs precise hand movements for everything from basic letters symbols to complex gestures for daily use. It is the only app where you can perform the hand gestures in front of the camera and get feedback in real-time.

Our learning system uses short and crisply designed walk through of lessons and uses badges and stars as rewards to motivate you through your journey!

Inspiration

When our team interviewed the target user(students) for our app we encountered a common situation that he/she never understood the English language structure. They have a hard time practicing gestures around others. They don’t understand books if there are no illustrations in it. Sometimes the school would also lack the infrastructure to provide an advanced setup for gesture learning. If a camera-based application is available where the user can practice simultaneously learning the English language with graphics, it would not be a boring task.

What it does

The app uses on-device machine learning models for analyzing hand gestures. The hands and fingers are detected in an input video from the front camera. They are then fed to another model for performing gesture recognition. All this happens in real-time due to pruned models. The video is completely inferred on the phone without any need of sending it to the cloud. The major features apart from self-learning for the user which app provides are speedy feedback, built-in privacy, zero network data usage.

Challenges we ran into

  • Data collection
  • Training for on-device models
  • UX for the application
  • Scalability over all words, sign and language

Accomplishments that we're proud of

  • Interactive learning structure
  • Pocket size lessons for a fun experience
  • Gamification for motivation: badges and stars
  • Processing happens all in mobile, no data usage required
  • Protects your privacy as the data never leaves the device

What's next for Leepi

Learning a new language is mostly dependent on the inclination and capability of the user. If the learning procedure is fun and initiates the user to practice continuously, the process is flawless. The app will keep on trying to improve the same way of learning. The backend is a machine learning model. It will improve with users with continuous training and validation. On the business side, partnerships with communities involved in the field and adding more user base for different languages will be the priority.

Announcement

Leepi( लिपि ) has been released on 15 June as a part of winning Google Android Developer Challenge 2020. By the support from Google's technical and design teams, Leepi( लिपि ) has been made keeping privacy, scalability, reliability and user learning experience in picture.

Google has been promoting this application by showcasing it across the world. Various reviewers have received customized package and Android Developer magazine as a part of promotion. An online landing page is also made for the same. Here are the related articles to it -

https://developer.android.com/helpful-innovation#leepi-section

https://www.blog.google/products/android/developer-challenge-winners/

https://android-developers.googleblog.com/2020/06/dev-challenge-winners.html

Built With

Share this project:

Updates