Handy

"Learn Sign Language, One Sign at a Time"

Comment

Inspiration

Based on data from the Indonesian Ministry of Health, there are 18.9 million Indonesians with hearing disabilities. Of these 18.9 million people, not all of them can do activities and work like non-disabled people. This is not because they are not capable, but we have not been able to realize an inclusive society. Most people have difficulty communicating with people with hearing disabilities due to the difficulty of learning and memorizing sign language. This is where we offer Handy. An interactive and competitive American Sign Language (ASL) learning application. Here we provide many interactive and fun learning units for people to easily understand. We also provide a competitive mode to keep challenging people's knowledge of sign language.

What it does

Handy transforms traditional sign language education into an engaging experience through gamification. Users progress through various levels, compete in rated multiplayer games, and see their progress on leaderboards. This approach keeps users motivated and makes learning enjoyable.

How we built it

We designed our application as a Progressive Web App (PWA) to combine the best of web and mobile apps. PWAs offer features like offline access, push notifications, and home screen installation, providing a native-like experience while maintaining the reach of the web.

Frontend

We chose React for our frontend due to its component-based architecture, which allows for reusable UI components and efficient rendering with shadcn-ui for our ui frameworks library.

To enhance our development experience, we utilized Vite as our build tool. Vite provides a fast and optimized development environment by leveraging native ES modules, which significantly reduces build and hot-reload times. This allows us to iterate quickly and maintain a smooth development workflow.

We also opted for typescript to provide type safety and improved code quality. By using TypeScript, our codebase will be more robust and maintainable, lastly, we can catch errors early in the development process.

Additionally, we incorporated Lottie files for animations, which enables us to add high-quality, scalable animations to our UI without sacrificing performance. Lottie’s lightweight nature helps enhance user engagement while maintaining a smooth user experience.

Backend

For our backend, we chose NestJS, it is built with TypeScript and provides out-of-the-box support for dependency injection, making it an excellent choice for building robust server-side applications. For real-time communication, we integrated Socket.IO with our NestJS backend and React frontend. Socket.IO enables bidirectional event-based communication between the server and clients, which is crucial for features such as queues and matchmaking.

To handle file uploads, we utilized Multer, a file upload middleware for Node.js. Multer help to store multipart/form-data, which is used for handling profile picture uploads in our application.

For scheduling and automating tasks, we implemented a Cron scheduler. This enables us to run periodic jobs, such as finishing unfinished games, ensuring our application runs smoothly and efficiently.

Database

We choose PostgreSQL as our database for its robustness, scalability, and support for advanced SQL query features. To interact with our PostgreSQL database, we use Prisma as our ORM (Object-Relational Mapping) tool. Prisma simplifies database access by providing a type-safe intuitive query builder and also includes a schema modeling tool that allows us to define and migrate our database schema seamlessly. This enhances productivity and reduces the likelihood of database-related errors.

AI

To enhance our computer vision capabilities, we used MediaPipe, a framework developed by Google for building cross-platform machine learning pipelines. MediaPipe simplifies the development of complex machine learning workflows.

Additionally, we created our own model using datasets that we sourced and cited from Kaggle. By utilizing a couple of Kaggle datasets, we identified the best datasets for our needs. For example, we used MNIST as a foundational dataset before proceeding with the creation of our latest model.

Cloud Infrastructure and CI\CD Implementation

To ensure efficient development and deployment, we have implemented CI/CD using GitHub Actions, Docker, and a VPS/cloud service provided by Niaga Hoster.

For continuous integration and deployment (CI/CD), we utilized GitHub Actions to automate our workflow. GitHub Actions handle code testing, Docker image building, and automatic deployment to our VPS/cloud service hosted by Niaga Hoster. We containerized our application with Docker, ensuring consistent performance across environments, and used Docker Compose on our VPS for efficient container management and uptime.

Challenges we ran into

At the beginning of our development journey, our team faced some significant challenges. Initially, we were uncertain about how to make the necessary resources freely available and struggled with the complexities of running our model on the client side. However, after extensive research and experimentation, we discovered MediaPipe. This powerful framework not only streamlined our machine learning workflows but also provided the tools we needed to implement our sign language recognition model effectively on the client side.

Accomplishments that we're proud of

Gamified Learning

We transformed traditional educational content into interactive and engaging experiences by incorporating game design elements. With Handy, users are encouraged to learn sign language through various gamification strategies, such as levels, rated multiplayer games, and leaderboards.

Realtime Matchmaking

One of the key parts of our app is the matchmaking algorithm. We want users to be able to play as quickly as possible, and we made this happen by using WebSocket technology.

Sign Language Recognition

This is our cutting-edge feature that recognizes and interprets sign language gestures. We used OpenCV and MediaPipe to enable users to practice and show their sign language gestures to the camera. By making the process real-time, our app makes the learning experience more interactive and effective. Moreover, our model is optimized to be lightweight, using only 8 MB of model size. This ensures that it can run smoothly on any kind of browser, making it accessible and convenient for users regardless of their device capabilities.

What we learned

This project taught us a lot about building a complex application with multiple and diverse technologies. We discovered how crucial it is to choose the right tools and frameworks. React helped us create a dynamic frontend, while NestJS provided a solid backend foundation. MediaPipe was a game-changer for running our machine learning model on the client side. We also learned the benefits of TypeScript in catching errors early and how Prisma made database management smoother. Integrating Socket.IO showed us the importance of real-time communication, and Multer simplified file uploads for user profiles. Overall, this competition highlighted the importance of research, adaptability, and choosing the right tools to overcome challenges and build a successful application.

What's next for Handy

Personalized Learning Paths

Each user’s learning journey is unique we want to provide personalized learning paths based on individual progress and goals. This customization ensures that users can learn at their own pace and focus on areas where they need improvement.

Community and Social Features

We want a sense of community among users. Through forums, chat rooms, and group challenges, users can interact, share their experiences, and support each other in their learning journey. This social aspect enhances motivation and engagement.

Daily Goals and Streaks

To encourage consistent practice, we want to set daily learning goals and reward users for maintaining learning streaks. This feature helps users develop a regular practice routine, which is essential for mastering sign language.

Progress Tracking and Analytics

We want to provide detailed progress tracking and analytics so users can monitor their improvement over time, see which signs they have mastered, and identify areas that need more practice. This data-driven approach helps users stay on track and achieve their learning goals.

Built With

Share this project:

Updates