Bridging Silence: Malaysian Sign Language Recognition

Real-time MSL hand-gesture recognition built for Malaysians, filling a gap where no accessible tech exists. It empowers the Deaf community by turning sign gestures into instant readable text.

Comment 6

Inspiration

Growing up in Malaysia, I became aware of a silent but very real communication barrier faced by our Deaf and Hard-of-Hearing community. Malaysian Sign Language (MSL) is their primary mode of communication , yet almost no one outside the Deaf community understands it. What shocked me even more was ,there is no accessible, real-time MSL recognition tool available anywhere not in Malaysia, and not globally. ASL recognition tools exist, but Malaysian Sign Language has completely different gestures, alphabets and grammar. MSL users have been largely overlooked by mainstream assistive technology.

What it does

This gap inspired me to build a solution specifically for Malaysians, designed around the actual needs of our local Deaf community: ✔ real-time ✔ webcam-based ✔ works on any device ✔ converts hand signs to readable text ✔ optional speech output ✔ 100% free and accessible

How I built it

I combined computer vision, deep learning, and web streaming technology to create a full end-to-end assistive system.

  1. Data Collection (MSL-Specific) :

  2. Because no public datasets exist for MSL, I recorded my own datasets using MediaPipe landmark extraction and annotated each sample manually.

  3. Feature Extraction :

  4. Used MediaPipe Hands to extract 3D coordinates (x,y,z) of 21 hand landmarks.

  5. Applied normalization, scaling, and preprocessing for consistent model input.

  6. Deep Learning Model :

  7. Built and trained a fully connected PyTorch neural network to classify MSL alphabet gestures.

  8. Implemented a prediction stability buffer (rolling window) to avoid flickering output and ensure only stable predictions are displayed.

  9. Real-Time Inference Pipeline:

  10. Flask + Socket.IO server to handle frame streaming.

  11. The client sends webcam frames; the backend runs inference and sends predictions instantly.

  12. Added optional TTS using pyttsx3 so the recognized letter/word can be spoken aloud.

  13. Deployment :

  14. Containerized everything using Docker.

  15. Deployed on Hugging Face Spaces to allow anyone to try the system instantly.

  16. Designed the UI to be clean, minimal, and mobile-friendly.

Challenges I ran into

  • No publicly available MSL dataset → had to create my own.
  • Much fewer resources compared to ASL research.
  • Getting MediaPipe + PyTorch + WebSockets to run together in real-time.
  • Ensuring stable predictions despite hand jitter, angle changes, and lighting differences.
  • Deploying the full stack inside Docker on CPU-only environments. Each obstacle pushed me to learn more about computer vision pipelines, model optimization, asynchronous communication, and containerization.

Accomplishments that I am proud of

  • Built the first open, fully functional real-time Malaysian Sign Language recognition tool.
  • Successfully trained a deep learning model using my own MSL dataset.
  • Achieved real-time inference through a compact and efficient pipeline.
  • Made the entire project free, accessible, and online, so anyone in Malaysia can use it instantly.
  • Created a working prototype that can actually help people communicate.
  • Learned full-stack deployment, containerization, and real-time streaming from scratch.

What we learned

  • How to build end-to-end machine learning systems, not just train models.
  • The importance of accessibility-focused design.
  • Real-world challenges of dataset creation and annotation.
  • Using MediaPipe and PyTorch together efficiently.
  • Managing asynchronous communication with Socket.IO.
  • Deploying AI applications inside Docker and on cloud platforms.
  • Understanding the lived experiences and communication needs of the Deaf community.

What's next for Bridging Silence: Malaysian Sign Language Recognition

  1. Word-Level & Sentence-Level Recognition : Move from alphabet recognition to full MSL vocabulary.

  2. Dataset Expansion : Collaborate with Deaf associations to collect more diverse and accurate data.

  3. Real-Time Bi-Directional Communication : MSL → Text → Speech, and Speech/Text → Animated Sign Language avatars.

  4. Accessibility App : Package the system as an Android/iOS app usable offline.

  5. Multi-User / Classroom Mode : Enable real-time captioning during lessons, meetings, or public interactions.

  6. Open-Source Dataset Release : Create the first open Malaysian Sign Language dataset for the world.

My goal is to turn this prototype into a fully deployable assistive tool that empowers Deaf Malaysians wherever communication matters.

Built With

Share this project:

Updates