HearMeSign

Inspiration

We have multiple friends who use ASL, and we realized how often people want to communicate but don’t know where to start. We wanted to create a fun, interactive, and accessible way to learn ASL that lowers the barrier for beginners and encourages more inclusive communication.

What it does

HearMeSign analyzes your hand shape and movement in real time and translates them into ASL letters. It supports both static signs (A, B, C, etc.) and dynamic signs like J and Z, allowing users to spell full words naturally.

How we built it

We built HearMeSign in Python using MediaPipe for hand‑landmark detection and custom logic to classify each sign.
We created a dataset of ASL letters, calculated ideal distances between key hand landmarks, and built a tolerance‑based matching system. For dynamic letters, we implemented multi‑frame sequence detection so the program can recognize motion‑based signs.

Challenges we ran into

• Getting consistent hand‑landmark data across different lighting and angles
  
• Designing a classification system that works for both static and dynamic signs
  
• Handling noisy frames and stabilizing detection so letters don’t flicker
  
• Building a clean UI flow that lets users label new signs quickly
  
• Managing timing and frame buffering for letters like J and Z
  

Accomplishments that we're proud of

• Successfully recognizing both static and dynamic ASL letters
  
• Creating a real‑time system that feels smooth and responsive
  
• Designing our own dataset and classification logic from scratch
  
• Building an accessible tool that genuinely helps bridge communication gaps
  
• Finishing a functional prototype under tight hackathon time pressure
  

What we learned

• How to work with MediaPipe’s hand‑tracking pipeline
  
• How to design a gesture‑recognition system using distances and tolerances
  
• How to handle multi‑frame motion detection
  
• How to collaborate quickly and iterate under time constraints
  
• The importance of accessibility‑focused design
  

What's next for HearMeSign

• Adding full ASL alphabet support, including more dynamic signs
  
• Implementing word‑level recognition and predictive text
  
• Building a cleaner UI with visual feedback for each detected letter
  
• Creating a learning mode with tutorials and practice exercises
  
• Expanding to full ASL gestures, not just letters
  
• Deploying as a web or mobile app so anyone can learn ASL anywhere
  

Built With

• json
• python