We live in a wonderful world and communication plays a fundamental role in understanding it. Social media platforms are integral to modern entertainment, news dissemination, and interpersonal communication, but most of the existing communication formats are inaccessible to people who are deaf or hard of hearing (DHH), particularly those who primarily communicate via sign language (deaf signers). Because platforms rely heavily on audio and text, communication and accessing content poses challenges for millions of DHH people worldwide. Our inspiration is to make the virtual wolds a space where deaf signers are able to communicate their ideas to others and be more included. By means of technology we are striving to address the unique needs and challenges that arise due to their preference for an unwritten, visual language.
What it does
Our product enables accessibility and easy communication for deaf signers by utilizing the technology of hand gesture prediction, recognition, and real-time conversion to text via Snap-AR Lense, with further cross platforms integration (Zoom, FaceTime).
How we built it
Challenges we ran into
ASL relays information not only through hand shapes and motions, but also posture, and facial expressions. ASL does not have a standard written form, which can create barriers to using hand sign to text conversion. ASL has no standard expression and varies between the speakers, which poses a challenge for accuracy when converting to text.
Accomplishments that we're proud of
What we learned
What's next for ASL Sign real-time conversion
- Use machine learning and computer vision to improve accuracy of sign recognition - To provide a tool/editor that will allow these tech users to create their own signs to help them communicate more effectively.