Manibus

Manibus is an AI tool that enables self-driven learning for deaf students in English and American Sign Language, as well as helping families provide critical visual language exposure for deaf children.

Inspiration

While looking for ideas, we came across the fact that more than 90% of deaf high school students have a reading comprehension level lower than that of a seventh grader by the time they graduate.

While fluency in American Sign Language (ASL) has shown to have a direct correlation with better reading comprehension for deaf students, deaf students often lack access to ASL instructors and educational materials.

Furthermore, most deaf children do not reach conversational proficiency in either spoken or signed languages. Deaf children born to hearing parents, are more likely to not receive sufficient visual language input to acquire or learn languages and develop linguistic capabilities at critical moments in their infancy.

We thought this would be an interesting problem that would be particularly suited for an AI solution.

What it does

Manibus does two things. For one, it increases the amount of ASL exposure deaf children with hearing parents receive at home by translating speech to ASL.

Secondly, it enables deaf students to engage in self-driven learning with an interactive reader that translates text to ASL side-by-side, providing opportunities for deaf students to practice and learn both ASL and English outside of limited language instruction times in school.

How we built it

Manibus is a React + Vite web application with a FastAPI backend. Because Manibus can accept both speech and text for input, some additional steps were needed to convert speech effectively into text, and consequently ASL.

Upon picking up speech, Web Speech API transcribes the content of the speech, while MediaPipe and Web Audio are used to recognize different traits of the speech and the speaker like facial expressions and voice tones.

Gemini then translates the received input (text and emotion derived from text, or text, tone, and facial expressions for speech) into the appropriate ASL output by referencing data from ASL-LEX, an open-source dictionary indexing >2700 different signs, with the ASL output being translated back to English to be checked for fidelity in translation.

For the signs, Manibus was designed to play signs as a blender-made animation in three-js., crossfaded with a live face.

Challenges we ran into

Making a 3D avatar sufficiently detailed and able to imitate sign language turned out to be more difficult than we expected, but fortunately it turned out to be the only major obstacle we faced in development.

Accomplishments that we're proud of

The topic and field of the problem we decided to tackle, while interesting, was ultimately unfamiliar to us, and seeing our idea grow from a rough sketch in Google Docs to a live, breathing product was an especially proud feeling that we all shared.

What we learned

The key elements of implementing Manibus involved manipulating visual elements in the form of a 3D avatar in Blender linked to an MCP that handled the ASL translation. In the development process, we learned a lot about integrating 3D animations onto web applications.

As for hacking, we learned that an interesting idea, four hackers, and a table are all you need to spend an exhilarating weekend.

What's next for Manibus

Next steps for Manibus would be improving its translation performance, possibly by using python scripts to truly enable continuous signing, making 3D avatars more life-like and customizable by users, as well as increasing the corpus of ASL data Manibus draws on.

Built With

Share this project:

Updates

Submission history