3D Avatar-Based Sign Language Learning and Conversion System

SignKit transforms speech, text, and video into dynamic 3D sign language, leveraging AI and avatar technology to make communication truly inclusive and accessible for everyone.

Comment

Inspiration

The inspiration behind SignKit was to bridge the communication gap between hearing-impaired individuals and the general public. We wanted to create an accessible and intelligent system that makes sign language learning and communication easier using modern AI and 3D technologies.

What it does

SignKit converts speech, text, or videos into Indian Sign Language (ISL) using an AI Glossing Model and displays the output through a 3D animated avatar. The system processes user input from any of these sources, generates ISL gloss, and renders real-time sign language gestures to enhance accessibility, communication, and learning.

How we built it

We built the system by integrating Speech Recognition for voice-to-text conversion, Natural Language Processing (NLP) for text analysis, an AI Glossing Model for converting English text into ISL gloss, and a 3D Avatar Engine to animate sign gestures. Video input processing is included to extract content from live or recorded videos. A gesture database maps gloss words to corresponding animations, and an administrator module manages system updates.

Challenges we ran into

One of the major challenges was accurately converting English grammar into ISL gloss structure. Synchronizing gloss output with smooth 3D avatar animations was also technically complex. Ensuring real-time performance and maintaining gesture accuracy required careful system design.

Accomplishments that we're proud of

We successfully developed a working prototype that converts speech, text, or video into ISL gloss and displays it through a 3D avatar in real time. Integrating AI-based gloss conversion with 3D animation within a single system is one of our key accomplishments, demonstrating seamless, interactive sign language communication for enhanced accessibility and learning.

What we learned

Through this project, we gained practical knowledge in Artificial Intelligence, NLP, Speech Recognition, 3D animation integration, and system design. We also learned the importance of accessibility-focused technology development.

What's next for 3D Avatar-Based Sign Language Learning and Conversion System

In the future, we plan to improve gloss accuracy using advanced deep learning models, support multiple sign languages, enhance avatar realism, add mobile application support, and integrate real-time sign-to-text conversion for two-way communication.

Built With

  • ffmpeg.js-(video-processing)
  • firebase
  • git
  • languages:-javascript
  • nlp-libraries-(spacy
  • nltk)-platforms:-web-browsers
  • python-frameworks-/-libraries:-node.js
  • react-/-next.js
  • rest-apis-for-integration-other-technologies:-webgl
  • tensorflow.js
  • three.js-/-babylon.js
  • typescript
  • unity3d-/-unreal-engine-(for-3d-avatar-testing)-cloud-services:-aws
  • vercel-databases:-firebase-realtime-database-/-firestore-apis:-web-speech-api-(speech-recognition)
Share this project:

Updates

posted an update —

Excited to share that I’m participating in the Creator Colosseum Startup Competition with my project:

3D Avatar-Based Sign Language Learning and Conversion System

This project focuses on breaking communication barriers using AI-powered 3D avatars that convert speech into real-time sign language.

Key Highlights:

Real-time speech-to-sign conversion

Smooth and natural 3D gesture animations

Designed for railway stations, hospitals, airports, and public service centers

Scalable architecture for future web and mobile deployment

My goal is to build inclusive technology that empowers seamless communication for the deaf and hard-of-hearing community.

Accessibility is not optional — it is essential.

CreatorColosseum #StudentFounders #RealStartups #AI #Accessibility #SignLanguage #Innovation #TechForGood

Log in or sign up for Devpost to join the conversation.