MOUNA

A Unified Two-Way Communication Interface for Deaf and Mute Individuals On Digital Platforms

Comment

Inspiration

Communication should not become a barrier simply because someone communicates differently.

We were inspired by the everyday challenges faced by deaf and non-speaking individuals, especially in education and professional environments where communication increasingly happens through online meetings and video content. While platforms such as Google Meet make remote communication easier for most people, many deaf users still depend on interpreters, subtitles, or separate assistive tools to understand spoken conversations.

We wanted to explore a simple question:

What if online communication could automatically adapt to the way a person communicates?

That idea led us to MOUNA, an AI-powered Chrome extension designed to make online meetings, video calls, and video-based learning more accessible.

What it does

MOUNA is a real-time assistive communication extension that creates a bridge between spoken language, text, and sign language.

During an online meeting, when a hearing participant speaks, MOUNA captures the speech and converts it into meaningful sign-language representation through an animated 3D avatar displayed on the user's screen. This allows deaf users to follow the conversation without depending entirely on subtitles.

MOUNA also works in the opposite direction. A deaf or non-speaking user can communicate using sign-language gestures through their webcam. The system recognizes the gestures and converts them into text and synthesized speech, allowing other participants to understand and respond naturally.

Accessibility beyond meetings

One of the features we wanted to introduce was accessibility for online tutorials, lectures, and other video content.

When MOUNA is enabled while watching a video, the extension listens to the spoken content and converts the speech into sign-language representation through the animated avatar. This means deaf users can access audio-based tutorials and educational videos without depending entirely on subtitles.

Our goal is not simply to add another translation tool, but to make existing online content more accessible without requiring users to purchase additional hardware.

How we built it

MOUNA is designed as a Chrome extension, allowing it to work alongside commonly used online platforms rather than requiring users to switch to a completely different application.

The system follows two primary communication pipelines:

Speech → Text → Sign Representation → 3D Avatar

and

Sign Gesture → Text → Speech

For spoken content, audio is processed using speech-recognition technology to obtain the spoken text. The recognized text is then converted into a representation suitable for sign-language animation, which is presented through a 3D avatar.

For sign-language input, the user's webcam captures hand and body movements. Computer-vision techniques are used to identify the gestures and map them to corresponding words or phrases. The resulting text can then be converted into speech, so that hearing participants can understand the user.

The Chrome extension acts as the interface connecting these components with online meetings and video content.

Challenges we ran into

One of our biggest challenges was the complexity of sign language itself.

Sign language is not simply a word-by-word replacement for spoken English. Meaning can depend on hand movement, position, facial expressions, body posture, and context. This made accurate real-time sign recognition and generation much more challenging than a conventional speech-to-text system.

Another challenge was maintaining a balance between accuracy and response time. Processing audio, recognizing gestures, generating language representations, and animating a 3D avatar can introduce delays. We therefore had to think carefully about how different components communicate with each other.

Integrating the system into a browser environment was another challenge. MOUNA needs to operate alongside meeting platforms and video content while remaining simple enough for users to activate without complicated setup.

Accomplishments that we're proud of

We are proud that we transformed MOUNA from an idea into a working AI-powered Chrome extension concept for accessible online communication.

Some of the accomplishments we are particularly proud of are:

  • Built a Chrome extension-based accessibility solution that can be integrated into online meetings and video content without requiring dedicated hardware.
  • Created a two-way communication concept, supporting both speech-to-sign and sign-to-speech, rather than solving communication in only one direction.
  • Integrated an animated 3D avatar to represent spoken conversations through sign-language animations, making the experience more visual and engaging.
  • Extended the system beyond online meetings by developing the concept of using MOUNA while watching tutorials, lectures, and other video content, allowing spoken audio to be represented through the avatar.
  • Explored the integration of speech recognition, AI language processing, computer vision, sign-language recognition, text-to-speech, and 3D animation into a single workflow.
  • Designed MOUNA with real-time interaction and low latency as important goals, since delays can significantly affect natural communication.
  • Most importantly, we are proud that our project is built around a real accessibility problem rather than technology alone. We wanted to create something that could help deaf and non-speaking users participate more independently in digital communication and learning.

For us, the biggest accomplishment is bringing together multiple AI and web technologies to work toward a single goal: making digital communication and educational content more inclusive.

What we learned

Building MOUNA taught us that creating an assistive AI system is much more than connecting different AI models together.

We learned about:

  • Speech recognition and speech-to-text processing
  • Computer vision and gesture recognition
  • Sign-language representation
  • 3D avatar animation
  • Browser extension development
  • Real-time communication pipelines
  • Human-computer interaction and accessibility
  • Designing AI systems around actual user needs rather than only technical capabilities

We also learned that latency is extremely important. A system can be technically accurate but still become difficult to use if the translation takes too long. This made us focus on keeping the communication pipeline as close to real time as possible.

What's next for MOUNA

MOUNA is built around a simple idea:

Accessibility should be part of the communication platform, not an additional barrier around it.

We envision MOUNA becoming a lightweight accessibility layer that can work across online meetings, educational videos, tutorials, and other digital content.

In the future, we want to:

  • Improve sign-language recognition
  • Support more sign languages
  • Improve avatar animation and facial expressions
  • Reduce translation latency
  • Expand compatibility with more online meeting and video platforms
  • Make the system available to a wider range of users

Ultimately, we want MOUNA to help create a digital environment where people can participate in conversations and access knowledge regardless of how they communicate.

Built With

Share this project:

Updates

Submission history