EchoSign

Inspiration

EchoSign was inspired by the need to make video calls more accessible for individuals who are mute and deaf. Recognising the importance of inclusive communication in today’s digital world, we saw a gap in tools that could translate ASL into spoken language, especially in a conversational, natural way. Our aim was to empower users of American Sign Language, enabling them to communicate with fluency and ease.

What it does

EchoSign is a web tool that translates ASL into both text and spoken English, designed for use with video chat platforms like Zoom. It does more than just translate signs—EchoSign also adds grammar and sentence structure to produce spoken-like English, allowing individuals who are mute to engage in conversations seamlessly.

How we built it

We built EchoSign using the videos recorded by us as the foundation for ASL training data. For the model, we implemented a Multi-Layer Perceptron (MLP) network, carefully tuning it to recognize ASL gestures accurately. EchoSign’s interface was developed using Streamlit to ensure a simple, accessible user experience. Though it’s not directly integrated with Zoom, EchoSign can easily be used alongside any video chat platform.

Challenges we ran into

Creating EchoSign came with unique challenges. Translating ASL in real time required robust data processing, model tuning, and precise data management. Making the output sound natural with correct grammar also posed a challenge, as ASL has a unique syntax. We had to experiment with multiple AI configurations before settling on an MLP model that met our accuracy and usability needs. We also experimented with WASL dataset before deciding on using our videos for the training data.

Accomplishments that we're proud of

We’re proud of building a functional and accessible ASL translation tool that can be used with video chats. EchoSign’s ability to add grammar and enhance naturalness in the output is something we’re especially proud of, knowing it makes the tool more practical for real-life conversations.

What we learned

This project taught us the entire process of developing an AI model from scratch, including data processing, model selection, training, and refinement. Using the videos we made and an MLP model, we gained hands-on experience in handling complex data and designing a solution that could be used in real-world settings. Building a user-friendly interface with Streamlit also highlighted the importance of usability in making technology accessible.

What's next for EchoSign

Our next steps for EchoSign involve improving accuracy and expanding compatibility with other video platforms. We’re also exploring the possibility of translating additional sign languages to make EchoSign a versatile tool for global users.