articulator by Inclusive Innovators
NSBEHacks 2023 Karl Zhu, Andy Wang, Jessica Tang
In our audible society, those with hearing and speech impairments constantly live in frustration and agony. They face a massive language barrier - if they want to shop or dine out, they can only communicate with those knowing the sign language. Many times, staff are eager to assist the customer with hearing loss, but don't know the sign language.
Articulator is the tool to surpass this barrier. Made with Cohere AI, our clients - staff workers use Articulator to turn their words into American Sign Language to assist their customers. Simply speak into the microphone, and a video composed of the correct sign language will play and allow the customer to understand your thoughts. Because sign languages heavily uses body expressions, Articulator displays a sentiment emoji to convey the sentiment of the sentence, helping the customer better understand your words.
Articulator uses 6 different AI models. First, a speech recognition model converts your words to text. Then, a Cohere summarizer is used to simplify long speeches into digestible words. The text is then passed to a Cohere toxicity classifier, sentence by sentence, to detect noninclusive and toxic sentences, and all harmful sentences are passed through a Cohere generator to rephrase and clarify it. Finally, Cohere sentiment analysis is performed to gauge the sentiment of the text, and words or word groups are matched with a library of sign language videos. For words not found in the library of videos, we use Cohere Embed to find the closest word to replace it semantically.
Log in or sign up for Devpost to join the conversation.