SonicSense

Inspiration

In our world of rapid technological advancement & automation, we have successfully progressed the lives of millions, yet, we have often forgotten those who needed the help the most. According to the World Health Organization (WHO), there are currently 1.5 Billion people worldwide living with hearing loss, 430 million of which is disabling. Despite 20% of our globe's population suffering, very little has been done to improve the quality of life for these individuals. We knew we had to do something to fix this...

In line with the UN SDGs #1 (No Poverty), #3 (Good Health & Wellbeing) & #10 (Reduced Inequalities) in particular, we looked to create a solution that would provide those with hearing disabilities a way to enjoy life & receive the same opportunities as all of us.

What it does

With real-time spoken language translation into American Sign Language (ASL), our sign language translation tool transforms communication for those suffering from hearing impairments. Our program translates spoken words into ASL signals with effortlessly by leveraging AI technology methods dependent on Google MediaPipe Hand Landmarker, guaranteeing accessibility and inclusion in a range of settings. The program allows users to translate words into ASL by simply speaking into their device's microphone. This makes it easier for people to communicate in social, professional, and educational contexts. The program communicates with ASL using digitalized hand signals visible on the left side of the screen. Users can observe these hand signals to interpret the prompt, while simultaneously adding more audio input to the program.

How we built it

Frontend: Our website is designed utilizing HTML, CSS, & JS.

Backend: The majority of our backend & the spine of our product is built utilizing Python.

ASL-Specific Tools: We utilized the Google MediaPipe Hand Landmarker to map the coordinates of the hand which was utilized to subsequently interpret the ASL signs from recorded footage and/or produce ASL demo footage. We referenced the ASL Dictionary to match hand movements with their corresponding ASL interpretations.

AI Integration: We deployed OpenAI's Whisper to generate the transcribed script and/or ASL footage.

Challenges we ran into

We originally ran into a few struggles around integrating the AI framework into our project, however, after reading through manuals & watching tutorials we were able to successfully overcome this issue. We also faced a few time-consuming bugs which after thorough consultation with the available mentors, we were able to fix & subsequently move on with the project. Furthermore, we also encountered some issues with the UI/UX & creating visually-appealing designs for the users. We decided to go back to the drawing board for this, & utilized Figma to brainstorm ideas & prototype designs for what we wanted our platform to look like. This was definitely a step in the right direction, one step back, two steps forward if you will.

Accomplishments that we're proud of

The accomplishments and influence of our sign language interpretation software fill us with immense pride. We've created a technology that effectively overcomes communication obstacles and enables people with hearing impairments to participate completely in their environment via unceasing innovation and devotion. The smooth integration of AI technology into our program represents our dedication to accessibility and diversity while also facilitating real-time translation of spoken English into sign language. This technology has the potential to reach over 1.5 billion people suffering from hearing loss, enabling them to better communicate with their peers and more effectively digest media.

Perhaps the most profound impact of this technology is its ability to revolutionize the domain of video media and the news as a built-in sign language extension. This improved accessibility can have a profound impact on literacy rates as media is made simpler to consume and will improve the quality of life of people around the globe. We are immensely proud of our application and are inspired to keep pushing the boundaries of technological innovation for the benefit of society after seeing the transformative impact it has on the lives of innumerable users, from improving educational opportunities to fostering meaningful connections.

What we learned

Throughout the development of SonicSense, we honed our talents in numerous skills both in the technical & non-technical spaces. On the technical side of things, we strengthened our talents in HTML, CSS, JS, & Python amongst others. Furthermore, we learned how to successfully integrate & utilize OpenAI's Whisper technology. In line with the goals of the Hackathon, we broadened our horizons and got to leverage the true power of generative AI for creating good. Supported by the workshops on computer vision, we were able to successfully integrate Google's MediaPipe to further strengthen our product.

Beyond the aforementioned technical skills, we also developed numerous in the non-technical space. Upon conducting our research for the project, we developed a deeper understanding of ASL & the struggles that those with hearing disabilities go through. We gained an understanding of the everyday challenges they face, & how the widespread deployment of ASL can help fix this issue. We decided to play around with ASL itself & even learned a few simple words! Furthermore, we all developed our soft skills in collaboration & pitching amongst others. This Hackathon was truly an amazing learning experience for us & the skills we've developed will prove to be extremely useful in our near futures.

What's next for SonicSense

We have a lot of big plans for the future of SonicSense. We'd like to add support for more languages, both different types of sign languages & spoken languages for transcription. Furthermore, we would like to form partnerships with governments which can allow us to get official status to be utilized in many government applications, regulations, & companies. Furthermore, we would love to work with corporations such as Google or Microsoft, where we can partner to produce courses specifically tailored towards those with hearing disabilities by way of both the content & production of the courses. This can increase the access to education for this otherwise underrepresented & marginalized group of hundreds of millions. We would also like to promote our platform more in developing nations through local partnerships & community initiatives which can allow us to reach the community that truly needs this support the most. We will also look into producing this program as a Chrome/Browser extension as well which can allow for simpler & easier usage.

Built With

• css
• html
• javascript
• mediapipe
• openaiwhisper
• python