Unmute

Homepage
Dashboard
Call Window

Inspiration

According to the U.S. Department of Health and Human Services, 1 in 8 people in the United States (13%, or 30 million) ages 12 or older have hearing loss in both ears, and more than 90% of children with Selective Mutism also have social phobia or social anxiety. Our primary goal for this project is to assist people within this range and make phone calls more accessible.

What it does

Unmute’s mission is to connect individuals who have difficulty hearing or are mute to an essential service - phone calls. The user inputs a phone call and a starting message into the website, and our system will dial them using our toll-free number. When the receiver answers the call, anything they say into the phone will be transcribed and displayed in the chat window on the same webpage. The user can type in and submit text messages in the chat window, which will be converted to speech for the receiver to hear. The user also receives a post-call summary, can access a call history log, and can customize voice settings.

How we built it

Unmute is a phone service built primarily using TypeScript and React, leveraging Node.js as the backend framework. We used Vite as a build tool and development server, Tailwind CSS for styling the components, and PostgreSQL for the database. Used both the Twilio and OpenAI API to integrate the humanized voice bot to text and the text-to-speech features that our ideas implement. We leveraged Generative AI(Repl.it, ChatGPT) to create parts of the backend & UI, as well as debug any errors that came along the way.

Challenges we ran into

Our biggest challenge was integrating OpenAI and Twilio into the program. After we created a framework of the product, implemented Twilio’s API into the program, and got the toll-free number to call any personable, our biggest issue was still turning the user’s text to speech and using verbal recognition to convert the speech to text all while living within the program. When integrating the API to turn text into speech, we started with integrating Twilio’s voice program. Although it was a success, we still were not able to install live audio transcribing nor integrate OpenAI’s text-to-speech API. After hours of debugging and testing, we finally had a prototype where all features were available as well as integration of OpenAI’s text-to-speech voice.

Accomplishments that we're proud of

Our biggest accomplishment that we are proud of is successfully integrating OpenAI and Twilio into our program. This was a daunting task that initially seemed unclear, as we encountered several challenges along the way. We faced multiple failures and had to debug countless lines and failed testing many times. However, after extensive testing and iterative adjustments to the code, we ultimately achieved the integration of OpenAI and Twilio highlighting our application's functionality.

What we learned

Through the integration of OpenAI and Twilio, we learned how to integrate multiple APIs by reading documentation and understanding the different functionalities provided by each service. Additionally, we learned how building small prototypes can help test functionalities in isolation, which is much more effective for identifying and solving issues being integrated at a larger scale.

What's next for Unmute

Our next goal for this project is to integrate Stripe’s ID verification to prevent abuse of our API as having full access to a toll-free number can attract unethical use customers. Additionally, we also plan to create a user feedback loop to collect feedback to understand the needs of our customers better. Through this, we can enable more features and see their usage patterns and analytics for call data to prove insights. Lastly, we plan to integrate monitoring tools to analyze performance. This will help in identifying bottlenecks and optimizing the user experience.

*The link to the website is included, but it is not functional due to API keys not being included.