Inspiration
A study found out that when it comes to texting vs calling, calling wins if your objective is to connect with someone in a meaningful way. Put yourself in the position of someone who has difficulty hearing or speaking. You want to make an emergency call but you can't because you're trapped in a fire. Instead, you’ll be forced to utilise a relay operator, which can take longer than regular operators. You do not want to, but you will have to.
Around 70 million individuals worldwide are unable to communicate due to lack of speech, and 360 million suffer from hearing loss. Our elders’ hearing and oration abilities deteriorate with age, posing a significant challenge for them while conversing on calls. Furthermore, around 81 percent of millennials experience anxiousness before making phone calls.
Why is there such discrimination? We intended to make calls easier, more accessible, and more inclusive. As a result, we created Connect, an assistive calling tool that can be simply operated on any device, including laptops, desktop computers, and mobile phones, for smooth calls.
What it does
Connect enables everyone to converse conveniently via phone calls. The main functionality of the webapp is as follows- Text inputted by the sender is converted to speech for the receiver. Converts the receiver's speech to text for the sender. Send the user’s location to the receiver for emergencies and other purposes.
Individuals who are deaf, have difficulty with speech, or the elderly people can use the app to-
- Make bookings at your preferred hotels and restaurants.
- Emergency Calling along with the ability to send whereabouts.
- Refill prescriptions from the pharmacy.
- Converse with ease on regular calls
- Check recent calling activity
- While in an emergency, the user may simply press the emergency option without logging in to initiate a call to the helpline. The web-app creates a message that is pre-populated with the person's location and an emergency message.
As the app has been authorized and authenticated, the user may log in or register to gain access to additional features. We have recently included options such as restaurant table booking and medication ordering. Because of the flexibility of the code, this may be readily expanded to cover other needs. The user can fill out the form provided. These forms have some particular fields and inputs that auto-generate the initial message for the user. The software also allows users to send their whereabouts.
The call is received by the receiver as soon as the first message is sent by the user. The text message sent is converted to speech. The receiver's responses are then turned to text for the user. The connection might go on for hours without interruption. The main metadata about the call is then added to the user's ‘recent calls’ panel.
How we built it
For building this full stack webapp, Node.js and Express.js were utilised for the app's backend. The user information was saved in the MongoDB database. Twilio was utilised to provide a link between the web and traditional phone calls. The software utilises Twilio's programmable voice to translate text to speech. We utilised the GCloud speech to text API to translate text from the Twilio call stream. We used HTML, CSS, Javascript, and frameworks like Bootstrap and Axios for the frontend. We also utilised the GeoApify API and the HTML Geolocation API to attach the location.
Challenges we ran into
We encountered difficulties connecting to a stream in Node.js while working with Twilio because the documentation only included the Python Flask example. As a result, generating and connecting to a stream took a long time. This was our first project in which we used Twilio, Web sockets, and Google Cloud APIs. We needed to learn more about web sockets and how to communicate within a stream using a web socket after we figured out streams. Because performing a POST request each time would have terminated the call, using web sockets was the only option. We were able to create a link between the sender on the app and the recipient on a conventional call after researching about it. The issue now was to extract speech from the stream and then translate it into writing. We went through the GCloud Speech-To-Text API documentations and ultimately got it to function.
Accomplishments that we're proud of
We are quite happy to have created an application that has the potential to scale and become associated with the phone dialer, but with better functionality and inclusiveness. We are happy to have constructed this application with all of these new technologies - learning them, integrating them into the app, and finding alternatives within the course of this year’s Superposition VI.
What we learned
We discovered a lot about web sockets and real-time online communication. Twilio and its numerous features, such as Programmable Voice and Streams, piqued our interest. It was exciting to utilise the Google Cloud APIs for the first time and see them operate in our nodejs app.
What's next for Connect
We intend to do the following in the future-
- Include support for more languages for conversing easily.
- Include a group call function that does not exclude those with special needs.
- We hope to scale out webapp in the future, with the added functionality for everyone on the platform to create their own contact through the webapp.
Testing
To test the app, establish an account on Twilio, generate a free Twilio contact, and add your private contact to the verified caller ids panel. Then, go to the Reserve Table section or Pharmacy section in our web-app and enter your Twilio number and your Personal number in the forms provided.
Log in or sign up for Devpost to join the conversation.