-
-
-
-
-
Note: TEXTBOX to response is not apparent in the deployed site. It is here to "Send message" as a user (look at cursor)
-
Restaurant workflow
-
-
-
-
-
-
-
Quick Links
Deployed App NOTE: We only implemented an example Restaurants flow (so only the Restaurant item is fully functional). Feel free to create an account, clicking to restaurant, filling out the form with the information you'd like in Restaurant, and enter your phone number in the +11234567890 to receive the call as the Restaurant. Side note: If you don't put in the your own phone number, it will go to the CallBuddy developer who may be a bit confused :)
Introduction
While many businesses communicate with their customers through multiple online channels, it would be hard for businesses to ignore how calls to businesses have significantly increased in response to the mass adoption of mobile phones. In fact, there are many reasons why people opt to call instead of using other methods:
Source: http://www.biakelsey.com/research-data/current-research/call-commerce-1-trillion-economic-engine/
Simply put, communicating by voice is faster, easier, more effective, and convenient — but mainly for the hearing community. For the deaf and hard of hearing (HoH), talking on the phone is something they struggle with every day. Our teammate, who is HoH, faces this problem themselves: they get severe anxiety every time they have to call the pharmacy to refill their medication or call the local restaurant to make a reservation.
Traditionally, a relay operator paired with hardware is the only option to translate speech-to-text on phone conversations, but lack of widespread adoption, slow and laborious processes resulting in delayed relays, and scammers abusing free relay services lead to access challenges or being hung up on. And if, for example, the deaf and HoH community seeking emergency assistance from a 911 call center, encountering any challenge can have serious consequences.
That’s why we created CallBuddy, an assistive communication app that enables deaf and hard-of-hearing people to access services by making and receiving calls from anyone without having a relay operator present or using any additional hardware.
Purpose & Motivation
We listened to challenges and pain points of accessing services that one of our teammates, who is HoH, faced. Whether carrying out something as mundane as making restaurant reservations or accessing essential and daily services like asking for help or information from a doctor, pharmacy, or even 911, sometimes having a real-time phone conversation is best. However, having phone conversations is challenging for those who are deaf and HoH.
As mentioned earlier, lack of widespread adoption of traditional technologies on the business end, slow and laborious processes resulting in delayed relays, and scammers abusing free relay services lead to access challenges or being hung up on. And if, for example, the deaf and HoH community seeking emergency assistance from a 911 call center, encountering any challenge can have serious consequences
These days, texting, videophone, and Relay Calls are essential digital tools that deaf and HoH people use to communicate. But imagine trying to call a doctor, lawyer or loved one and wanting to have a private conversation. With the current relay system, this is not possible. Some places like a doctor’s office choose not to use a relay/TTY (teletypewriter). Additionally, most businesses use phone systems instead of cell phones, which don’t support SMS (short message service) messaging like texting.
This is why we created CallBuddy — to enable the deaf and HoH community to communicate directly with the hearing community in a way that doesn’t require additional persons or hardware.
How does the application work?
With CallBuddy, users can make calls from their mobile phone or on the Web. After users sign into their account, which is connected to a Twilio number, they can choose who to call.
For at-home emergency calls, make quick selections that answer baseline questions a dispatcher would ask and automatically populate the initial message sent and spoken to the dispatcher from these selections and account profile like the home address. Prior to calling, users will always be able to confirm what will be said on the phone prior to calling.
After the call starts via the user’s Twilio number and someone on the receiving end picks up the call, the user sees in text on either phone or Web what is being said to the receiving end through text-to-speech. The person on the receiving end responds by voice and the response appears as text to the user through speech-to-text. Because this communication is happening in real-time, if the deaf or HoH user is taking a while to type out text, the called party will be notified that said user is typing a response. Conversely, users will be notified if the called party puts them on hold. This back-and-forth communication continues until the call ends.
For contacting restaurants, make a reservation with some pre-filled info from account profile or ask a general question. With the app, users can see the restaurant’s response and reply back and forth as needed.
For medical calls, refill a prescription or contact a care provider. Info from account profile will auto-populate into any relevant fields
For general calls, input a phone number to start a conversation by texting and converting text-to-speech and vice-versa.
There is also a way to save favorite workflows for services users commonly use as well as see a history of their latest calls.
How was the application developed?
React and Next.js was used to develop the front end website. Auth0 was used for authentication. Python/Flask, PubNub and Twilio was used for the backend and SMS messaging. The app is deployed on Heroku and domain is hosted by GoDaddy.
The wireframe and prototype was created using Figma.
How to use the application
1.) A deaf or HoH user signs up or logs in. User can fill in account profile to help populate information when requesting for services
2.) User selects the category of the service they’d like to request along with any subcategories
3.) User fills in baseline information to include in the initial contact. User has the option to use streamlined forms for common phone calls.
4.) User confirms message to be sent to destination phone number
5.) After user confirms, the call begins. At this point, the user will start to receive in the chat interface live transcriptions of what is said over the phone.
6.) The user can then respond using text chat. Once sent, the text is converted to speech so that the person on the receiving end will hear the response.
7.) The back-and-forth communication process repeats, and phone call remains in progress until disconnected
Difficulties & Challenges faced during the design and/or development process.
We used design thinking to understand pain points from the user's perspective and using that understanding to create a solution, rather than being enamored with a solution and thinking how can users use this. During this process, we thought a lot about what are some common tasks in making a call, what it means to automate common tasks on phone calls
This being one of our first times working with Figma, we had a lot to learn with:
- Customization, including favorite and quick links
- Adding preset messages and auto-completes to support real-time conversation
- Add feedback functions
This was also one of our first times working with Next.js framework, Twilio, Pubnub, and Auth0 on the backend. Voice transcription was through Microsoft Azure Speech-to-Text and text-to-speech was through Twilio. One of the main challenges was to create a deployment system in Heroku that would work with both flask and next.js, since these frameworks didn’t work well with each other and there were minimal documentations. Twilio also did not have a good flask socket implementation (we had to required the socket library and made a bridge because Heroku would shut down our socket within 10 seconds). There was a fairly nontrivial amount of engineering work to make everything work together. This is why we only had enough time to implement 1 flow (restaurant), however it is not difficult to see how with a little more work we could implement all of the other flows.
Additionally, we enjoyed exploring the SMS capabilities of Twilio and learning how to obtain geolocation that can be auto-relayed in the conversation, which would be useful for emergency services if the supported needed is for outside where the user is at. We didn’t have enough time to explore this, but it would be a great extension for the project.
Go-to-Market (How will the application be available to the public, and is it scalable?)
The current version of CallBuddy is available for anyone with an Internet connection to review at http://callbuddy.co. It can be accessed via desktop or on mobile phone. We would eventually work to make this a progressive web app (PWA) as well, while availing the current version if users have legacy phones or other for which PWA’s are not compatible with.
Another use case for this app is to enable individuals, especially young kids, with a delay in speech development to communicate with people who care about them.
Additionally, in the world of call commerce, businesses would be missing out in helping to fulfill others’ needs if they do not adapt to the changing demographics whereby by 2050, nearly 2.5 billion people will have some degree of hearing loss according to the World Health Organization. That is, there is opportunity for even businesses, educational institutions, organizations whose clients or people served do request for this service can use the app to contact the deaf and HoH community and more to ensure they receive the same access.
We imagine that, if this were to be productionized and commercialized, we can charge users a monthly fee (such as $8/month) to give them their own number through Twilio. This means that they can give out the number to everyone and receive inbound calls as text too. This means that deaf and HoH people no longer have to worry about getting surprise phone calls: they can function normally with CallBuddy, make and receive calls as texts, and take control of their life.
What’s Next
Partnerships
There is potential for future partnerships with 911 dispatching to refine the information intake process. We could also link our app to social media to enable conversations with friends and family.
There could be potential to create a business login where we partner with businesses, educational institutions, and more to provide ways to facilitate smooth conversations by phone from those who can hear and by chat with those who cannot hear as well.
Product Development
There is opportunity in better understanding the workflow for accessing Emergency Services. We started off with in-home emergencies to limit the scope since conversations can head into any direction. There are also emergencies that happen outside like natural disasters, and we’d want to empower those who are deaf or HoH to be able to take civic action.
We would also leverage cloud capabilities to translate text to different languages.
Global Expansion
This demo has been created with the United States market in mind. We would look to expand to other markets and adapt the templates and content to match cultural nuances or trends
Log in or sign up for Devpost to join the conversation.