Using speech-to-text and text-to-speech technology to help the elderly when they require assistance and cannot reach their phones (i.e. in case of a fall)

What it does

By recording snippets of an elder's conversations and transcribing it to text, Elder Care Extraordinaire detects the tone of the conversation and offers to call for help if the elder's tone is perceived as very sad. Unless the tone is perceived to be sad, fearful or angry, the app does not react, and the recordings and transcriptions are not saved anywhere for the user's privacy.

How we built it

Working with Nuance's API for speech-to-text and text-to-speech we established a conversation protocol between the elder and the app. We modified Nuance's sample program to set the page to be recording by default in 20-second snippets, which are then transcribed to text by Nuance's could services. We then set up this text to be analyzed by IBMs' Watson's Tone Analyzer, and when the text is perceived to be mostly sad, fearful or angry (more than 0.5), we return to Nuance's services to being a simple dialog between the elder and the app where the system asks "Do you need help?" and the elder can respond "yes" to trigger a call for help (either their caretaker or the police) or "no" to end the dialog and return to regular recording. The different modules are connected with PHP and JS, and hosted on AWS's elastic beanstalk, which our domains redirect to via Route 53.

Challenges we ran into

None of us had ever worked with Nuance's or Watson's APIs, so learning the ins-and-outs of each was a challenge. To work with them and build our webpage, some team members who had never before used javascript or html had lots of learning to do! Also, setting a up a web-server and domain isn't very easy the first time.

Accomplishments that we're proud of

We're proud to have bridged together our webpage display with features from both Nuance and IBM.

What's next for ECE?

In home elder care is really only the tip of the iceberg! Longer term monitoring could be used to measure and detect changes in mental health, the onset of Alzheimer's or dementia creating slight changes in the speaking style and tone of a user. On a larger scale it could be used in hospitals to determine which departments' patients have higher levels of anger, fear or sadness, so resources can be easily targeted to improve quality of care.

Very similar technology could also be used in call centers, where conversations are already monitored. By assessing the tone of both clients and employees, a central AI system could intelligently pair employees and clients based on their tones in previous conversations. Managers can resolve an employee's worsening tone before an incident occurs. The applications and extensions for such a system are endless!

Share this project: