Inspiration

Dementia is a condition which affects over 50% of aged care facility residents in Australia. The most common form of dementia is Alzheimer’s disease, for which there is currently no cure. Throughout our lives we have had friends and family who have suffered from this condition. We wanted, in our own small way, to assist in the fight against dementia. We hope that our AI assistant, Nicole, demonstrates the positive impact technology can have in this area; and if you'd like to learn more about dementia, or how you can help, please visit fightdementia.org.au.

What it does

Nicole currently offers a reminder and calling service, all via the traditional home phone. The reminders provide prompts for dementia patients, set by their carers. The reminders are completely customisable, for example a carer could set a daily reminder for their patient to take medication or to attend a therapy session. Nicole's Australian voice encourages patients to carry out their routines with a sense of independence, all via a familiar technology, their home phone.

Nicole also offers AI assisted voice dialing, similar to that found on most smartphones today. Patients can call friends and family by simply speaking their names. Simple voice dialling encourages communication by reducing the friction in making a call to friends and family.

How we built it

Nicole consists of 3 parts: a website, a reminder calling service and an interactive voice service.

We created a website for carers to manage patient reminders and contacts. The site is completely serverless, leveraging AWS S3, Cognito and DynamoDB.

The reminder service uses Lambda to convert the written prompt into speech (using Polly) and to call the patient using Twilio.

The interactive voice service allows a patient to call Nicole (using a quick dial saved on their phone) and to ask for the time of day or to make a call to their friends and family. This was initially part of the reminder service, but we had issues with the voice interaction in our first build (more on this below). As a result, the voice service now uses AWS Connect with Lex.

connect

Challenges we ran into

When we first discussed entering the AWS Chatbot Challenge, we assumed that we'd be building a Slack bot. Unfortunately, our target audience aren't likely to use Slack! This in itself presented an unexpected challenge, to move chatbots out of messaging products and into aged care facilities (ultimately via the phone).

In our first build of Nicole we called the patient using Twilio, generated Nicole's voice using Polly, recorded the user's response with Twilio, transcoded the audio to PCM, transcribed the patient's audio with Lex, parsed the patient's intents with Lex, generated a response with Polly - and so on!

Whilst we were able to get this to work, the delay between the patient speaking and Nicole responding was in the order of 10 (and sometimes 20) seconds. We optimised this flow as much as we could, but the conversation never felt natural. We also didn't have much luck with Lex transcribing Twilio audio.

As we were troubleshooting, AWS announced Connect (their Call Centre as a Service) in Sydney. Connect provided the same feature set we were using with Twilio, but with native Lex integration*. We ported our code to Connect (which was a cinch with the drag and drop UI) and found that conversation response times dropped significantly to between 2 and 4 seconds!

What we learned

We all tried something new when building Nicole. Whether that was writing frontend JavaScript for the website, Node.js Lambda functions or Lex; we each took a step out of our comfort zone to gain exposure to new AWS services and skills. More than anything though, we learnt what a small team with an aspirational goal can achieve in short order, and we hope this inspires others to do the same.

What's next for Nicole

We were hoping to implement additional functionality (such as integrating Facebook data to interact more socially with patients), but the voice interaction tuning and troubleshooting took up much more time than we'd planned. Hopefully we can build out Nicole's skills over time, and we'd love to hear from you with feature requests and feedback.

*Lex integration was only available in us-east-1 at the time of writing. We trialled Connect in Sydney, but ultimately we moved Nicole to us-east-1 to leverage Lex.

Built With

Share this project:
×

Updates