After Adi experienced a rough time with her father before he passed, always worrying about him coping when he's home alone (her mom wouldn't leave the house for more than a couple of hours) and after receiving a call from him in the middle of the night asking for help getting up because he fell, we wanted to make a technology to help grandma and her family without compromising her privacy

What it does

We are using a webcam to react to grandma's facial expression. We then communicate with her with speech according to her reaction. For example: If she looks ill, check what is the cause and weather she needs medical assistance. If she's sad, asks her what's wrong and suggest some solutions to cheer her up.

How I built it

We used Microsoft's Azure Face API for face detection and facial attributes. The speech to text and text to speech were done using IBM watson's API.

Challenges I ran into

We spent a lot of time installing and configuring Microsoft's IoT devkit. In the end, when we realized how complicated it is to fuse it's information with a webcam, we decided to focus on a different direction (images and speech). We coded in python and the Face API is not well documented, and code samples don't work. We had to adjust the right API out of different sources. We also had latency issues caused by the wifi connection, and processing the API's in the cloud.

Accomplishments that I'm proud of

We are proud to contribute to a project that helps the elderly, when most of the technology today focuses on young people. The elderly are often left alone all day, they can use some company, even if virtual, and someone to alert about an emergency situation without compromising their privacy. we think our project can do just that.

What I learned

It's my first time working with natural language processing and especially speech. First time using Microsoft Azure and various API's.

What's next for Operation Grandma

We can use smart sensors to monitor the house - like temperature, expending the number of cameras so that we can see grandma in all of the house. detect emergency situations (fall, medical emergencies). add more scenarios to the communication with grandma. Smarter text analysis including sentiment analysis and intent. Extend the system to support hearing-impaired by connecting our system to their hearing aids.

Share this project: