PerfectGrandkid

You might not be a perfect grandkid ... But we are!

Comment

Perfect Grandkid Health Bot

PerfectGrandkid

Inspiration

Our grandparents live in different places around the world, such as Russia and Taiwan, all far far away from where we currently are --- the US. We realized how family means to our grandparents after coming to college, as we are physically thousands of miles away, in a different time zone, and living a busy life. It is difficult for us to know how our grandparents are and to accompany them often. Also, sometimes the elders hesitate to talk about their mental health issues fearing that they will become a burden for others. Thus, we aim to provide both text and visual accompany and professional mental care and health evaluation to our beloved grandparents. I am not a perfect grandkid, but our bot - is!

What it does

Our bot allows our grandparents to dynamically interact with their "perfect grandkid", which is our bot, to alleviate loneliness! Our bot thoughtfully determines whether our grandparents are in a good mood or not. If yes, the bot will joyfully chat with our grandparents, just like how we would. If not, similar to our reaction, we would worry about them and ask about their situation, ask for their mood and feelings, and comprehensively provide a summary of the state of our grandparents. If an emergency for accompany is needed, the bot will directly contact their real grandkid so that their grandparents can receive the most care as possible.

How we built it

Using Mircrosoft Azure, we created a Web App with a CI/CD Pipeline linked to our repository on GitHub. Every time a push is made to GitHub, Azure DevOps will trigger an automatic build, which passes the build artifact through the pipeline for a new release.

Also using Microsoft Azure, we created a Health Bot Application that we deployed also on Azure, and is then called on in our main GitHub repo. The Health Bot Application poses as the perfect grandkid, providing care to elder family members living in isolation, and keeps track of their health condition through conversations with the Chat Bot. In addition to the Chat Bot, we also have a AR Model component made possible with Apple's ARKit. By hosting .usdz files on a webpage and viewing them through Safari on an iPhone, we are able to place AR Objects in real-life, viewable through the phone.

We attempted to create 3D facial scan models using Apple's ARKit and CoreML, but only proved successful in local runs on our phone, which we will be demonstrating, and is a big component that we wish to integrate to our overall service.

Challenges we ran into

  • Setting up Microsoft Azure with the new UI version
  • One of our goals is the make an AR character using 3D scanner of ourselves to create an AR application and it took us longer than expected in learning the new .usdz 3d modelling method and debugging the correct routes in Xcode

Accomplishments that we're proud of

  • Collaboration and discussion of cool ideas that have social impacts among our team members and many more women from all over the country
  • Collaboratively solved problems and sought advice from professionals in the field
  • Although not all of the functions we hope to include are implemented, we enjoy seeking different technologies, software, and methods to carry out the functions
  • We learned how to customize MS Azure Health Bot for our purposes in less than 36 hours
  • We have created a working AR model
  • We have researched well-being and mental health improvement techniques that can be used on elderly population

What we learned

  • New ideas
  • New issues
  • New solutions
  • New technologies, software, implementations, platforms (e.g MS Azure, JSON, Apple AR kit, CI/CD Pipeline)

What's next for PerfectGrandkid

  • We aspire to combine text, voice, and AR or provide the most comprehensive accompany and care for our grandparents and to maximize the similarity of the bot to the actual grandkid. Other than just having text, we hope to have the bot "speak" so that grandparents don't even have to type as typing can sometimes be very time consuming for them. Moreover, to add on the realisticness of the experience, an AR 3d moving grandkid can be present on the screen, with the use of machine learning from videos of the real grandkid to simulate the grandkid's habits and movements.

Built With

Share this project:

Updates