Always taking the correct medicine at the right time sounds easy, but it definitely isn't, especially not for children. There are three problems associated with the inability to stick to the medicine schedule - overwhelming information from too many drugs, the inconvenience of having the instructions on multiple drug bottles, and lastly the bitter taste of some drugs.

What it does

Matt is a mobile app available for iOS and Android that encourages children to stick to their medicine schedules. Matt, our virtual dog, is a pun on the word 'medicine' and children are able to feed virtual Matt by taking selfies with their own medicines at the scheduled medicine times. We gamify the process of taking medicine and encourage children to stick to their schedules with appealing GIFs and badges that they accumulate after they stick to their medicine schedules successfully. Matt also serves a practical purpose by including a calendar and alerts screen to keep track of various medicine timings, and a prescription screen that records detailed consumption instructions of the medicine, which may sometimes be overlooked in the chaos of leaving a hospital. Beyond the mainstream instructions, there are even personalized recommendations and tips such as 'crushing the pill' for some big pills or taking the medicine with chocolate if it is particularly bitter. The information gets stored on our web server built on Google Cloud, where we also built a display for doctors and nurses who can ensure that everything is going the right way and will be alerted in case the kids aren't taking their medicine as they are supposed to be.

How we built it

We built an Android and an iOS app with the React Native framework that allows us rendering native apps from JavaScript code. Within the app, we access the system's camera module, where we send the taken selfies with the medicine to our cloud service. There, we have trained a neural network on recognizing different kinds of medicine, to be precise, six different pills. As we trained the image recognition on a pre-trained object detection network, we were able to get very noticeable with only very few data. After the hackathon, we want to collect more data to make the recognition as precise as possible. We have hosted a MongoDB on GCP, so all potential users already can access the data, analyze the pictures and get notified about their medical schedule. Our server is partly built with both JavaScript (node.js) and python (flask) to use the advantages of both frameworks. The server is hosted on Google App Engine.

Challenges we ran into

We had an issue with doubled primary keys in our database, where we decided also to host a Redis DB in order to pass our problems. Also, Google Cloud Platform gave us a difficult time by not allowing us to classify our images, where even the mentors from GCP were very surprised.

Accomplishments that we're proud of

We think this idea of a practical and also emotionally-sensitive app could actually go beyond the boundaries of a hackathon and be implemented on the market.

What we learned

Hackathons, sleepless nights and learning new technologies can be tough but knowing it goes to a meaningful cause is a great motivation. We hope our idea can contribute to the current state of healthcare affairs.

What's next for Matt

With Matt, our aim is not to make become the next big startup. Matt should not be a money machine. We don't want to bother our users with unnecessary ads. We want to make planet earth a better place, for every child, every parent and also anybody else who can benefit from our application, starting with a cooperation with UPMC children's hospital in Pittsburgh, PA and trying to reach out as far as possible.

Share this project: