Forgetting to take your pills is a problem that affects our everyday lives. Pills are ubiquitous, and range from birth control pills to allergy pills such as Allegra, and forgetting to take them have far-ranging consequences. We are avid users of Google Calendar, and use it for events and class schedules. It would be nice to see reminders to take our pills from Google Calendar, without needing to insert the information by hand.

What it does

PillDrops is an app that allows you to scan prescription labels using your phone to take pictures and automatically inserts reminders on Google Calendar to take your pills correctly according to the instructions on the prescription label.

How we built it

We built the mobile app (the front-end) using javascript and react native, which runs on both Android and iOS. The back-end was built using a python server and websockets, which does the processing to be sent back to the mobile app. The image recognition was done through tesseract, an open source ocr (optical character recognition) engine that was written in C++, and the semantic understanding of prescription labels uses natural language processing to autocorrect tesseract errors.

Challenges we ran into

Building the websocket was tough. For example, we have never transmitted images over websockets before and had to use base64 encoding, which was really fun, not to mention the challenges that come with building mobile apps in general. We also had trouble getting accurate readings of prescription labels from tesseract and using natural language processing to correct for typos. Finally, learning we learned the Google Calendar API from scratch and struggled with getting the system to authenticate correct, and reading the documentation.

Accomplishments that we're proud of

We successfully got tesseract to work with images taken from the mobile app, learned the Google calendar API, and became much, much better at mobile app development.

What we learned

It is very important that apps have low friction of adoption and we learned a lot from thinking about how this app would integrate into people's everyday lives. Technically, we learned the Google calendar API, tesseract, and how to build a webserver that connects to mobile apps. We also learned how to transmit images over the web through base64 encoding.

What's next for PillPop

Coordinating with prescription companies so that the type of medication is more recognizable, spatially. The ocr engine / nlp spell check had a hard time autocorrecting for the medication types because of how obscure names are.

Built With

Share this project: