Covid-19 has taken a massive toll on mental and emotional health. Our solution is to foster a supportive community of users with Pillzen, a prescription label app that reads and inputs a label into a reminder app.

According to, “more than 80 percent of older patients (ages 57 to 85 years) use at least one prescription medication on a daily basis, with more than 50 percent taking more than five medications or supplements daily.” With that amount of medications, some hazardous in the wrong amounts, there needs to be a way to accurately track and remember what medications you have or have not taken.

Our group members have family members who take multiple medications and have first hand seen how difficult it is to keep track of the timing of medication. Finally, to accommodate for the difficulty of typing lengthy prescriptions (especially for seniors who are not tech-savvy), we offer file upload and camera features to leverage machine learning and process prescription images.

What it does

Once users log in to their Pillzen account, they can submit pictures of their medication prescriptions. Pillzen utilizes the Google Cloud Vision API to extract text in prescription image inputs with machine learning. Based on the prescription text, Pillzen uses Twilio’s API to deliver personalized SMS messages and/or phone calls to notify users when and how to take the appropriate medications each day.

Inspired by open-source collaboration, Pillzen offers a supplemental superchat service to help users connect with each other for emotional support, and receive real-time assistance from the Pillzen support center (which also offers Teledoc appointments and mental health counseling).

How we built it

We developed the front-end with React and user credentials are stored in our Google Cloud Firestore database. In the dashboard, users have the option to upload or take a picture of their prescription label. After choosing their preferred methods of notification, users receive confirmation of a successful submission. Then, our Flask backend calls Google Cloud’s Vision API to use machine learning and extract text from the submitted prescription label. At that point, we parse the retrieved data and store it as a reminder in a SQLite database.

When it’s time to remind users to take their medication, our Flask backend uses the Twilio API to deliver automated SMS messages and phone calls. When a text message is sent, users must reply YES to confirm they took the pill. Users also have the option to receive phone-call notifications.

The Pillzen Superchat was built with a React frontend and Firebase React Hooks to store user and message information in our Google Cloud Firestore Database. We deployed our Flask backend with AWS EC2 and we hosted our Pillzen app with’s custom domain on Vercel. Finally, we hosted the Pillzen Superchat using Firebase.

Challenges we ran into

We faced numerous challenges while trying to connect the React frontend with the Flask backend. We used ngrok to test data transfer from the frontend to the backend. We had trouble with simultaneously sending the prescription label image with additional user data to the backend. Our solution was to store the prescription label image as a temporary file in Firestore to be downloaded by the Flask backend.

Accomplishments that we're proud of

We are proud that we utilized Twilio’s API to send SMS messages and calls. This functionality allows for users to take control of their medical health and customize their Pillzen experience.

We incorporated the Pillzen Superchat to help connect our community of users with each other and the Pillzen support center. This powered our two-in-one solution to address mental and emotional health along with timely medication reminders.

Finally, this was our first time working with a NoSQL database which we incorporated through the Google Cloud Firestore database to support user login and Superchat features of Pillzen.

What we've learned

Privacy and security are of great concern to maintain user trust in our app. We must ensure HIPAA compliance and other industry best practices to protect our users’ data while offering convenient services.

The SMS messaging and phone call notification system we developed with the Twilio API is extremely powerful and has the potential to improve the quality of life for millions of people across the globe.

UN’s Good Health & Well-being Sustainable Goal. We meet this as we promote good health and well-being in our app (medication, emotional, and mental health support for all ages).

What's next for Pillzen?

Ultimately, our next step for Pillzen is to attempt to upload it to the Apple Store and Google Play Store where we can empower millions of patients.

Also, we would like to develop Twilio API-powered SMS and voice chatbots to provide a faster method of scheduling teledoc appointments.

Finally, we hope to incorporate an in-app social media platform for users to personally connect each other and improve the mental and emotional health of the Pillzen community.

Share this project: