Inspiration
Hi! I am a creator of Snapill. Recently, I had to undergo wisdom tooth surgery. The surgery wasn't that bad at all, however, I was prescribed around 5 different medications, from Tylenol to NSAIDs to Steroids. It was hard to keep track of all these medications, and as a result, I ended up going through more pain during my recovery time.
However, to other people, the punishment of medication error is far more dangerous. Around half of drug-related deaths in the US is related to prescription drug use and error (22,000 people, NCBI). Furthermore, it is much harder for older people or those going through complex diseases/surgeries to keep track of their medication requirements. My personal experience, along with these pressing issues, has motivated me to design Snapill, a Computer-Vision based Medication App, in order to improve patient safety and awareness towards the power a single pill has.
What it does
You may notice Snapill is very similar to the popular messaging app Snapchat! This was intentional, as our goal is to ... . Asides from the UI, Snapill is very powerful and differs from a common medication app.
Specifically, Snapill is the first app that leverages advanced Computer Vision techniques like homography with your common LLM. The combination allows for robustness and covers several user failure points (mainly: bad lighting). Snapill allows a patient to scan their medication vial, and automatically generates important metadata like prescription name, expiration date, dosage, and possible times to take the pill. Furthermore, we believe awareness is a strong factor for increased patient safety, so we incorporated the Cerebras AI Inference models to allow users to "chat" with their medications (we weren't joking when we alluded to SnapChat). Users are able to request more information about the drug through the chat, along with a personal vanguard that scans for incompatibility issues between medications. Finally, Snapill provides reminders when it is time for users to take their medications (we couldn't forget that part, could we?)
How we built it
Knowing we needed data, me and Shishir worked with Scriptpro, a pharmaceutical company in Kansas. After pitching our ideas to a senior software engineer (a warm reception), we were able to acquire around 20 vials with different medication names. They were used to train both a Roboflow model to detect labels, and test a custom OpenCV unwarping pipeline.
Snapill is a combination of a react-native app ran on the user's phone that connects to a Flask backend hosted on Heroku. This allows us to use homemade openCV functions while calling APIs such as Roboflow, Cerebras, and Firebase. User data, auth, and storage is all stored through firebase, allowing for simple public downloads from other APIs. We tested the app using both an Android simulator and a physical iPhone. See below for more information about our backend process!
Challenges we ran into
Our first challenge was that UPenn did not allow direct connection in a way. Specifically, we weren't able to run a local backend on the computer and forward its port so a phone could send requests on the same WiFi. This led us to deploy the backend on Heroku, which unfortunately slowed down build times but taught us a lot about deploying on production lines!
Another challenge we ran into was integrating our custom OpenCV code with Roboflow's Workflow design. However, it made us glad we chose to use a Python backend to process Computer Vision! We solved this issue by sandwiching our custom unwarp code between two different Roboflow Workflows.
Accomplishments that we're proud of
The main accomplishment we're proud of was our CV pipeline. Regularly, OCR (Optical Character Recognition) is used for parsing data from images, however we noticed it has a very high error rate when trying to detect text on curved surfaces. To solve this, me and Shishir researched and utilized a pipeline that uses concepts from linear algebra to essentially map the text on the curved label to a flat rectangle. This wasn't easy because there were very few resources for using OCR in edge cases like this. But we curved around (pun not intended) and explored the depths of OpenCV to achieve our goal. OCR was much more accurate now and was able to safely parse label data.
Even cooler, is that it could be scaled to pretty much any text on the curved part of a cylinder! All the method needs are certain extreme points, and the rest is up to interpolation. After Pennapps, I plan to test the pipeline more, and see how to automatically detect the points using edge/contour recognition.
What we learned
For one of us, this was his first time both attending a hackathon AND actually creating a mobile app. While he knew React from previous projects, he still had to learn how to use expo along with several other libraries, and was able to pull it off! We all also learnt about how important reliability is when making patient technology solutions. Handling important data like dosage should be done so carefully.
What's next for Snapill
The next update for Snapill would be finding a way to make the refilling process easier. Usually, there are phone numbers on vial labels, which would be a good next step to detect.
Log in or sign up for Devpost to join the conversation.