Inspiration
Preventable medical errors are the 3rd largest cause of death in the United States (after heart disease and cancer). The most common types of medical errors are medication errors, which result in 1.5 million injuries and $3 billion in compilation costs yearly. Even in cases without serious injury or hospitalization, struggles with identifying pills makes patients less likely to adhere to their prescription schedule, inevitably leading to adverse effects such as lower blood pressure control.
This problem only gets worse as people across all age groups increase their consumption of prescription medication. The situation can be especially critical for the elderly, who are also increasingly likely to be on multiple prescriptions at once.
Our medical systems do try to address these issues, but the current solutions have many limitations. Some pharmacies, for example, hold “brown bag consultations” where patients can bring in unknown pills. However, such events can be difficult to access by those who can’t travel due to age, disability, lack of transportation, budget (i.e: gas money), or distance. There are also “pill registries” where users can input details about medication (shape, color, size, etc) in order to identify them. But for elderly patients, such tools can be slow and difficult to use, especially if they struggle with the sight or knowledge to correctly observe details about their pills.
These unaddressed struggles inspired our team to create an easy, efficient, and accessible way to accurately identify pills.
What it does
Dose Detective is a mobile application that classifies images of pills. Our app directly takes users to the home page, which has a camera widget. All the user has to do is take a picture of their pill, and the application will use a machine learning image classification algorithm to identify what kind of medication it is. Using this tool, people are less likely to confuse pills that look alike, reducing the chance of medical errors. It is particularly useful for elderly and chronically ill patients who often take multiple medications a day.
How we built it
We worked with a comprehensive dataset from Kaggle, which contained thousands of images featuring over 100 different types of pills. To begin, we conducted a preliminary analysis of the dataset, using Matplotlib to visualize key aspects such as the distribution of pill types and other important characteristics. This step allowed us to better understand the data and prepare for the subsequent model development.
Next, we used PyTorch to create a neural network designed for multiclass classification, carefully training the model over multiple epochs to ensure it could effectively differentiate between the various pill types. Once the model was trained, we employed AI_Edge_Torch to quantize it, reducing its size and improving its efficiency for mobile deployment. We then exported the model to a TensorFlow Lite format, which is optimized for running on mobile devices.
For the app's development, we chose Flutter as the framework to ensure that the app could be easily deployed across multiple platforms, such as Android and iOS. The app was designed to use the phone's camera, integrating TensorFlow Lite and its plugin to run the model locally. This enables users to take real-time photos of pills, with the app instantly identifying the pill type based on the model's predictions. Our focus on cross-platform compatibility and local processing allows the app to be both accessible and efficient for a wide range of users.
Challenges we ran into
We ran into issues while trying to use the TensorFlow Helper Plugin for Flutter, which we primarily relied on to deploy the ML model on edge devices. After running into several errors, we learned that TensorFlow Helper was deprecated, and it had been rebranded to LiteRT. To overcome this, we used TensorFlow Lite for ML model conversion, which ran successfully on the Android Studio IDE.
Accomplishments that we're proud of
We are proud of successfully deploying a machine learning model to our mobile application, which involved several intermediary steps that were challenging to run correctly. We are also happy we could embed a camera widget into our app, which makes it simple for the user to add an image.
What we learned
We learned how to develop front-end and back-end for an Android application using Flutter, Dart, and Android Studio. We also learned how to train a machine learning model using PyTorch and use TensorFlow Lite to deploy the model.
What's next for Dose Detective
We hope to integrate a database of medication information so users can learn about the pills they are consuming. We also hope to add a feature than can track pill usage and set reminders for taking medications.
Log in or sign up for Devpost to join the conversation.