Inspiration
Medication errors, one of the most common types of errors that occur in the medical field, are failures to carry out a planned action properly during the process of prescribing, administration, or otherwise dealing with medication (Fahimi et al., 2009; Callen et al., 2010; Härkänen et al., 2013; Wittich et al., 2014). Although these errors are often benign, they occasionally cause adverse drug effects, leading to severe injury or even death (Van Doormaal et al., 2009; Wittich et al., 2014; Bates & Slight, 2014). Therefore, reducing the prevalence of such errors would benefit the well-being of both hospitals and patients (Callen et al., 2009; Wittich et al., 2014; Laatikainen et al., 2021). One subset of medication errors is transcription errors: errors that occur specifically due to inaccurate entry of patient data into medical records (Fahimi et al., 2009). This happens because human operators are fallible. Whether due to illegible handwriting by medical professionals or typos by record-keepers, transcription errors occur in 5–30% of scenarios in which errors could arise, varying from hospital to hospital (Callen et al., 2010; Lewis et al., 2012; Wittich et al., 2014). Transcription errors are simple but can have devastating consequences — therefore, a need exists to double-check data entries and minimize inaccuracies. Enter the Patient Medical Discrepancy Identifier (PDMI). We developed a program that uses artificial intelligence to determine whether data entries line up with current medical knowledge, flagging entries that may be inaccurate or cause harm to the patient. This program would eliminate the need for a human to do the menial task of double-checking every record for accuracy (simply needing someone to review the flagged content), which would greatly benefit all hospitals, particularly those that are chronically understaffed. In accordance with the increasing ease of access to powerful artificial intelligence models, we hope that our program could help shrink the quality gap between well-funded and poorly-funded medical units.
What it does
Our project checks for the following discrepancies in patient transcriptions: diagnosis-diagnosis discrepancies, prescription-diagnosis discrepancies, prescription-prescription interactions, and possible allergic reactions. Diagnosis-diagnosis discrepancies are when a patient’s listed medical conditions do not make sense together. For example, while not impossible, it is unlikely that a patient would have hypertension and hypotension simultaneously (which would probably warrant its own unique entry regardless). Prescription-diagnosis discrepancies occur when the medication prescribed or currently being taken by the patient does not match the medical condition(s) listed in the patient record (e.g. a patient is prescribed methotrexate, an immunosuppressant and chemotherapy drug, when afflicted with type 2 diabetes mellitus). Prescription-prescription interactions occur when one medication a patient is taking affects another medication, which can have effects ranging from reduced effectiveness of the medications to endangerment of the patient. For example, blood thinners such as warfarin should not be taken with painkillers such as ibuprofen due to risk of internal bleeding. Finally, allergic reactions are relatively straightforward: the program flags if a patient is prescribed a drug that they may be allergic to. The program essentially flags any of the aforementioned issues and returns a reasoning for the flag, such as “patient may be allergic to medication X” or “medication Y is not appropriate to treat condition Z.”
How we built it
We insert patient information as a string into a dictionary as follows… { “Name” : “example_name”, “Diagnoses” : [“D1”, “D2”], “Prescriptions” : [“P1”, “P2”], “Allergies” : [“A1”, “A2”] } Then, using the Gemini API model 1.5 Pro, we generate text by combining the patient info as string input (formatted as JSON) and adding it with another string that acts as a prompt (prompt located in Appendix A) Gemini 1.5 returns the following… { “Name” : “string” “Diagnoses” : [], “Prescriptions” : [], “Allergies” : [], “Explanation” : { “Diagnoses” : “”, “Prescriptions” : “”, “Allergies” : “” } } Where if a corresponding index in the input dictionary has a discrepancies, it is marked in the output dictionary within each corresponding array.
Challenges we ran into
The main challenge this project faced (like most projects using generative AI) are accuracy problems with Gemini’s outputs. Of the couple of errors generated, all of them were false negatives. Of note, almost all errors were failure to identify prescription-diagnosis discrepancies. Additionally, the model may sometimes fail to report all discrepancies, and the likelihood of that problem occuring increased the more complex a patient’s medical history was.
The Gemini quota limit also acted as a barrier to our project. Despite using an API key allowing for an increased limit, iterating through our array of patients would take around 4-5 minutes each time we wanted to run the model which added an additional time constraint to the project.
Accomplishments that we're proud of
The quota limit, while a nuisance, was something that the team was able to overcome by implementing the time.sleep() function to prevent crashes due to a quota limit exception.
To recap, our key takeaways are as such: most of the accuracy and quota problems can be addressed with more data/training and a limit increase respectively. Despite those problems we successfully worked with these issues to complete the project, showing promising results warranting future investment on this program.
What we learned
We learnt that whilst the main challenge of accuracy would be difficult to overcome within the time span of the hackathon, this is an issue that can be solved with more data and additional training. Accuracy of the program would increase significantly if the 1.5 model could be fine-tuned by the user.
What's next for the Patient Medical Discrepancies Identifier
We would like to improve on this project by improving its accuracy and adding features. That would include another additional pair with a key “dosage” which would contain an array of dosage amounts corresponding to the prescriptions provided.
This technology could find some use in hospitals, particularly in places like the Intensive Care Unit (ICU) where human resources to double check for medical transcription errors may not be readily available. Having this tool on hand and running could spot a potentially fatal error.
Resources
Bates, D. W., & Slight, S. P. (2014). Medication errors: What is their impact? Mayo Clinic Proceedings, 89(8), 1027–1029. https://doi.org/10.1016/j.mayocp.2014.06.014 Callen, J., McIntosh, J. H., & Li, J. (2010). Accuracy of medication documentation in hospital discharge summaries: A retrospective analysis of medication transcription errors in manual and electronic discharge summaries. International Journal of Medical Informatics, 79(1), 58–64. https://doi.org/10.1016/j.ijmedinf.2009.09.002 Fahimi, F., Abbasi Nazari, M., Abrishami, R., Sistanizad, M., Mazidi, T., Faghihi, T., Soltani, R., & Baniasadi, S. (2009). Transcription errors observed in a teaching hospital. Archives of Iranian medicine, 12(2), 173–175. Härkänen, M., Turunen, H., Saano, S., & Vehviläinen‐Julkunen, K. (2013). Detecting medication errors: Analysis based on a hospital’s incident reports. International Journal of Nursing Practice, 21(2), 141–146. https://doi.org/10.1111/ijn.12227 Laatikainen, O., Sneck, S., & Turpeinen, M. (2021). Medication-related adverse events in health care—what have we learned? A narrative overview of the current knowledge. European Journal of Clinical Pharmacology, 78(2), 159–170. https://doi.org/10.1007/s00228-021-03213-x Lewis, P. R., Dornan, T., Taylor, D., Tully, M. P., Wass, V., & Ashcroft, D. M. (2009). Prevalence, incidence and nature of prescribing errors in hospital inpatients. Drug Safety, 32(5), 379–389. https://doi.org/10.2165/00002018-200932050-00002 Van Doormaal, J. E., Van Den Bemt, P. M. L. A., Mol, P. G. M., Zaal, R. J., Egberts, T. C. G., Haaijer‐Ruskamp, F., & Kosterink, J. G. W. (2009). Medication errors: the impact of prescribing and transcribing errors on preventable harm in hospitalised patients. Quality & Safety in Health Care (Print), 18(1), 22–27. https://doi.org/10.1136/qshc.2007.023812 Wittich, C. M., Burkle, C. M., & Lanier, W. L. (2014). Medication Errors: An Overview for Clinicians. Mayo Clinic Proceedings, 89(8), 1116–1125. https://doi.org/10.1016/j.mayocp.2014.05.007
Appendix A
Prompt for Gemini: When given a Python dictionary, check each key to see if there are conflicts in medication, diagnoses and allergies. For example, an incorrect medication for a particular diagnosis or an allergy with a prescription. If there are, return the same dictionary as JSON format. Each key with must have an array as a value with the array containing the indexes of the conflicts. Additionally, add another pair with a key "explanations" and another dictionary as a value with that dictionary containing explanations for the reponse. If there are no conflicts, the array value must be an empty array and the explanations key must contain a dictionary with empty strings. Here are example responses. Only respond with the "Output".
Example 1: Input: { "Name" : "Patient A", "Diagnoses" : ["High Blood Pressure", "Low Blood Pressure"], "Prescriptions" : ["Xanax", "Advil"], "Allergies" : ["Pollen", "Advil"] }
Output: { "Name" : "Patient A", "Diagnoses" : [0, 1], "Prescriptions" : [1], "Allergies" : [1], "Explanation" : { "Diagnoses" : "Patient cannot have high and low blood pressure.", "Prescriptions" : "Patient has an allergy to Advil.", "Allergies" : "Patient was prescribed Advil." } }
Example 2: Input: { "Name": "Patient 1", "Diagnosis": ["Depression"], "Prescriptions": ["Acetaminophen"], "Allergies": [] }
Output: { "Name" : "Patient 1", "Diagnoses" : [0], "Prescriptions" : [0], "Allergies" : [], "Explanation" : { "Diagnoses" : "", "Prescriptions" : "Should not use Acetaminophen to treat Depression. It is a painkiller.", "Allergies" : "" } }
Example 3: Input: { "Name": "Patient 5", "Diagnosis": ["Asthma"], "Prescriptions": ["Sildenfil"], "Allergies": [] }
Output: { "Name" : "Patient 5", "Diagnoses" : [0], "Prescriptions" : [0], "Allergies" : [], "Explanation" : { "Diagnoses" : "", "Prescription" : "Should not use Sildenfil to treat Depression. It is used to treat Erectile Dysfunction.", "Allergies" : "" } }
Example 4: Input: { "Name" : "Patient Alpha", "Diagnoses" : ["Chronic Headaches"], "Prescriptions" : ["Apixban", "Advil"], "Allergies" : [] }
Output: { "Name" : "Patient Alpha", "Diagnoses" : [], "Prescriptions" : [0, 1], "Allergies" : [], "Explanation" : { "Diagnoses" : "", "Prescription" : "Should not use Apixban and Advil together. Interaction between blood thinner and pain killer can increase risk of bleeding.", "Allergies" : "" } }
Example 5: Input: { "Name": "Patient Beta", "Diagnosis": ["Chronic Bronchitis"], "Prescriptions": ["Advil", "Dabigatran"], "Allergies": [] }
Output: { "Name" : "Patient Beta", "Diagnosis" : [0], "Prescriptions" : [0, 1], "Allergies" : [], "Explanation" : { "Diagnoses" : "", "Prescriptions" : "Should not use Dabigatran or Advil to treat Chronic Bronchitis. Dabigatran is a blood thinner and Advil is a pain killer. Should not use Dabigatran and Advil together. Can increase risk of bleeding.", "Allergies" : "" } }
Log in or sign up for Devpost to join the conversation.