Chart Review Companion

Inspiration

I have been working in healthcare analytics for the past 10 years in a variety of roles such as at a large academic hospital and a small startup focused on applying artificial intelligence to the healthcare industry. I am very interested in finding ways to incorporate the latest machine learning models into healthcare and I believe that large language models (LLMs) provide an opportunity to enhance the efficiency of chart reviews.

Chart review is the process of gathering relevant data from patients’ medical records. It is commonly used in clinical trials to monitor and extract data. During my experience I often see chart reviews as a burdensome process for clinicians because it is time consuming and expensive.

Applying LLMs to improve chart reviews can benefit many areas in healthcare. For example, it can reduce the costs involved in drug development. This is important given the average cost to bring a pharmaceutical drug to market is $985.3 million and rising (Oliver 2020). Such high costs restrict drug development to only commercially viable drugs. Reducing these costs could lead to lower medication prices and broaden the range of treatable conditions, ultimately benefiting patients (Atlas 2020). My goal is to leverage AI technology to improve the current drug development chart review process.

What it does

Chart Review Companion aims to cut drug development costs by automating and accelerating the chart review process. It will read the clinical chart and compare it with the trial protocol to identify discrepancies and catch errors, a task typically managed by contract research organizations.

Our app enables users to simply upload the clinical trial protocol and patient charts to a webpage. Our AI system then compares the clinical charts with the clinical trial protocol and outputs potential issues in the patient care. For instance, it can flag deviations from the recommended protocol or highlight patients at risk for specified safety measures. Additionally, the Summary tab provides an overview of all identified issues, enabling users to detect common trends or problems.

Overall, I hope to build Chart Review Companion as a comprehensive solution for Clinical Research Associates (or similar roles) to use as a platform for their chart reviews.

How we built it

I utilized state-of-the-art language models and natural language processing techniques to create a system capable of accurately extracting relevant information from clinical charts. The system processes queries with the context of the entire patient chart and clinical trial protocol to identify issues.

In order to test our system, I fabricated 10 patient charts that could have been part of this clinical trial found in clinicaltrials.gov. Four of these charts included deliberate errors, such as incorrect drug dosage, wrong cycle day for drug delivery, improper timing of administration, and contraindicated drugs. These errors were introduced in order to ensure our system could appropriately flag them for additional review. I experimented with various prompting strategies in order to ensure that all 4 errors were correctly identified by the LLM.

Challenges we ran into

One major challenge was dealing with the complexity and variability of medical language and terminology. I had to develop a custom prompt for our specific use case and expect there may need to be “fine-tuning” of the LLM for subsequent use cases. Finding a suitable dataset to test our LLM's issue identification strategy was difficult due to strict regulations on healthcare data. I had to fabricate the 10 patient charts myself. Although I have the clinical experience needed to do this, it is time-consuming and not scalable.

Accomplishments that we're proud of

The app is working and live! I hope the system can significantly reduce the time and effort required for clinical chart review, potentially saving millions of dollars in drug development costs. The accurate and efficient extraction of relevant data from complex medical records is a significant technological achievement.

What we learned

This project provided valuable insights into the challenges of working with medical data and the necessity of developing robust and interpretable AI systems for healthcare applications. I also learned the importance of close collaboration between domain experts and AI researchers.

What's next for Chart Review Companion

I plan to refine and validate our system across a broader range of clinical trials and therapeutic areas. Ideally, I hope to partner with a clinical research organization to pilot our software with clinical research associates, enabling us to gather real user feedback and iteratively improve our product. Additionally, I aim to integrate our solution with electronic health record systems and clinical trial management software to streamline the entire drug development process.

Citations

Atlas, Scott. "How to Reduce Prescription Drug Prices: First, Do No Harm." Missouri medicine 117, no. 1 (2020): 14.

Wouters, Olivier J., Martin McKee, and Jeroen Luyten. "Estimated research and development investment needed to bring a new medicine to market, 2009-2018." Jama 323, no. 9 (2020): 844-853.