Cancer-Rx-Approve: write letters for cancer drug denials

Inspiration and introduction

I have worked in the healthcare data science space for about 10 years. I have experience working for a large healthcare provider in the analytics department and had the opportunity to develop NLP software for providers and payers. While I have a seen a tremendous amount of progress during this time, I still see many areas where artificial intelligence could be utilized to improve patient outcomes. I am interested in finding ways to apply the latest advancements in AI to the healthcare space to make clinicians jobs better and create a better experience for patients.

The average chemotherapy costs per patient in the 24 months after a cancer diagnosis is between $71,90 - $182,655 (Blumen 2016). In response to the growth of high-cost medications in cancer care, many public and private insurers have increased the frequency of requirements for prior authorization (Agarwal 2017). Prior authorization is a burdensome administrative process where providers must get approval from insurance companies prior to delivering care to patients.

Healthcare insurance organizations deny roughly 1.40% of prior authorization requests (Schwartz 2022). With roughly 1 in 3 people getting cancer in their lifetime, this small percentage amounts to a significant number of patients. While there are many reasons a prior authorization is denied, often clinicians still believe the drug is necessary. When this occurs, clinicians must write letters to the insurance company outlining the clinical details and need for the drugs. These letters take up valuable time. In the letters, clinicians need to cite scientific studies to rationalize their choice of treatment and how it relates to the personalized details about the patient. Denials increase provider administrative work, physician burnout, and ultimately lead to worse patient outcomes. In the meantime, patients are unable to receive the treatments that they need. Streamlining the required work to receive prior authorization would save time for doctors and improve patient care.

Large language models have the potential to write well-crafted letters and effectively synthesize complex cancer data (Li 2023). We propose fine tuning large open source models with cancer data to develop an autonomous system that clinicians can use to accurately generate letters in response to prior authorization denials. The large language model would synthesize information from the patients’ medical records as well as recent research on the drugs to create a compelling letter. These letters would not only save clinicians valuable time, but also reduce the time patients need to wait to receive their life-saving cancer care.

What it does

We built a cloud-based web application that clinicians can use to generate letters to send to insurance companies when requests for chemotherapy are denied.

Input:

Clinical details of a case that was denied and why it should be approved:

The patient [Jane Smith] has breast cancer and was denied Trastuzumab. The patient tumor is HER2-overexpressing. I would like start with adjuvant treatment with doxorubicin, cyclophosphamide, paclitaxe, and Trastuzumab.

Output:

A personalized letter to send to the insurance provider to document (generated via LLM):

[Your Name] [Your Title] [Your Institution] [Date]

[Insurance Provider's Name] [Insurance Provider's Address] [City, State, ZIP]

Subject: Request for Coverage of Trastuzumab for Jane Smith

Dear [Insurance Provider's Name],

I am writing to appeal the denial of coverage for Trastuzumab for my patient, Jane Smith, who has been diagnosed with HER2-overexpressing breast cancer. I believe that Trastuzumab is an essential component of her adjuvant treatment plan, and its denial could have a detrimental impact on her overall prognosis and quality of life.

Jane Smith was recently diagnosed with invasive breast cancer. The biopsy confirmed HER2 overexpression in her tumor, which indicates an aggressive subtype of breast cancer with a higher risk of recurrence and worse prognosis. HER2 overexpression has been established as an important biomarker for targeted therapy with Trastuzumab in breast cancer patients. Trastuzumab, a monoclonal antibody targeting the HER2 receptor, has revolutionized the treatment of HER2-positive breast cancer. Its use in the adjuvant setting has consistently demonstrated significant improvements in disease-free survival (DFS), overall survival (OS), and reduced risk of recurrence when combined with standard chemotherapy. Several landmark clinical trials have established the efficacy and safety of Trastuzumab as an integral part of the treatment regimen for HER2-positive breast cancer.

For instance, the HERA trial (HERceptin Adjuvant) conducted by Piccart-Gebhart et al. (2005) demonstrated a 46% reduction in the risk of recurrence and a 33% reduction in the risk of death in patients who received one year of adjuvant Trastuzumab, compared to those who did not. The BCIRG 006 trial (Slamon et al., 2011) revealed a 40% reduction in the risk of recurrence and a 34% reduction in the risk of death with the addition of Trastuzumab to adjuvant chemotherapy.

The National Comprehensive Cancer Network (NCCN) guidelines, widely recognized as a standard of care for oncology, strongly recommend the inclusion of Trastuzumab in the adjuvant treatment of HER2-positive breast cancer. The guidelines emphasize the importance of Trastuzumab in improving outcomes and reducing the risk of recurrence, thus underscoring its clinical significance.

Given the well-established efficacy and safety profile of Trastuzumab, it is imperative that Jane Smith be granted coverage for this essential therapy. By denying access to Trastuzumab, we risk compromising her chances of achieving optimal disease control and long-term survival.

I kindly request a reconsideration of your decision and urge you to provide the necessary coverage for Trastuzumab for Jane Smith. Her health and well-being depend on timely access to appropriate treatment options, and I am confident that approving coverage for Trastuzumab aligns with the best interests of both Jane and your organization.

Thank you for your attention to this matter. I would be more than willing to provide any additional information or discuss the case further to assist in the reconsideration process.

Sincerely,

[Your Name] [Your Title] [Your Contact Information]

How we built it

We experimented with three open source large language models: tiiuae/falcon-7b-instruct, databricks/dolly-v2-3b, and google/flan-t5-small. We entered the same prompt (see Input example above) for each model and then annotated the quality of the output letter on a scale of 0-10 (see Figure 1).

After seeing the baseline performance of these models, we wanted to fine-tune the LLMs with cancer data to improve the quality of the letter. We utilized publicly available structured datasets (n=1046) from seer.cancer.gov and drugcomb. We developed custom python scripts to transform that data into a format that could then be used for LORA fine-tuning. We also included a small amount (n=10) of hand-crafted letters to insurance companies in our fine-tuning dataset.

We then annotated the quality of the output letters from the fine-tuned models on a scale of 0-10 (see Figure 1). The best model, fine-tuned tiiuae/falcon-7b-instruct was then deployed on AWS Sagemaker and connected to a Flask app deployed on an AWS EC2 server.

Figure 1: The performance of base and fine-tuned LLMs when asked to generate denail appeal letters.

Challenges we ran into

One challenge was finding open source datasets on cancer drugs and transforming that data into a format that could then be used to fine-tune a large language model. We were able to find large, structured datasets on cancer data on seer.cancer.gov and drugcomb. We then developed custom python scripts to take these datasets and transform them into a format used to fine-tune a LLM.

We also were very conscious of the costs associated with hosting and fine-tuning LLMs. These constraints forced us to use smaller models, but still enabled us to prototype our idea with the understanding that if it shows commercial promise, we can upgrade our base model.

Finally, because our end goal was to write a letter, it was difficult to determine a subjective way to judge the output of the models. Evaluating the "quality" of the letter that was generated was a subjective process. We developed a scoring system where an annotator would rank the letter output on a score of 0-10. Although imperfect, this allowed us to determine what types of base models and fine-tuning worked the best for our desired task.

Accomplishments that we're proud of

We have a working application! Check it out here Cancer-Rx-Approve. Clinicians can use this tool to generate letters that will save them time and result in patients getting the cancer care they need faster.

What we learned

Fine-tuning large language models on a specific dataset improves the performance if the desired task aligns with the fine-tuning dataset.
The LLM space is rapidly expanding. During the development of this project, a new LLM (Falcon) was released that ended up with the best performance.
Hosting LLMs are expensive! Applications developed with LLM backbones will need significant revenue to offset their costs.

What's next for Cancer-Rx-Approve

Create a button that providers can click to automatically send the letter to the correct insurance provider.
Develop a more objective way to judge the output of the models. Evaluating the quality of the letter that was generated was a subjective process.
Since new cancer drugs are continuously developed and approved by the FDA, this model will require continuous fine-tuning as state-of-the-art treatment becomes available.
Integrate electronic health records and HIPPA compliance so we can process protected health information.
Pilot the tool with a small cancer care center

Citations

Agarwal, Ankit, Rachel A. Freedman, Felicia Goicuria, Catherine Rhinehart, Kathleen Murphy, Eileen Kelly, Erin Mullaney, Myra St. Amand, Phuong Nguyen, and Nancy U. Lin. "Prior authorization for medications in a breast oncology practice: navigation of a complex process." Journal of Oncology Practice 13, no. 4 (2017): e273-e282.

Blumen, Helen, Kathryn Fitch, and Vincent Polkus. "Comparison of treatment costs for breast cancer, by tumor stage and type of service." American health & drug benefits 9, no. 1 (2016): 23.

Li, Tianhao, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xianqian Jiang, Ying Ding, and Yejin Kim. "CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models." arXiv preprint arXiv:2304.10946 (2023).

Schwartz, Aaron L., Yujun Chen, Chris L. Jagmin, Dorothea J. Verbrugge, Troyen A. Brennan, Peter W. Groeneveld, and Joseph P. Newhouse. "Coverage Denials: Government And Private Insurer Policies For Medical Necessity In Medicare: Study examines medical necessity coverage denials in Medicare and private insurers." Health Affairs 41, no. 1 (2022): 120-128.