MedScribe

Landing Page

Inspiration

We have all experienced the long wait at the doctor's offices and we all know that trips to the doctor’s office are very time-consuming and expensive to fit into people’s busy schedules. In addition, doctors are often overbooked to see a higher volume of patients, leading to higher burnout that risks patients who may need more critical care.

According to Time Magazine, "Appointments get compressed to fifteen minutes, and a typical primary care doctor's panel swells to over 2,000 patients. All the extra work spills into doctors' off hours." With only 15 minutes allocated, doctors may struggle to fully provide all details of a diagnosis and patients may struggle to remember and juggle all the information dumped on them. Furthermore, overworking doctors risk the health of all patients.

With appointments condensed to just 15 minutes and doctors overloaded with patients, the current healthcare system strains both doctors and patients alike. Thus, we propose MedScribe, a personal assistant that empowers patients to gain more control over their healthcare.

What it does

MedScribe revolutionizes the patient experience by enabling patients to chat with recordings of their doctor visits. This allows patients to fully process a doctor's instructions with additional medical knowledge, takes the some of burden off of the doctor's shoulders, and breaks down geographical barriers for rural patients seeking healthcare. Overall, MedScribe saves valuable time for both patients and doctors, reduces healthcare costs, and allows patients to take a more active role in their health.

How we built it

As a real doctor who sees, hear, and read, we utilize the power of multimodal large language models and audio embeddings to process text databases, audio recordings, and image inputs from a patient's visit to get the full picture of a patient's condition. We utilize OpenAI Whisper to embed audio extracted from videos of visits and add them to a vector database alongside text embeddings of medical knowledge, doctors' notes, etc. We then use OpenAI's GPT4 for multimodal understanding and produce a chat conversation with the patient for question answering.

Challenges we ran into

It's the first time most of our team members have done full-stack web development, so linking the backend with the frontend with FastAPI using Python was quite challenging for us. Coming up with a framework that linked all three modalities was also tricky.