MediLingual

Italian translation from a patient to a doctor speaking English

Inspiration

At the heart of MediLingual's creation lies the story of Parthiv's father, an immigrant doctor who couldn't speak English and had to swiftly adapt and learn to support patients. Not only as a doctor but as a patient himself, he experienced severe hardships in receiving and giving care. He realized that the quality of care provided during times when his English was very limited was subpar and that patients could have been treated much better if the language barrier was destroyed.

At the heart of MediLingual's creation lies the story of Brayden's mother, an ER nurse dedicated to serving a large Hispanic community. She routinely has to laboriously pull out her phone and painstakingly pass her phone back and forth, dealing with poor translations of both language and medical lingo.

Recognizing this challenge, we envisioned a better way—a universal translator tailored specifically for medical contexts. Thus, MediLingual was born, empowering healthcare professionals like Brayden's mother to communicate seamlessly with patients and Parthiv's father to communicate as a patient and a doctor, ensuring that language barriers never compromise the quality of care.

What it does

MediLingual provides bedside advanced medical transcription services by recording that records audio data, automatically detects the language, removes all the inaccessible/untranslatable medical lingo, scrapes the symptoms from the dialogue, and provides a clean and easily readable transcript for both doctor and patient.

How we built it

We've built a Flask API to store audio files locally. With Whisper, we transcribe and translate these files, ensuring accuracy. Then, using OpenAI's API, we refine, parse, and translate the text. Our React frontend features a sleek, easy-to-use design for healthcare workers and patients.

Challenges we ran into

Incorporating a medical jargon to layman terms Medical jargon is largely untranslatable and often unintelligible to the patient, ESPECIALLY across the language barrier. It was hard to prompt GPT to try to detect what jargon people might not understand, as this is case-by-case. We thought to scrape a medical dictionary and send this along with our prompt to try and replace jargon with more intelligible definitions, which would end up using more layman terms and a description. We found this to be quite a large text, so we dabbled in few-shot prompting techniques, and it worked just as well, giving us clean and readable text that is way easier to translate.
HIPAA compliance Initially, we were intending to use an AES-encrypted database to store these audios so the patient and doctor could retrieve past conversations. This becomes a massive liability as recording and storing a patient's audio requires consent, and the case of a data breach comes with its own load of problems. To utilize a database while maintaining security, we decided to use Supabase running on a docker container locally We also wanted to minimize calls to databases that might store this sensitive information, so we tried using a locally hosted LLM (Ollama), but we had problems with the quality of the translation and wanted to show the true capabilities of this software, so we are using OpenAI.
Audio Ironically, the hardest, most time-consuming, and grueling task was literally just getting audio from the front end and storing it locally. But we got it done!

Accomplishments that we're proud of

We are proud of making new friends, learning Flask, and creating a project that personally helps the people closest to us.

What we learned

We learned Flask, Tailwind CSS, the trenches of React, Google Cloud storage/API (it worked but is not used), locally hosting LLMs, medical jargon and HIPAA, and most importantly, how to get audio from the front end.