Medslate AI

Inspiration

Our team’s members have experience working in medical environments through various roles. Throughout our time in these environments, we often noticed the issue of patients struggling to understand, or misunderstanding, what their doctor is saying. This is a tremendous issue and one that causes confusion and harm to countless patients every year. Inspired to tackle the problem, and improve the lives of patients and medical professionals, we decided to create Medslate.

What it does

Medslate is intended to be used during appointments by either a medical professional or a patient. During the meeting, Medslate will record the conversation between the doctor and patient to produce a transcript. After the recording is complete, users can produce a “Medslation” with three central parts: A simplified description of the transcript, key points from the appointment given in simple language, and clarification questions that the patient may find helpful to ask or for the doctor to address.

How we built it

Read research articles to learn more about the issue of comprehension during medical appointments. We used this step to formulate ideas about the issue and decide how we might use software to solve it.
Designed the application and its layout, alongside diagramming the architecture, on blackboards.
Decided on our tech stack and researched possible tools we may have to use to develop our application.
Developed the UI of the application alongside most of the necessary backend routing for displaying information on the front-end.
Designed secure cloud storage for audio files. For this, we decided to use AWS S3. We then integrated the system to record voice in our web application, which we used to test our backend.
Integrated OpenAI’s GPT-4 model to process transcripts and display the final output to users on our front-end.

Challenges we ran into

Voice recording: None of our team was familiar with how to integrate voice recording in a web application. However, we researched the technology, and found a way to produce audio files by recording from a microphone. However, we were struggling to understand how to store our files and use them to transcribe the appointments to text. After research and deliberation, we decided to use AWS Transcribe and AWS S3.
Cloud (AWS): Uploading audio files to S3 was difficult, since there was a lack of documentation on how to do so. This was a problem we ran into early-on, and it required us to do a lot of diagramming of the architecture and reading of documentation. Another problem was when we were attempting to integrate AWS Transcribe with our S3 bucket.
Large Language Models: Going into HopHacks, we did not have much experience with OpenAI’s API. Learning the API was a challenging, but rewarding, component of our development experience.
Application Design: We set out to make an application with the least amount of clutter and unnecessary pages. An easy user experience, suitable for quick set-up during medical appointments, was a significant goal of ours.

Accomplishments that we're proud of

Learning technology: Each of our team members was able to spend time reading and learning during the hackathon, and we all became better developers as a result.
Using new tech: We were very happy to be able to use novel software (such as OpenAI’s LLMs) in our application and to explore the use of these technologies.
Teamwork: All of our team contributed from their respective skills, and we had a great time working together :)

What we learned

Became more familiar with full-stack application development, cloud services, and voice integration
Learned about AWS software and how we can apply it for our projects
Familiarity with machine learning-based application development through the OpenAI API

What's next for Medslate

We plan to expand Medslate with features such as multiple language support. In addition, we would like to see if we can train the LLM to produce better outputs for patients. We would also like to develop the application for other devices, outside of web browsers. Furthermore, we would like to broaden our features to improve patient’s understanding of their medical appointments and history. If we are successful, we would love to test our application with hospitals/clinics!