Inspiration
Having many friends and family members take part in the medical school journey, it seemed like practicing patient interactions was a major pain point that could be improved upon. This led us to look at how we could leverage new technologies to solve that problem.
What it does
Our web application is able to take in a video file of a medical student's interactions with a patient, and present scores and feedback based on a detailed scoring rubric.
How we built it
We utilized Cohere's NLP library, along with FER, Google Cloud's speech-to-text capabilities, React to process our videos. To build the application, we utilized React and Flask.
Challenges we ran into
We had a lot of trouble at first using Cohere's NLP library, since it is a new technology for us. We also struggled with parsing through text and data to create the rubric, given that none of us are medical students. Lastly, we struggled with creating a large dataset to train the NLP on.
Accomplishments that we're proud of
For a few of us, this was our first time using Natural Language Processors and creating an entire full stack project. We are proud of our success in using the Cohere's NLP library, and how well the results turned out despite a small training dataset.
What we learned
We learned how to use sentiment analysis and other classification tools to analyze how empathetic a physician is when giving care. We also learned how to call the data and results that we process in the backend on the frontend.
What's next for Automated Physician Assessment
We want to include the full list of fields that physicians are assessed on in these exams. So far, we accessed use of jargon and types of questions, but there are many more such as pacing, transitional statements, and spectrum of concern.
Log in or sign up for Devpost to join the conversation.