MediCord

Making every second in hospitals count by transcribing and summarizing recordings of hospital visits.

Comment

Inspiration

During this pandemic, doctors have had less time to visit their patients because they have more patients and more guidelines to follow. This has made doctor visits more valuable, but sometimes they can be rushed. Patients simply cannot get the care that they need and deserve, and it is vital that they get as much information/help as possible from their visits to hospitals. That is why we created MediCord, which lets users record their conversations with their doctors and automatically transcribes and summarizes them using machine learning. This way, they can keep track of their doctor visit history while also having access to summaries that save them time by giving them only the most important information.

What it does

MediCord lets users create accounts to store their records of doctor visits, or sessions. After creating an account, a user can add any number of sessions, each with a title and an uploaded audio file (a recording of a doctor visit). After inputting this information, our web application uses machine learning to transcribe that audio file into text and summarize it. Each account stores all of the sessions a user has ever made, which means users can keep track of their medical records for a long time.

How we built it

We built the frontend of the application using React.js, SCSS, Tailwind CSS, and React Bootstrap. For our backend, we used several technologies. We used MongoDB for authentication and storing session information for each user. We used Firebase Storage to create public download URLs for the audio files given by the user, which let us transcribe the file. To transcribe the file, we used AssemblyAI Speech to Text API. This API takes in a link to an audio file (created using Firebase) and returns a transcript of the audio by using deep learning and natural language processing. To summarize this transcript, we used a Node.js library called node-summarizer. All of these technologies communicated with the frontend using Node.js and Express.js.

Challenges we ran into

We ran into several challenges with the backend, primarily with the audio transcription. We started out by using the Google Cloud Speech to Text API, but it did not work for our hack. We switched to IBM Watson's Speech to Text API, but that also did not work. We then found AssemblyAI's Speech to Text API, which worked for us, so we used that in our final product.

Accomplishments that we're proud of

We are proud of getting the Speech to Text and summarization features to work, as all of us don't have much experience with machine learning, let alone AI/ML APIs.

What we learned

We learned a lot about troubleshooting and being flexible, as even though we faced several roadblocks on the way to our final product, we worked through it and made sure our final product was as good as it could be. We also got experience with all of the technologies we used and reading documentation.

What's next for MediCord

Hopefully, we can help real people with this product. We also want to add more features powered by machine learning so users can get more information out of their audio files.

Share this project:

Updates