Synoptel

Inspiration

Most countries in the world, including the United States, have rapidly shifted to telemedicine after the advent of the novel coronavirus. This includes the aged population namely those with chronic conditions like heart disease. According to the National Council on Aging, about 78 percent of older adults have at least one chronic disease that require regular check ups. These patients cannot leave their homes because they fall under the vulnerable population required to remain in isolation.

Old age brings in struggles of using technology and the inability to understand and retain vital information from video calls. We want to address both these challenges through the power of artificial intelligence. By providing the aged with an easy to use application that serves as a one-stop shop for summarised medical consultation records, we aim to create a more comfortable and secure telemedicine environment.

What it does

Synoptel is a web-based application that converts the doctor-patient conversation from live audio/video calls to text. It extracts elements such as the symptoms of the patient, the diagnosis by the clinician, medication proposed, and other treatments advised. This information is presented to the patient in the form of tabulated notes/lists and is saved on the patient's end. Before reaching the patient, the notes are reviewed by the clinician himself ensuring utmost accuracy. This ensures that the patient clearly understands everything discussed during the call and reduces the risk of patients forgetting to the names of medications and/or following instructions given by the doctor. The doctor also has access to these notes so as to have a simplified version of patient records available.

In the present era of humanitarian crisis (CoViD-19), where access to healthcare services is getting difficult, this serves as a tool to bridge the gap.

How I built it

After the video conferencing gets over, we would be converting the video codec into an audio embedding using an inbuilt library for audio/video processing. This audio would then be converted into transcripts (as words and sentences, using the Amazon AWS Transcribe API). We then filtered out relevant entities and major metadata using the AWS Comprehend-Medical model, which helped in removing all the useless words (stopwords), and applying an Extractive BERT model to convert the leftover sentences into word embeddings, and then into feature vectors. These vectors were then classified in metric space as symptoms, medications or treatment procedures. The final summary, after being approved by the doctor, was shown to the patient, in the form of a web application.

Challenges I ran into

1. We had problems integrating AWS with the web app with minimal latency
2. The categories of the labels often overlapped, so we needed to decide individual categories for some borderline cases
3. Integrating the app with a live application is the future scope and has nearly been completed

Accomplishments that I'm proud of

1. Joining a team of brilliant minds and extreme talent cross professional fields.
2. Approaching a healthcare problem that could prevent a number of deaths.
3. Successfully making our way through stats and tech highs.
4. Having had the chance to be mentored by stalwarts from every sector.
5. Working around 48 hours, we’ve explored softwares and mastered them overnight, running on (and sometimes crawling) coffee & calls - all from our safe space.

What I learned

Precise case notes are crucial for not only the doctor but also the patient. So many conditions get worsened because the patient cannot fully grasp the doctor's instructions. We learned that this problem has become much more serious during this pandemic. Put together, my team members and I have explored and learned softwares we were previously alien to. We learned how to work together over online communication media.

What's next for Synoptel

The reach of Synoptel is diverse. It is open to exploration and we believe we have the capability to develop a system for:

1. Sending alerts to remind patients to take their medications based on call notes
2. Alerting the hospital system/your clinician via video/text message alert
3. Setting reminders of your daily therapeutic doses for the patient
4. Reminders added to your near ones’ calendar too so you have a full support system.
5. A full power Chatbot to ease patient interaction

Built With

• amazon-web-services-(aws-transcribe
• angular.js
• aws-comprehend-apis)
• react.js