4M13 - Git Pushers - InclusiveHealth

With globalization, our cities are becoming diverse. This can led to communication barriers, especially in matters such as healthcare. Our project aims to solve this, increasing inclusivity in cities

Comment

Inspiration

The inspiration for this project came from personal stories of our teammates and their experiences through the healthcare system. Many were exposed to the challenges their loved ones faced when trying to communicate with healthcare providers. Some always went to the clinic with someone who could translate or went through the struggle of trying to call someone to translate remotely for them. This led to the inspiration of creating InclusiveHealth, an easy-to-use tool to preserve the dignity of patients through their interactions with doctors.

What it does

We developed a multilingual web application that can automatically detect the language being spoken, provides real-time translation, and produces transcripts and summaries of the conversation. This can be used in healthcare settings between patients and providers. The application is login-protected with an email confirmation code sent upon sign up. This allows an additional security to the patient's data and helps it maintain confidentiality. The user, once signed-in, can then utilize the record feature for live-action translation in 5+ languages. The transcript of this is then stored for future review and summarization for the patient.

How to access the code

1st link is to the github repo we used initially to add our workflow and worked on it together

2nd link is to the cleaned+updated frontend version of the github repo(used in the the 1st link)

3rd link is to the vercel link to the deployed version of the website we created

How we built it

Our pipeline works by first getting the user input from our website. The frontend was built using next.js and tailwind. For routing with files, next.js was used, also utilizing react. We additionally added the login in/sign up authentication to keep the website secure. The authentication was done using AWS's Cognito while the database used Amazon's DynamoDB.

After signing in, doctors can create sessions and invite patients through their email and patients are able to accept the session invitation. The user, i.e. both the doctor and the patient can click the record button and speak into it. This input is saved as an .mp3 file and is then uploaded and stored onto a bucket in AWS's S3. From here, the backend calls AWS's Transcribe, Translate, and Polly extensions to convert the mp3 audio video into text, translate it, and perform text-to-speech. This is done via AWS SDK (boto3). The outputted audio file is then stored in another S3 bucket that is accessed by the frontend and displayed for the user to listen/read.

Challenges we ran into

  • During the ideation process, we struggled with narrowing our ideas done and connecting it to the given prompt. This pushed us to approach mentors and get their input as to what they think. We received impressive and clear feedbacks as to what we should choose to do.

  • When we initially started coding the backend, we wanted to use OpenAI's API to handle transcribing and translation. However, we realized that we had to pay to use their API. This encouraged us into using the Transcribe, Translate and Polly provided by AWS, allowing us to explore and extend our understanding of services provided by the AWS.

  • Another challenge we ran into was getting the Log in and Sign up connected with Cognito and the backend authentication. During the integration of AWS Cognito with our frontend login and backend authentication flow, we struggled with a redundant issues that prevented us from successful user login. Specifically, attempts to log in with valid email and password credentials consistently resulted in HTTP 502 and 503 errors. These errors meant the backend service unavailability or misconfiguration in the authentication pipeline. After extensive debugging, we found out that the root for this issue was a misconfigured Cognito user pool setting: the email attribute was not enabled as a valid login alias. This resulted in Cognito rejecting login attempts using email addresses, even though users were registered with them. Once we updated this configuration, the login flow began working as expected.

Accomplishments that we're proud of

We are proud of the following:

  • The live translation happening in real time and the auto-detection of the languages spoken
  • The login authentication since it took us a long time to debug the login Overall, we are extremely proud of the deployed version of the website since the UI heavily focuses on a clean interface while enhancing user experience.

What we learned

Most of us were not familiar with the services/packages provided from AWS. This project allowed us to explore this area and get an hands-on experience with Cognito and DynamoDB from AWS. Additionally, we also didn't have experience with using the Transcribe, Translate and Polly, a text-to-speech API. This gave us direct experience with using a text-to-Speech API in our program.

What's next for 4M13 - Git Pushers - InclusiveHealth

Next steps to follow for InclusiveHealth are as follows:

  • Implementing the ai chat with the patients for them to interact with after the session with doctor for deeper understanding of the condition or answering any questions they might have.
  • Implementing a way for the patients to access their past records and make the website as a one place to go for the health records/information
  • Adding in a hardware component which is portable and can be easy and can be placed at the doctor's clinic to use especially for countries with languages barriers

Built With

Share this project:

Updates