Inspiration
Medical providers and emergency responders alike are exposed to a wide variety of patients every day, on average taking as many as 30 minutes per interaction. In these patient interactions, providers must remember most, if not all, details of the interaction to create a comprehensive report on the patient. To build an effective profile on the patient, providers also need to remember events where the patient exhibited a change in emotional state, and the reason for the change. Auralis was created to provide the aid for providers to recall important details throughout a patient interaction that may be lost without, building an extensible framework for training medical staff.
What it does
Auralis takes a video input of a provider-patient interaction and performs extensive analysis on the video for moments where the patient's emotional state changed, identifying those moments in a consumable manner for the provider. Auralis also provides a full transcript of the conversation, breaking it down into consumable, navigable chunks for effective readability. Finally, Auralis displays all uploaded provider-patient interactions in a concise list on a beautifully modern dashboard.
Privacy
Auralis takes patient privacy seriously, complying with HIPPA regulations. Auralis was built with the idea that videos of provider-patient interactions were ephemeral; the videos are to be archived complying with any set retention policy, saving important details of the conversation while retaining confidentiality.
How we built it
Frontend
Auralis utilizes NextJS for the frontend with Tailwind CSS and shadcn components.
Backend
Auralis was built on top of the AWS platform utilizing Cognito for secure authentication, S3 for reliable, secure storage of video files, EventBridge for communicating new file uploads from an S3 bucket, Trancribe for accurate and fast speech to text transcription of provider-patient interactions, and Transcribe Call Analytics for performing sentiment analysis on the interaction, noting positive and negative sentiments throughout the conversation. To tie together the processing pipeline, Auralis uses Step Functions to execute transcription and sentiment analysis, triggered by EventBridge.
Challenges we ran into
One of the biggest challenges we ran into was deciding how we wanted to perform speech transcription and sentiment analysis. We originally believed that developing the pipeline from open source tools would be best; however, upon further investigation, AWS proved to be a more effective solution due to how much of the underlying infrastructure was already implemented. Our original vision for Auralis was to do real-time analysis on a video stream with a GoPro camera, though the GoPro camera we had was incompatible with the Open GoPro SDK and could not be leveraged to produce a live video stream in the time allotted.
Accomplishments that we're proud of
We're most proud of the solution to look to AWS for implementing the majority of our infrastructure. AWS allowed us to focus on the solution rather than the underlying infrastructure, optimizing both time and cost of implementation.
What we learned
Going into this hackathon, we had never implemented a video processing pipeline like the one we built in Auralis, nor had we ever utilized AWS services to build out infrastructure. Building on top of AWS in such a short time gave us a wide breadth of experience and makes us confident to build more with the platform.
What's next for Auralis
Future work in Auralis will look to transforming the medical training landscape, providing prospective medical practitioners with ample reference on patient interaction and extensive feedback on their own performance with a patient, comparing past patient interactions.
Log in or sign up for Devpost to join the conversation.