ClinAssist

What it does

ClinAssist accepts pdfs and references them to execute a variety of clinical tests. We have currently implemented the Mini Mental State Exam as per the guidelines of the Merck Manuals. From this documentation, the agent can administer the examination through pure audio input and output, assist the patient whenever questions arise, and summarize the patient's performance upon the conclusion of the test. This framework allows clinicians to focus on administering treatment and providing care, rather than performing routine diagnostic work. By automating these repetitive tests, we save clinicians valuable time and allow them to more effectively treat their patients.

How we built it

We started by creating a pipeline between our test scripts and the LLM. We did this by using Python's pdfplumber library to scrape text content from pdfs and convert them into a readable format. From there, we built in the audio component of our project with a Python backend that used GPT-transcribe and GPT-Text-To-Speech to dynamically converse with users. With this complete, we create a summarization agent to summarize the test results and store them in our SQLLite database. We retrieved this data to implement a test history page that displayed summaries of users' taken tests. In order to secure this process, we introduced user authentication using FirebaseAuth, which ensured that users could only see their own data. Finally, to ensure that the LLM output was consistently polite and professional towards the user, we placed guardrails on the agent's outputs that blocked toxic language.

Challenges we ran into

We initially had issues finding the proper library to scrape the pdf files. After jumping between libraries with dependency issues, we eventually settled on a simpler library that was suitable mainly for less complex pdf structures. After that, we faced hurdles in recording and outputting audio. We first tried to return raw bytes to the frontend and eventually had to encode and re-encode our audio data for it to successfully output. Then we spent a lot of time honing our agents' prompts to ensure that outputs were as consistent and relevant as possible. A common issue was finding the balance between having precise and detailed prompts and inundating the model with too much information.

What we learned

We learned a lot about full-stack development, especially due to the constant communication that existed between the backend and frontend. We had to be more careful about how we managed our tasks to ensure that that model was able to respond quickly to users. Many of the issues we faced forced us to be strategic about our approaches to implementing our vision.

What's next?

We would like to enhance the integration features for tests other than the Mini Mental State Evaluation. We have a pipeline that allows for the upload of pdfs to the database, and we are looking forward to eventually allowing users to upload their own tests for the agent to conduct. Our main goal for the future is to increase the flexibility of our product, allowing it to adapt more smoothly to various situations and become an easy-to-use tool.

Built With

• javascript
• next.js
• openai
• python
• react