Inspiration
Our friends who work as a clinician have always had a really hard and long day. Us as graduate students focused on AI wanted to see how we can leverage our domain knowledge to make their work easier. We have experiences with clinicians where we try to express our symptoms and concerns but they are carried away with taking scribe notes. BOOM! why can't we do a voice recordings of an interaction with the doctor and patient, transcribe it in real time.
What it does
Why stick to only transcribing when AI can do much more? We have built a complete doctor's portal which is future proof with the patient's EHR and accepts X-rays as well. We have built a multi-modal-RAG agent which also provides clinicians with potential questions to analyze what the patient's condition with less burnout.
How we built it
We have a image model which analyzes the patient's X-ray. We fetch patient's medications, EHR and add a fusion of both of these data and pass it on to a RAG which increases it's confidence score with every interaction with the patient. Using this complete approach the application supports the clinician with burnout and support with diagnosis.
Challenges we ran into
We have very less publicly available medical data. Did not have access to free API keys, and definitely very less time to build a ready for deployment application.
Accomplishments that we're proud of
Our idea which turned into a POC with all the limited resources along with very very less time.
What we learned
We have learned a lot about medical domain and how an idea can come into reality and how much it can contribute to the society.
What's next for Multimodal Clinical Copilot
Expand the domain from X-rays to more MRIs, blood tests, etc.
Built With
- chromadb
- classifiers
- cnn
- fastapi
- genai
- groq
- langchain
- llm
- rag
- whisperx
Log in or sign up for Devpost to join the conversation.