AI Clinical Lab

App UI
Intergration of OpenAI GPT-4 in the app

Inspiration

The inspiration behind integrating multimodal analysis into AI Clinical Note stemmed from the need to comprehensively understand medical data from diverse sources and data formats, aiming to save time for doctors and healthcare providers during diagnosis and treatment planning.

AI Clinical Note aligns with Track 1 challenges by modernizing symptom monitoring, bringing patients and healthcare providers closer together. It harnesses real-world data through multimodal analysis, enabling efficient diagnosis and treatment planning while prioritizing patient privacy and compliance with regulatory standards.

App UI

What it does

The AI Clinical Note integrates OpenAI GPT-4 to analyze multimodal data from lab reports, voice notes, and chest X-ray images. By leveraging this cutting-edge technology, it highlights abnormal lab results or signs and provides suggestions on potential conditions and diagnoses for patients. This comprehensive approach enhances the efficiency and accuracy of diagnosis, ultimately improving patient care and outcomes. Through AI summarization and suggestions from GenAI, we lay out all the information doctors need for quick diagnosis at a glance. Hence, we can achieve more efficient diagnosis in a shorter amount of time and enable doctors to focus more time and energy on patient treatment.

How we built it

Structure

We constructed AI Clinical Note by harnessing the OpenAI GPT-4 API for text analysis from multimodal data. We utilize the Python Speech Recognition package to extract text from voice notes and the pretrained TorchXRayVision package to generate a potential list of lung-related diagnoses with probabilities from chest X-ray images. Moreover, we employ the PyPDF2 package to extract text from PDF documents. These extracted texts from different modalities are amalgamated into a cohesive body of text, serving as a reference for GPT-4 to address prompt questions by highlighting abnormal lab results, complaints from voice notes, and potential conditions with high probabilities identified in chest X-rays.

Challenges we ran into

Find the functioning packages to process the PDF test, chest X-ray image and voice note to work seamlessly for our multimodal analysis pipeline that seamlessly posed significant technical challenges due to differences in data structures and formats.

Navigating the Azure Portal and OpenAI Studio can be quite a challenging task, especially when managing resources and deployments. It often requires time to familiarize oneself with the interfaces and understand where to find specific settings or features. As for managing costs, it's essential to keep a close eye on the usage of resources and the associated pricing. Leveraging the free credits efficiently requires careful planning and monitoring to avoid unexpected charges. Balancing the computational requirements of the models with available credits is crucial for optimizing costs while maintaining the desired level of performance.

Accomplishments that we're proud of

Despite the complexities involved, our team successfully created AI Clinical Note —a cutting-edge tool that combines multimodal analysis with responsible AI principles. We take pride in providing healthcare providers with a reliable and compliant solution that accelerates the lab test analysis process while prioritizing patient privacy and safety. What we learned Through the development of AI Clinical Note, we gained valuable insights into the challenges and opportunities associated with multimodal AI analysis in healthcare. We learned the importance of balancing innovation with regulatory compliance and ethical considerations. Moreover, we deepened our understanding of the critical role of responsible AI practices in building trust and transparency in AI-powered healthcare solutions.

What's next?

Looking ahead, we aim to further enhance AI Clinical Note's capabilities and expand its reach:

• We plan to continue refining the voice analysis algorithms by integrating the Whisper model and also GPT-4 Turbo Vision model from OpenAI to improve accuracy and efficiency in voice processing.

• Ensuring compliance with HIPAA regulations and FDA requirements for medical software necessitated meticulous attention to data privacy, security, and regulatory standards.

• Aligning the responsible AI principles with the need for speed in processing lab reports required balancing efficiency with ethical considerations to ensure patient safety and trust.