What's Up Doc

• 

  What's up Doc

Inspiration

Residency students and new doctors often struggle with real-world decision-making despite having strong theoretical knowledge. We wanted to build a tool that helps bridge the gap between textbook learning and practical application. Additionally, doctors deal with overwhelming amounts of patient interactions, and we saw an opportunity to streamline communication and decision-making using AI.

What it does

Our project, What's Up Doc, is an LLM-powered web application designed to assist new doctors in two key ways:

AI-Powered Medication Recommendations: Using a Retrieval-Augmented Generation (RAG) chatbot, doctors can explore medications across pharmaceutical companies and receive recommendations based on a patient’s medical history. This helps new doctors make informed prescribing decisions.

Patient Interaction Assistance: The app can analyze voicemail messages from patients, perform sentiment analysis, and generate a concise summary along with suggested next steps for the doctor. This reduces cognitive load and helps prioritize patient needs efficiently.

How we built it

Our app leverages Google Gemini’s advanced capabilities alongside LangChain as the core framework for orchestrating AI-driven workflows. We integrate Assembly AI for high-accuracy transcription and sentiment analysis, enabling precise and meaningful summaries of patient interactions. Our approach follows the Retrieval-Augmented Generation (RAG) paradigm, where we curate and utilize multiple datasets to enhance the accuracy and contextual relevance of our responses. This combination ensures that doctors receive reliable medication recommendations and streamlined patient communication insights, improving decision-making efficiency in real-world medical settings.

Challenges we ran into

Setting Up and Integrating APIs – While we had experience with AI models, integrating Google Gemini with LangChain and Assembly AI required a learning curve. Ensuring smooth communication between different APIs and handling authentication issues took longer than expected.

Optimizing RAG for Medical Data – Implementing Retrieval-Augmented Generation (RAG) meant finding relevant datasets and fine-tuning our retrieval pipeline. We struggled with filtering out irrelevant information and ensuring our chatbot provided accurate, context-aware medication recommendations.

Latency Issues – Running LLM-powered queries while retrieving information from multiple sources caused noticeable delays. We had to optimize our API calls and caching strategies to make the chatbot and transcription features more responsive.

Building a User-Friendly Interface – Since doctors have limited time, we wanted a clean, intuitive UI. Balancing functionality with simplicity was challenging, especially when presenting complex AI-driven insights in an accessible way.

Accomplishments that we're proud of

Successfully Integrated Google Gemini with LangChain – We built a working RAG-powered chatbot that retrieves and recommends medications based on patient history, making it a valuable tool for new doctors. Built a Functional and User-Friendly Web App – We designed an intuitive interface that simplifies complex AI-powered insights, making it easy for doctors to navigate and use efficiently.

Learned and Applied New Technologies – As first-time participants, we quickly adapted to LLM integration, RAG, and AI-powered transcription, expanding our skill set in a short time.

Completed a Fully Functional Prototype in 36 Hours – Working under intense time constraints, we successfully developed, tested, and deployed a prototype that can make an impact in the healthcare space.

What we learned

The importance of data preprocessing in AI-powered medical applications. How to integrate LLMs with external knowledge bases for RAG-based responses. Effective team collaboration under time pressure—splitting work efficiently was key! Real-world medical AI solutions need to be interpretable and responsible to ensure safety and compliance.

What's next for What's Up Doc

Enhancing medical knowledge retrieval by incorporating more real-world case studies. Improving sentiment analysis accuracy by training on domain-specific healthcare datasets. Deploying the app and testing it with real users to refine its usability.

Built With

• flask
• gemini
• javascript
• langchain
• python
• rag
• react