clinic_assist_with_online_and_offline_support

Offline-first AI for low-resource clinics. Predicts diabetes, heart disease & pneumonia, explains results with XAI, and uses Gemini for plain-language/Bangla summaries online.

Comment

Clinic Assist with Online and Offline Support

What it does

Clinic Assist is an Explainable AI (XAI) powered medical decision support system designed specifically for resource-constrained healthcare environments in Bangladesh. The platform addresses a critical challenge: doctors in rural and underserved areas must see a high volume of patients daily with limited diagnostic tools and no access to specialist consultations. Our system acts as an intelligent assistant that not only predicts medical conditions but also explains its reasoning in a way that doctors can understand, trust, and act upon.

The system supports three core medical prediction modules:

1. Diabetes Risk Prediction — Using tabular clinical inputs such as glucose levels, BMI, insulin, blood pressure, skin thickness, pregnancies, diabetes pedigree function, and age, the system predicts diabetes risk. It provides rule-based natural language explanations alongside static SHAP and LIME visualizations that show exactly which factors contributed most to the prediction. For example, if a patient's glucose level is the primary driver of a high-risk prediction, the system highlights this clearly so the doctor can focus their clinical attention accordingly.

2. Heart Disease Prediction — This module processes engineered clinical features including age, sex, chest pain type (cp), resting blood pressure (trestbps), cholesterol (chol), fasting blood sugar (fbs), resting ECG (restecg), maximum heart rate achieved (thalach), exercise-induced angina (exang), oldpeak, slope, number of major vessels colored by fluoroscopy (ca), and thalassemia (thal). Using a pipeline-based inference approach, it generates risk assessments with interpretable feature importance scores, helping doctors understand which cardiovascular indicators are most concerning for each patient.

3. Pneumonia Detection — This module accepts uploaded chest X-ray images (supporting PNG, JPG, BMP, and TIF formats up to 200MB) and uses a deep learning CNN model to detect pneumonia patterns. It generates Grad-CAM attention maps — visual heatmaps overlaid on the X-ray that show exactly which regions of the lung the model is focusing on when making its prediction. This allows doctors to visually verify whether the AI is looking at clinically relevant areas. The module also displays learning curves showing Focal Loss, Accuracy, and AUC metrics for both training and validation sets, giving clinicians confidence in the model's reliability.

Dual-Mode Operation — Online and Offline:

The system is designed to work in two modes to accommodate the reality of healthcare infrastructure in Bangladesh:

  • Offline Mode: The entire system runs locally on cheap, low-spec hardware using pre-trained models. All predictions, SHAP/LIME explanations, and Grad-CAM visualizations are generated without any internet connection. This is critical for rural clinics where internet connectivity is unreliable or nonexistent. The models are optimized to run efficiently on older computers that are commonly available in these settings.

  • Online Mode: When internet connectivity is available, the system connects to the Gemini API to provide enhanced capabilities. Doctors can chat with the AI about their predictions, ask follow-up questions, request second opinions, and get deeper explanations. Crucially, doctors can ask for explanations in Bengali, making the system accessible to healthcare workers who are more comfortable communicating in their native language. This multilingual support is a key differentiator, as most AI medical tools only operate in English.

Every prediction across all modules is accompanied by a clear disclaimer: "This prediction is for decision support only and must be reviewed by a qualified clinician." The system is designed as a decision-support prototype, not an autonomous diagnosis system — it augments the doctor's expertise rather than replacing it.

How we built it

The project was developed as part of a final year thesis on Explainable AI for Black-Box Models in Medical Diagnosis, and the architecture reflects both academic rigor and practical deployment considerations.

Machine Learning Models:

  • For the tabular prediction tasks (Diabetes and Heart Disease), we trained ensemble models including Random Forest and Gradient Boosting classifiers. These models were chosen for their strong performance on structured clinical data and their compatibility with post-hoc explanation methods.
  • For the Pneumonia Detection task, we built a Convolutional Neural Network (CNN) trained on chest X-ray images. The model uses Focal Loss as its optimization objective to handle class imbalance between normal and pneumonia cases, which is common in medical imaging datasets.

Explainable AI (XAI) Integration:

  • SHAP (SHapley Additive exPlanations): We integrated SHAP values to provide both global feature importance (which features matter most across all patients) and local explanations (why this specific prediction was made for this specific patient). SHAP values are computed offline and stored as static artifacts for the offline mode.
  • LIME (Local Interpretable Model-agnostic Explanations): LIME is used to generate instance-level explanations by perturbing input features and observing how predictions change. This gives doctors an intuitive understanding of model sensitivity to each clinical parameter.
  • Grad-CAM (Gradient-weighted Class Activation Mapping): For the pneumonia detection CNN, Grad-CAM generates heatmaps that highlight the regions of the X-ray image that most influenced the model's decision. These attention maps are overlaid on the original images so doctors can visually verify the model's focus areas.
  • Rule-Based Natural Language Summaries: We built a rule-based explanation engine that converts SHAP/LIME outputs into plain-language clinical summaries. For example, instead of showing raw SHAP values, the system might say: "This patient's elevated glucose level (120 mg/dL) and BMI (28.0) are the primary factors contributing to the elevated diabetes risk prediction."

Frontend and User Interface: The web interface was built with a clean, professional design featuring a teal sidebar navigation with modules for Home, Diabetes Prediction, Heart Disease Prediction, Pneumonia Detection, and About/Methodology. Each prediction module has its own dedicated page with:

  • Patient input forms with appropriate field types (numeric inputs with increment/decrement buttons for continuous variables, dropdown selectors for categorical variables)
  • A prominent yellow disclaimer banner stating the decision-support-only nature of the predictions
  • A "Predict" button that triggers the inference pipeline
  • Sections for displaying prediction results and explainability figures

Backend Architecture: The backend is built in Python and handles:

  • Model inference using pre-trained model files
  • SHAP/LIME computation and caching for offline use
  • Grad-CAM generation for image inputs
  • API integration with Gemini when online
  • Bengali language translation of explanations via Gemini API

Offline Optimization: To ensure the system runs on cheap local hardware, we:

  • Pre-computed and cached SHAP/LIME values so they don't need to be recalculated at inference time
  • Optimized model sizes through pruning and quantization
  • Used lightweight web frameworks that don't require heavy server infrastructure
  • Designed the UI to be responsive even on older browsers and slower machines

Gemini API Integration (Online Mode): When the system detects an internet connection, it activates the Gemini API integration. This enables:

  • Conversational AI where doctors can ask questions about predictions
  • Bengali language explanations generated on-the-fly
  • Context-aware follow-up discussions about patient cases
  • Additional clinical insights that go beyond the static model outputs

Challenges we ran into

1. Balancing Model Accuracy with Explainability One of the fundamental tensions in XAI is that the most accurate models (deep neural networks, complex ensembles) are often the hardest to explain, while the most interpretable models (decision trees, linear regression) may not achieve sufficient accuracy for clinical use. We had to carefully select and tune our models to find the sweet spot — achieving high predictive performance while still being able to generate meaningful, trustworthy explanations. This required extensive experimentation with different model architectures and XAI techniques.

2. Making XAI Work Offline SHAP and LIME computations are computationally expensive, especially for large datasets. Running these in real-time on low-spec hardware would have been impractical. Our solution was to pre-compute explanation artifacts during a setup phase and store them locally. However, this meant we had to design a system that could handle the storage and retrieval of these artifacts efficiently, and ensure that the pre-computed explanations remained relevant and accurate for the deployed models.

3. Grad-CAM Quality and Reliability Generating meaningful Grad-CAM attention maps for chest X-rays proved challenging. Early versions of our CNN produced heatmaps that were either too diffuse (highlighting the entire image) or too focused (highlighting irrelevant artifacts). We had to iterate on the CNN architecture, training procedure, and Grad-CAM parameters to produce attention maps that consistently highlighted clinically relevant lung regions. We also implemented static fallback figures for cases where Grad-CAM generation fails, ensuring the system always provides some form of visual explanation.

4. Bengali Language Medical Explanations Translating complex medical AI explanations into clear, accurate Bengali was surprisingly difficult. Medical terminology doesn't always have direct Bengali equivalents, and the nuances of clinical reasoning can be lost in translation. We had to carefully craft our Gemini API prompts to ensure that the Bengali explanations were both medically accurate and culturally appropriate for Bangladeshi healthcare workers. This required multiple rounds of testing and refinement with native Bengali speakers who had medical backgrounds.

5. Hardware Constraints and Performance Optimization Running machine learning models, especially CNNs for image processing, on cheap local hardware was a significant challenge. We had to optimize our models through techniques like quantization, pruning, and using efficient inference frameworks. We also had to carefully manage memory usage and processing time to ensure the system remained responsive even on older computers with limited RAM and processing power.

6. Designing for Clinical Trust Doctors are naturally skeptical of AI systems, especially "black-box" models that don't explain their reasoning. We had to design our explanations in a way that aligns with how doctors think clinically — not just showing feature importance scores, but framing explanations in terms of clinical relevance and risk factors that doctors already understand. This required close attention to the language and presentation of our explanations.

7. Handling Diverse Input Types The system needed to handle both tabular clinical data (with mixed continuous and categorical variables) and image data (chest X-rays). Building a unified interface and inference pipeline that could handle both types of inputs seamlessly, while providing appropriate explanations for each, required careful architectural planning.

8. Ensuring the System Doesn't Overstep Its Role A critical design challenge was ensuring the system presents itself as a decision-support tool, not an autonomous diagnostic system. We had to carefully design the UI, language, and disclaimer messaging to make it clear that the AI is augmenting the doctor's expertise, not replacing it. This is especially important in resource-constrained settings where there might be pressure to rely too heavily on automated systems.

Accomplishments that we're proud of

Successfully Integrated Three Different XAI Techniques Across Different Data Modalities We're particularly proud that we didn't just apply one XAI technique to one type of data. We successfully integrated SHAP and LIME for tabular clinical data (Diabetes and Heart Disease predictions) and Grad-CAM for medical imaging (Pneumonia detection). Each technique was chosen specifically because it's the most appropriate explanation method for that data type and prediction task. This multi-modal XAI approach is relatively rare in practical deployments and demonstrates a deep understanding of when and how to apply different explainability methods.

Built a Truly Functional Offline-First System Many AI systems claim to work offline but actually require periodic cloud connectivity for model updates, explanation generation, or API calls. Our system is genuinely offline-first — all predictions, explanations, and visualizations work completely without internet. The pre-computed SHAP/LIME artifacts and locally-running CNN model mean that a rural clinic with no internet access gets the full functionality of the system. This is a significant engineering achievement that required careful planning and optimization.

Multilingual AI Explanations in Bengali The ability to provide AI-generated medical explanations in Bengali is something we're very proud of. Most XAI research and tools are English-only, which creates a significant barrier to adoption in non-English-speaking regions. By integrating Gemini API for Bengali explanations, we've made our system accessible to a much wider range of healthcare workers in Bangladesh. This feature alone could significantly increase the practical impact of the system in rural areas where English proficiency among healthcare workers varies.

Designed for Real-World Deployment Constraints We didn't just build a research prototype — we built something designed to actually work in the conditions where it's needed. The system runs on cheap, low-spec hardware that's commonly available in Bangladeshi clinics. It works offline when internet is unavailable. It provides explanations in the local language. It's designed to help doctors see more patients per day without compromising diagnostic quality. These practical considerations are often overlooked in academic AI projects, and we're proud that we prioritized them.

Maintained Clinical Appropriateness Throughout Every aspect of the system is designed to support, not replace, clinical judgment. The prominent disclaimers, the decision-support framing, the rule-based explanations that align with clinical reasoning — all of these show that we thought carefully about how the system would be used in practice. We're proud that we didn't get caught up in making the AI seem more autonomous or impressive than it should be. The system knows its role and stays in it.

Completed as a Rigorous Final Year Thesis This isn't just a hackathon project or a weekend build — it's the result of a full final year thesis with proper research methodology, literature review, model evaluation, and XAI technique comparison. The academic rigor behind the project gives us confidence that the approaches we've taken are well-founded and the results are reliable.

Created a Complete, Polished User Interface The system has a professional, clean interface with proper navigation, input forms, disclaimer banners, prediction buttons, and explanation display sections. It doesn't look like a rough prototype — it looks like a product that could be deployed in a real clinic. We're proud of the attention to detail in the UI design, from the teal navigation sidebar to the appropriately styled input fields with increment/decrement controls.

What we learned

XAI Is Not One-Size-Fits-All Perhaps the most important lesson from this project is that different types of medical data and different prediction tasks require different explanation methods. SHAP and LIME work well for tabular clinical data because they can show how each numerical or categorical feature contributes to a prediction. But for image data like chest X-rays, these methods don't provide intuitive explanations — Grad-CAM's visual heatmaps are far more useful for showing where the model is "looking" in an image. Understanding when to use which XAI technique, and why, was a crucial learning.

Offline Capability Requires Fundamental Architectural Decisions Building for offline operation isn't just about removing API calls — it requires rethinking the entire system architecture. Pre-computing explanations, caching model outputs, optimizing for local hardware, and designing a UI that works without dynamic server responses all require different approaches than a cloud-native system. We learned that offline-first design should be considered from the very beginning of a project, not added as an afterthought.

Clinical Context Is Essential for Meaningful Explanations Raw SHAP values or LIME coefficients mean very little to a doctor. What matters is translating these technical outputs into clinical insights — "this patient's glucose level is the primary risk factor" is far more useful than "feature 2 has a SHAP value of 0.34." We learned that the best XAI systems don't just show model internals; they translate those internals into the language and framework that the end user already understands.

Resource Constraints Drive Innovation Working within the constraints of cheap hardware, limited internet, and non-English-speaking users forced us to be creative in ways we wouldn't have been if we were building for a well-resourced environment. The offline pre-computation strategy, the Bengali language integration, the hardware optimization — these solutions emerged directly from the constraints we faced. We learned that constraints aren't just limitations; they're design drivers that can lead to better, more practical solutions.

Trust Is Built Through Transparency, Not Accuracy Alone A highly accurate model that can't explain itself will be less useful in clinical practice than a slightly less accurate model that provides clear, trustworthy explanations. We learned that in medical AI, explainability isn't a nice-to-have feature — it's a fundamental requirement for adoption. Doctors need to understand why a model makes a prediction before they'll trust it enough to use it in their practice.

The Gap Between Research and Deployment Is Real Academic XAI research often focuses on novel explanation methods and benchmark performance, but deploying XAI in a real clinical setting involves many practical challenges that papers don't address: hardware limitations, user interface design, language barriers, offline operation, clinical workflow integration, and trust-building. We learned that bridging this gap requires skills beyond just machine learning — it requires systems thinking, user experience design, and an understanding of the deployment environment.

Multilingual AI Is Harder Than It Seems Generating accurate medical explanations in Bengali taught us that multilingual AI isn't just about translation — it's about cultural adaptation, medical terminology mapping, and ensuring that the tone and framing of explanations are appropriate for the target audience. A direct translation of an English medical explanation might be technically correct but clinically confusing or culturally inappropriate in Bengali.

Decision Support ≠ Autonomous Diagnosis We learned the critical importance of designing AI systems that clearly position themselves as tools to augment human expertise, not replace it. The way we frame the system's capabilities, the disclaimers we include, the language we use — all of these shape how doctors perceive and use the system. Getting this framing right is essential for ethical AI deployment in healthcare.

What's next for clinic_assist_with_online_and_offline_support

Expand to Additional Medical Conditions The current system covers Diabetes, Heart Disease, and Pneumonia — three significant health challenges in Bangladesh. The next step is to expand the platform to cover additional conditions that are prevalent in resource-constrained settings, such as tuberculosis (using chest X-ray analysis), diabetic retinopathy (using retinal fundus images), malaria (using blood smear images), and maternal health risk assessment. Each new module would follow the same XAI-first approach with appropriate explanation methods for the data type.

Develop a Mobile Application Many community health workers in rural Bangladesh conduct home visits and don't have access to a computer. Developing a lightweight mobile application (Android-first, given the market share in Bangladesh) would extend the system's reach to these frontline healthcare workers. The mobile app would need to be optimized for low-end smartphones, work offline, and provide simplified explanations suitable for health workers who may not have full medical training.

Implement Federated Learning for Privacy-Preserving Model Improvement Currently, the models are trained on static datasets and deployed as-is. A federated learning approach would allow multiple clinics to collaboratively improve the models by sharing model updates (not patient data) while keeping sensitive patient information local. This would enable the system to continuously improve its accuracy across different populations and regions without compromising patient privacy — a critical consideration in healthcare AI.

Add Voice Interface in Bengali Many doctors and health workers in rural Bangladesh are more comfortable speaking than typing, especially when dealing with complex medical terminology. Adding a Bengali voice interface would allow doctors to speak patient parameters into the system and receive spoken explanations of predictions. This would significantly improve usability, especially for older healthcare workers who may not be comfortable with keyboard input.

Integrate with Existing Health Record Systems Currently, doctors must manually enter patient data into the system. Integrating with electronic health record (EHR) systems used in Bangladeshi hospitals and clinics would streamline the workflow, reduce data entry errors, and allow the AI to access a more complete patient history for better predictions. This integration would need to support both digital records and semi-digital systems (like spreadsheet-based records) that are common in resource-constrained settings.

Conduct Real-World Clinical Trials The most important next step is to validate the system's impact in real clinical settings. We plan to partner with rural clinics and community health centers in Bangladesh to conduct controlled studies measuring: (1) whether the system improves diagnostic accuracy, (2) whether it increases the number of patients a doctor can see per day, (3) whether doctors find the explanations trustworthy and useful, and (4) whether patients benefit from faster, more accurate diagnoses. These trials would provide the evidence needed for broader adoption and potential government or NGO support.

Open Source the Platform To maximize impact, we plan to open-source the codebase, models, and documentation so that other researchers, developers, and healthcare organizations can adapt the system for their own contexts. The XAI techniques, offline architecture, and multilingual support patterns we've developed could be valuable for similar projects in other resource-constrained regions around the world.

Add Real-Time XAI Computation for Online Mode Currently, offline mode uses pre-computed SHAP/LIME artifacts. For the online mode, we plan to implement real-time XAI computation so that doctors can get explanations for any input combination, not just the pre-computed scenarios. This would make the online mode significantly more powerful and flexible, allowing for more nuanced explorations of patient cases.

Build a Doctor Feedback Loop We want to add a feature where doctors can provide feedback on predictions — whether they agreed with the AI's assessment, whether the explanation was helpful, and what additional information would have been useful. This feedback would be used to improve both the models and the explanation generation, creating a continuous improvement cycle driven by actual clinical use.

Explore Integration with Telemedicine Platforms In areas where specialist doctors are scarce, telemedicine is becoming increasingly important. Integrating our XAI system with telemedicine platforms would allow remote specialists to see not just the patient data and AI prediction, but also the explanation of how the prediction was made. This would make remote consultations more efficient and trustworthy, as the specialist can quickly understand the local AI's reasoning before providing their own assessment.

Built With

Share this project:

Updates

Submission history