Inspiration
Here is a complete, compelling draft for your Devpost Project details section based on the README you shared. You can copy and paste this directly into the editor!Inspiration In modern healthcare, patients are often handed complex lab reports or radiological scans filled with dense medical jargon, leaving them anxious and confused. While general-purpose LLMs can help explain these results, uploading highly sensitive health data (PHI) to cloud-based AI providers raises massive privacy and compliance concerns.We realized the world needed a bridge between complex medical diagnostics and patient understanding that doesn't compromise data security. We were inspired to build MedLens AI a platform where privacy meets explainability, ensuring that "black box" AI is replaced by transparent, mathematically grounded insights kept completely on the client side.
What it does
MedLens AI is a privacy-first medical analytics platform that instantly parses complex medical lab report PDFs and radiological images (X-Ray, CT, MRI). Once a user uploads their file, the platform provides:Visual Explainability: Overlays PyTorch Grad-CAM heatmaps onto radiological scans, highlighting the exact focal opacity regions the AI is analyzing.Mathematical Risk Scoring: Uses an XGBoost model paired with SHAP attributions to quantify exactly how individual lab metrics (like Glucose or eGFR) contribute to cardiovascular or metabolic risk scores.Evidence-Based Context: Utilizes a FAISS Vector RAG pipeline to retrieve relevant clinical protocols (ADA, ACC/AHA, WHO) matched directly to the patient’s diagnostics.Medical AI Copilot: Features a context-aware chat assistant powered by OpenRouter (DeepSeek-R1 / Llama-3.3-70B) to break down findings into patient-friendly language.Zero Data Leakage: All session records are stored 100% locally in the browser via IndexedDB.
How we built it
We architected MedLens AI to be highly modular, splitting the heavy ML workloads from the snappy user interface:Frontend: Built with Next.js 15 (App Router), React 19, and Tailwind CSS. We used Recharts for data visualization and Dexie.js to manage the local IndexedDB, ensuring a strict client-side history without requiring a server database.Backend: Powered by FastAPI (Python 3.11+). It handles the complex document ingestion using PyMuPDF and OCR pipelines. Machine Learning / CV: We fine-tuned DenseNet121 and EfficientNet-B0 models using PyTorch for our image analysis. We implemented OpenCV to handle the Grad-CAM heatmap overlays and used XGBoost & SHAP to calculate and visualize feature-level risk attributions.RAG / NLP: We embedded over 500 clinical guideline chunks using sentence-transformers and indexed them via FAISS-CPU. The chat copilot integrates with the OpenRouter API to provide conversational context with actual RAG citations.
Challenges we ran into
Medical Data Extraction: Parsing unstructured PDFs and accurately mapping messy OCR outputs into deterministic JSON structures for our XGBoost model was incredibly difficult. We had to implement robust regex pattern matching to normalize varying lab telemetry formats.Explainable AI in Production: Generating Grad-CAM heatmaps dynamically required careful memory management in PyTorch to ensure the FastAPI backend didn't bottleneck during concurrent scan uploads.Strict Privacy Constraints: Designing an architecture that functions as a powerful AI tool while adhering to a strict "local-first" storage philosophy required bypassing traditional relational databases in favor of browser-native storage APIs.
Accomplishments that we're proud of
True Explainability: In healthcare, AI cannot be a black box. We are incredibly proud of successfully merging visual (Grad-CAM) and mathematical (SHAP) explainability into one unified dashboard.The Zero-Leakage Architecture: Successfully building a stateful, history-rich web application that relies entirely on IndexedDB, meaning users can chat with an advanced LLM about their health data without ever permanently storing it on our servers.Seamless Multi-modal Pipeline: Taking a raw image or PDF, extracting text, passing it through computer vision models, running tree-based risk assessments, and finalizing it with vector search—all in a matter of seconds.
What we learned
We gained a profound understanding of how to implement SHAP (SHapley Additive exPlanations) to demystify machine learning decisions. We also learned how to effectively combine traditional computer vision techniques with modern RAG architectures to ground generative AI in actual clinical facts, effectively mitigating LLM hallucinations in a high-stakes domain.
What's next for MedLens AI
We have a clear roadmap to make MedLens AI even more powerful and accessible:Native DICOM Support: Implementing pydicom to support native hospital imaging formats beyond standard image files.EHR Integration: Mapping our data outputs to FHIR R4 resources for seamless integration with existing Electronic Health Record systems.Edge Deployment: Exporting our models to ONNX and utilizing WebGPU to run the Grad-CAM inference entirely in the browser.Offline-First: Upgrading the frontend into an offline-first Progressive Web App (PWA) with service workers for ultimate accessibility in low-connectivity clinics.
Log in or sign up for Devpost to join the conversation.