DermaSense

DermaSense delivers instant, explainable AI skin analysis for both phone and dermoscopy images—top-2 accuracy 96.9%, melanoma specificity 97.3%. Privacy-first, expert tools for everyone.

Comment 1

Inspiration

Skin cancer and other dermatological conditions affect millions worldwide, but access to expert diagnosis is often limited—especially for those without health insurance or living in remote areas. We wanted to break the barrier between world-class dermatological intelligence and everyday accessibility. By combining state-of-the-art AI, real clinical explainability, and privacy-first design, DermaSense was born to empower everyone—from patients to clinicians—with instant, trustworthy skin health insights.

What it does

DermaSense is a dual-mode AI platform that delivers:

  • Instant, highly accurate skin condition analysis from either dermatoscopic (doctor-grade) or smartphone (consumer-grade) images.
  • Actionable explainability using dynamic Grad-CAM heatmaps and natural language, so users and clinicians know why the AI made its prediction.
  • Voice AI feedback—personalized and accessible, with the tone matched to diagnostic risk level.
  • Longitudinal lesion tracking—AI-powered comparisons for monitoring changes over time.
  • A secure clinical dashboard for professionals to review, triage, and annotate cases.
  • All patient-facing AI runs fully client-side, preserving user privacy—no image ever leaves the user's device unless they opt in for professional review.

How we built it

  • AI Models:

    • Two distinct models—EfficientNetB3 for clinical dermatoscopy and EfficientNetB4 for consumer phone images—trained from scratch on >150,000 expertly curated cases.
    • Advanced ML methods: focal loss, class weighting, aggressive data augmentation, Grad-CAM for XAI, and rigorous evaluation.

  • Backend:

    • Built with Python + FastAPI for async API endpoints.
    • Supabase for database and secure case image storage.
    • Google Gemini 1.5 Flash API for generative language explanations.
    • ElevenLabs for AI-driven, emotionally intelligent voice output.

  • Frontend:

    • Built with React + TypeScript (Bolt.new), styled via Tailwind CSS and animated with Framer Motion.
    • All AI inference runs in-browser for privacy.
    • Professional deployment on Netlify with a custom domain from IONOS.

Challenges we ran into

  • Class imbalance and dataset noise: Real-world skin datasets are imbalanced and messy. We spent days engineering loss functions and augmentation pipelines to fight this.
  • On-device inference: Running large AI models and Grad-CAM entirely client-side (for privacy) was a technical feat, requiring model size optimization and careful engineering.
  • Explainability: Translating AI reasoning into visual and natural language explanations that both laypeople and doctors can trust was a deep UX and ML challenge.
  • Seamless integration: Combining cloud APIs (Supabase, Gemini, ElevenLabs), on-device ML, and a real clinical dashboard without sacrificing speed, privacy, or reliability.
  • Deployment and compatibility: Ensuring everything works end-to-end on serverless infrastructure, across browsers, and with large ML assets, required constant iteration.

Accomplishments that we're proud of

  • Achieved expert-level accuracy: Our clinical model outperforms the published state-of-the-art on the ISIC dataset (85%+ top-1, 96%+ top-2, 0.95 AUC for melanoma).
  • Built a consumer AI that truly works on phone images: Robust, reliable, and outperforming every public benchmark we found.
  • End-to-end explainability: Visual Grad-CAM overlays, context-aware voice, and generative language explanations all work live.
  • Privacy-first by default: No image ever leaves a user’s device unless explicitly allowed.
  • Built, integrated, and deployed a full-stack, real-world-ready product in just weeks.
  • Live on a custom domain with a secure clinical dashboard—ready for real-world pilots.

What we learned

  • Bridging technical excellence and user trust is essential: AI isn’t enough without explainability, privacy, and emotional intelligence.
  • Real-world ML is more than just models: Data engineering, deployment, and UX can make or break impact.
  • Hackathons are about focus and iteration: Scoping ruthlessly and shipping vertical slices early was crucial.
  • Collaboration is key: Dividing responsibilities—AI, backend, frontend, integration, and UX—let us build something far more powerful together.

What's next for DermaSense

  • Full-scale longitudinal lesion tracking: AI-powered monitoring for early detection and personalized health.
  • Expanding diagnostic coverage: Growing from 7 to 100+ skin conditions in both clinical and consumer models.
  • Academic publication: Targeting top-tier AI/medical conferences (e.g., NeurIPS, MICCAI) to contribute our breakthroughs.
  • Production telehealth integration: Secure doctor-patient messaging, regulatory compliance, and real clinical deployment.
  • Global accessibility: Translating all interfaces and explanations, partnering with health orgs for broad, real-world impact.

DermaSense: An AI-Powered Dermatological Ecosystem – Redefining Precision & Accessibility for Global Health

DermaSense is a private, truly state-of-the-art, dual-model AI platform meticulously engineered to deliver instant, clinically-relevant dermatological insights for both medical professionals and everyday consumers. It represents a revolutionary leap in bridging the critical gap between patient concern and expert diagnosis. Built with unparalleled precision for the Bolt.new World's Largest Hackathon (May–June 2025), DermaSense is not merely a prototype; it's a foundational step toward democratizing advanced skin health, poised for impactful research and real-world deployment.

Pioneering Key Features: Unprecedented Functionality & User Trust

DermaSense is more than an application; it's a comprehensive, intelligent ecosystem designed for uncompromising impact, deep technical sophistication, and profound user trust.

  • Groundbreaking Dual AI Model System: At its core are two distinct, world-class AI models, each custom-built and rigorously optimized for specific, high-stakes use cases, demonstrating a depth of AI engineering rarely seen in hackathon projects:

    • A high-precision Clinical Model for deep diagnostic assistance with dermatoscopic images, rivaling expert-level performance.
    • A transformative Consumer Model for robust, broad-coverage screening of standard phone camera photos, democratizing advanced AI access for the public.

  • Transparent Explainable AI (XAI) with Dynamic Grad-CAM: We don't just provide answers; we provide actionable understanding. DermaSense employs dynamic Grad-CAM to generate vivid visual heatmaps that pinpoint the exact features of the lesion the AI focused on to make its decision. This sophisticated explainability builds unparalleled trust, offers crucial clinical insight, and demystifies the AI's reasoning.

  • Intelligent & Empathetic Explanations via Google Gemini 1.5 Flash: Beyond raw predictions, DermaSense leverages the cutting-edge Google Gemini 1.5 Flash API to transform complex clinical terms and statistical probabilities into clear, empathetic, and contextually responsible natural language explanations. By intelligently synthesizing the top 3 model predictions for richer context, Gemini provides nuanced interpretations that empower users without medical jargon.

  • Immersive Multi-Modal Experience with Context-Aware Voice: To deliver results in the most human and accessible way possible, DermaSense integrates advanced generative AI for rich, intuitive feedback:

    • ElevenLabs Voice AI provides natural, high-fidelity audio readouts of the AI explanations. Crucially, the system dynamically selects the ElevenLabs voice ID based on the diagnostic risk level (high, medium, or low), ensuring an emotionally appropriate and accessible auditory experience.

  • AI-Powered Lesion Tracking & Longitudinal Comparison: A state-of-the-art innovation, DermaSense is designed to enable longitudinal monitoring of skin conditions, lesions, or melanoma over time. Through intelligent, AI-powered comparison algorithms, users can meticulously track subtle changes in size, shape, and color—a capability that is paramount for early detection and proactive health management.

  • Robust Clinical Triage Dashboard with Secure Data Management: For healthcare professionals, we've engineered a secure, password-protected dashboard. This powerful tool enables dermatologists to efficiently review, triage, and annotate patient-submitted cases, viewing original lesions, Grad-CAM heatmaps, and the AI's initial diagnosis. This streamlines professional workflow, prioritizes critical care, and leverages Supabase for scalable, opt-in data logging, contributing to real-world public health insights.

  • Uncompromising Privacy-First Design: A cornerstone of our development and a testament to responsible AI, the core patient-facing analysis tool performs all ML model inference and Grad-CAM generation securely within the user's browser. This means no user images are ever uploaded or stored on our servers for analysis. The "Request a Professional Review" feature is strictly opt-in and requires explicit user consent for every submission, with encrypted data handled via Supabase, ensuring absolute data privacy and user control.

Our State-of-the-Art AI Models: Unprecedented Precision & Technical Depth

We didn't just build one model; we engineered two distinct, high-performing AI engines. Each model's architecture was chosen and refined through rigorous experimentation and meticulous optimization of training methodologies, showcasing a deep understanding of practical machine learning for real-world medical applications.

1. The Clinical Model (EfficientNet-B3): Surgical Precision for Medical Professionals

This model is a high-precision tool, meticulously designed for use by medical professionals with specialized dermatoscopic images. It aims to augment a dermatologist's diagnostic capabilities with unparalleled accuracy and reliability.

  • Architecture: EfficientNet-B3, a highly efficient and accurate convolutional neural network, selected as the champion after a comprehensive head-to-head experiment against EfficientNetB2, demonstrating superior performance in a head-to-head clinical evaluation.
  • Dataset: Trained on the challenging and comprehensive international ISIC 2019/2018 dermatoscopy datasets, renowned benchmarks in dermatological AI, reflecting real-world clinical diversity and complexity.

  • Advanced Training Methodology:

    • Focal Loss Implementation: We leveraged Focal Loss, a specialized loss function, to dynamically down-weight easy examples and significantly increase the focus on hard, misclassified cases (like rare melanomas) during training.
    • Strategic Class Weighting: We applied precise class weighting to the loss function, inversely proportional to class frequencies, preventing model bias towards common benign conditions and ensuring robust learning from critical, less frequent samples.
    • Two-Phase Fine-Tuning with Optimized Learning Rates: Initially, a custom classification head was specifically fine-tuned with re-weighting, followed by a global fine-tuning of the entire EfficientNet-B3 architecture with carefully annealed learning rates.

Performance (Comprehensive Test Set Metrics - 7 High-Risk Dermatological Conditions):

Metric Score Note
Overall Top-1 Accuracy 0.8565 (85.65%) Exceptional accuracy for directly identifying the correct condition within a 7-class, complex clinical problem.
Overall Top-2 Accuracy 0.9690 (96.90%) Unprecedented reliability; the correct diagnosis is almost always within the model's top two predictions.
Balanced Multi-Class Accuracy 0.8148 (81.48%) Significantly surpasses the winning ISIC 2019 benchmark of 64% by a landslide (≈17 percentage points improvement).
Melanoma vs. Others AUC 0.9492 World-class diagnostic capability—competes with, or exceeds, published PhD-level research results.
Melanoma Specificity 0.9733 Extremely low false positive rate—the model avoids unnecessary patient anxiety and follow-up.

  • Per-Class F1-Scores (Top-1):

    • actinic_keratosis: 0.7432
    • basal_cell_carcinoma: 0.9168
    • benign_mole: 0.9048
    • dermatofibroma: 0.8649
    • melanoma: 0.7244
    • seborrheic_keratosis: 0.8016
    • vascular_lesion: 0.9545

2. The Consumer Model (EfficientNet-B4): Democratizing Skin Health Intelligence

This model is a groundbreaking proof-of-concept, specifically designed to analyze real-world phone camera photos, making intelligent AI screening accessible to everyone.

  • Architecture: EfficientNet-B4, robust and efficient, fine-tuned for extracting meaningful features from consumer-captured images.
  • Dataset: Trained on a massive, custom-curated dataset of over 15,000 diverse consumer-grade photos from authoritative sources, including the MIDAS dataset.

  • Advanced Training Methodology:

    • Aggressive Data Augmentation Pipeline: Includes rotations, shifts, zooms, brightness variations, noise, and color jitter, greatly improving robustness and real-world generalization.
    • Focal Loss & Class Weighting: Essential for managing class imbalance and ensuring accurate detection across all specified classes.

Performance (Comprehensive Test Set Metrics - 7 Common Skin Conditions):

Metric Score Note
Overall Top-1 Accuracy 0.8582 (85.82%) Outstanding performance on noisy, real-world data.
Overall Top-2 Accuracy 0.9625 (96.25%) Exemplary reliability for public screening.
Balanced Multi-Class Accuracy 0.7970 (79.70%) Proves the model is robust and unbiased by class imbalance.
Melanoma vs. Others AUC 0.9572 World-class performance for a consumer model in distinguishing melanoma.
Melanoma Specificity 0.9833 Exceptional false positive reduction—users can trust the screening tool.

  • Per-Class F1-Scores (Top-1):

    • Acne: 0.9543
    • Benign_Mole: 0.7792
    • Eczema: 0.7742
    • Healthy_skin: 0.9468
    • Melanoma_Consumer: 0.3929
    • Psoriasis: 0.9134
    • Ringworm: 0.8485

Advanced Technology Stack: Engineering Excellence for Scalability & Impact

  • Frontend: Built with Bolt.new (React + TypeScript), styled using Tailwind CSS, animated with Framer Motion, and iconography by Lucide React.
  • Backend: Python + FastAPI for high-performance asynchronous API development. Netlify Serverless Functions for scalable AI endpoint hosting.
  • AI/ML Core: TensorFlow / Keras for model training and inference. Focal Loss and Grad-CAM for robust class balancing and explainability. Google Gemini 1.5 Flash for advanced language explanations. ElevenLabs for voice synthesis.
  • Database & Storage: Supabase (PostgreSQL, Supabase Storage) for secure clinical dashboard and user tracking.
  • Deployment: Netlify (full-stack deployment), custom domain via IONOS for professional presence.

Hackathon Challenge Submissions: Demonstrating Multi-Faceted Excellence

We are proud to compete for the following key challenge prizes, demonstrating the depth and breadth of DermaSense:

  • Startup Challenge (Supabase): Built a scalable backend and clinical dashboard using Supabase, showing a clear path to a commercial-grade telehealth product.
  • ** Voice AI Challenge (ElevenLabs):** Integrated ElevenLabs Voice AI for natural, risk-sensitive audio explanations—making our tool accessible to everyone.
  • Deploy Challenge (Netlify): Robust, professional deployment of both frontend and backend on Netlify, live and ready for global users.
  • ** Custom Domain Challenge (IONOS):** Production-grade hosting and custom domain for a seamless, real-world product experience.

The Vision: Beyond the Horizon – Shaping the Future of Digital Dermatology

While DermaSense delivers state-of-the-art functionality today, our roadmap includes:

  • Full AI-Powered Lesion Tracking & Comparison: Longitudinal tracking and AI-powered comparison of lesions, empowering users and clinicians with personalized, proactive care.
  • Path to Publication & Model Expansion: Preparing both models for academic publication (target: NeurIPS) and expanding coverage to 100+ skin conditions—transforming DermaSense into the world’s most comprehensive dermatological AI platform.
  • Production Telehealth Integration: Evolution to a fully compliant, production-grade telehealth platform with direct doctor-patient communication and global regulatory compliance (e.g., HIPAA).

Disclaimer: DermaSense is a hackathon project and is intended for informational and demonstrative purposes only. It is not a substitute for professional medical advice, diagnosis, or treatment. Always consult with a qualified healthcare professional for any skin concerns.

Built With

Share this project:

Updates