Skin cancer detection

Inspiration

Skin cancer diagnosis can be challenging because different skin lesions can look visually similar, while AI predictions can often feel like a “black box.” We wanted to build a system that not only predicts a skin disease but also explains why the prediction was made and provides clinically relevant context.

What it does

Our Skin Disease Diagnosis Assistant analyzes an uploaded skin lesion image and:

Classifies it across 7 skin lesion categories

Provides a confidence score and risk assessment

Retrieves visually similar cases for comparison

Generates a clinical explanation of the prediction

Presents results through an interactive web dashboard

Generates a downloadable PDF diagnostic report

How we built it

We combined Computer Vision, Retrieval-Augmented Generation (RAG), and LLM-based reasoning. A Vision Transformer (ViT) trained on the HAM10000 dataset performs lesion classification, while a vector database retrieves similar cases. The backend was built with FastAPI, and the frontend uses React, Vite, Tailwind CSS, and shadcn/ui. We also integrated confidence visualization, risk indicators, and automated report generation.

Challenges we ran into

One of our biggest challenges was connecting multiple AI components into a reliable end-to-end pipeline. We also faced issues with model integration, vector database configuration, frontend-backend communication, UI component compatibility, and generating a clean clinical-style PDF. Making the AI output understandable rather than simply displaying a prediction was another major challenge.

Accomplishments that we're proud of

We achieved 79.13% classification accuracy across seven skin lesion classes and a 91.4% similarity-search hit rate. More importantly, we built a complete working prototype that connects image analysis, retrieval, explainability, and visualization into a single application rather than treating each component independently.

What we learned

We learned that building an effective healthcare AI system requires more than model accuracy. Explainability, retrieval quality, user experience, confidence calibration, and responsible presentation of predictions are equally important. We also gained hands-on experience integrating computer vision models, RAG pipelines, vector databases, APIs, and modern frontend technologies.

What's next for Skin Cancer Detection

Next, we want to improve model performance and generalization using larger and more diverse datasets, add Grad-CAM visualizations to highlight the image regions influencing predictions, improve confidence calibration, and expand the retrieval system. We also plan to strengthen clinical validation and safety considerations so the system can evolve from a research prototype into a more robust clinical decision-support tool.

Built With

• fastapi
• javascript
• python
• pytorch
• qdrant
• rag
• react
• shadcn/ui
• tailwind-css
• vision-transformer-(vit)
• vite