Dermasisst

Inspiration

Skin lesions can be difficult to evaluate from an image, and it is not always obvious when a spot deserves professional attention. We wanted to build a prototype that could combine computer vision with a simple, understandable triage system rather than simply outputting a disease label.

What it does

DermaAssist analyzes an uploaded skin-lesion image using a ResNet18 image classifier trained on the HAM10000 dataset. It produces probabilities for seven common lesion categories and uses those probabilities together with user-reported changes such as growth, bleeding, itching, and pain to determine an urgency level.

The app has four possible outcomes: Cannot Assess, Prompt Review, Routine Review, and Nothing Flagged.

Gemini then checks whether the uploaded image appears to be a usable skin-lesion image and generates a plain-language explanation of the result and general information about the predicted condition. Gemini does not control the predicted condition or urgency level.

How we built it

We kept the existing ResNet18 model and built several components around it. A shared predictor handles image preprocessing and model inference, while a separate rule-based triage layer calculates urgency from the model probabilities and user answers.

We use hand-selected thresholds for the triage rules so that the system is intentionally biased toward recommending review rather than giving an all-clear. Gemini provides the natural-language explanation, while Gradio provides the web interface for uploading images, answering questions, and viewing the results.

We also added automated tests for the urgency rules and a fallback mode so the application can still provide the rule-based result if the Gemini API is unavailable.

Challenges we ran into

One of the biggest challenges was recognizing the limitations of the existing model. The reported 77% validation accuracy is optimistic because many validation images show lesions that also appear in the training set, and the model has no built-in way to recognize an image that is outside its intended scope.

We therefore avoided treating the classifier's prediction as a medical diagnosis. Instead, we separated prediction, urgency, and explanation into different components and made the urgency rules deterministic.

Another challenge was making the generative AI component safe and predictable. Gemini is given the model's predicted condition and rule-based urgency as fixed inputs and is instructed not to change them or provide personalized treatment advice.

Accomplishments that we're proud of

We are proud of building a complete end-to-end prototype while keeping the decision-making logic transparent. The system combines computer vision, deterministic rules, generative AI, and a web interface without allowing the language model to override the core prediction or urgency logic.

We also designed explicit fallback behavior, automated tests for the highest-risk rules, and a permanent prototype notice so that the application does not present its results as a medical diagnosis or an all-clear.

What we learned

We learned that building an AI application is not just about getting a model to produce a prediction. Understanding the limitations of the dataset, evaluation methodology, and model's intended input is just as important.

We also learned how deterministic rules and generative AI can complement each other: rules provide predictable decisions, while an LLM can make those decisions easier for a user to understand.

What's next for DermaAssist

Future work could include evaluating the model on a truly held-out test set, tuning the urgency thresholds using appropriate validation data, and training on more realistic smartphone photographs such as those in PAD-UFES-20.

We could also explore better model interpretability, improved out-of-distribution detection, and a more scalable backend. These improvements would be necessary before considering anything beyond a research and demonstration prototype.