Inspiration
We thought medical problems are typically underserved in hackathons, and with Jacob from a bioengineering background, and Lei from a ML research background we thought we might be able to design an interesting tool. During the introduction speech from the Mistral team on Saturday morning, we learned about Pixtral, so after a bit of research we saw most medical models don't have multimodal context, we decided to go with that for on the fly diagnosis (integrating clinical data, and imaging data). As for targeting dermatologists, this was because the skin lesion dataset we found was the only one we could access without a paywall or contacting the data curators.
What it does
The Clarivex-Pixtral-12B model is a fine-tuned version of the Pixtral-12B model, created during the Hack UK. It has been adapted specifically for the Diverse Dermatology Images (DDI) Multimodal Dataset. We created an API to connect to a front-facing interface chatbot which is designed as a copilot tool for dermatologists.
How we built it
For finetuning the Pixtral-12B model, we used Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA). The fine-tuning was completed on a single Nvidia H100 80GB GPU optimizing the model to handle both visual and clinical data for accurate skin disease diagnosis across diverse populations. Testing for zero shot predictions was done using the Mistral API. For the frontend, we designed it in Figma, then implemented using React, and the API for the fine-tuned model was designed using Flask.
Challenges we ran into
It took us a while to find an adequate multimodal medical dataset. Most datasets with good clinical annotations are either locked behind a paywall, unusable for commercial use cases, or require direct contact with hospitals or data curators - which wasn't exactly feasible for us during the time of the hackathon.
It took a lot of effort to fine tune Pixtral, a type of model we were unfamiliar with, and not many resources that were available for us to learn from.
Accomplishments that we're proud of
We're proud that we actually managed to finetune Pixtral. We're quite happy that we managed to build a working pipeline from back to front that we would be able to iterate and build upon.
What we learned
For two of us, it was our first hackathon. We learned how to work in a team with a constrained time budget. We learned how to finetune Pixtral, as well as how to use the Mistral APIs, and work with multimodal datasets.
What's next for Clarivex.ai
We'd like to improve performance of the finetuned model, so this would require assembling a large and comprehensive multimodal skin lesion imaging dataset paired with detailed clinical information and dermatologist reports. We speculate that pairing this data with other strategies common in biomedical image analysis such as segmentation masks, or synthetic data generation to impute for missing data could further improve the model performance, so these are a few strategies we would like to try. Aside from that, it would be great to get some real world feedback from dermatologists to see how useful they find a tool ours. Also, on a side note we could even implement a federated learning scheme to share weights between different dermatologists who are uploading images and creating new clinical reports, while simultaneously protecting privacy, which is very important to consider when handling patient data which is not in an open dataset.
Log in or sign up for Devpost to join the conversation.