RareSynth

Team Members:

Neil Mehra Georgiy Derkachev Lakshya Dhakar Shiven SIngla

Inspiration

We were inspired by the many stories out of there of people being unable to get a proper diagnosis. In order to solve this we believe there must be better documented cases for certain diseases. And thus our idea was born.

What it does

We trained a custom Diffusion model to generate images of medical conditions. We chose a MRI dataset just because it was the first one we saw. Otherwise we could in theory have used any other dataset. Which makes the model very adaptable. These generated images will be uploaded to the Streamlit web application at the press of a button. The user can practice determining what kind of tumor is shown in the generated image and their selection is confirmed using a classifier model that will also attempt to classify the generated image.

How we built it

We used pytorch and built the model from scratch to have greater control. It uses the base convolutional operations and thus we have greater control over the model and the operations it does. It also has a linear noise scheduler even though we were planning to use a cosine one instead. The classifier model is trained using a dataset from Hugging Face which contains around five thousand MRI samples. The ViT model will attempt to classify and train itself using this dataset. The web application is built using a Python library called Streamlit. The app is the bridge connecting the image generation model to the classifier model.

Challenges we ran into

There was a big challenge we had. We at first decided to use 32x32 images to save computational power. This ended up in really pixelated and noisy images. This might have been due to the low batch size which restricted the models loss. We increased the image size and batch size and the model performed much better. Another challenge we ran into was creating and uploading the data to the ViT model. The data from the dataset has to be processed using transformers, but getting this to work was a problem as we ran into many authentication errors while trying to upload the dataset. Another problem we ran into was generating the image onto the Streamlit web app and attempting to classify it.

Accomplishments that we're proud of

This was a very hard challenge. But due to our experience with tensor flow we managed to succeeded in PyTorch as well. We also did not have much experience with making image classifying models, but we proud to have successfully created a working model that does what we want it to.

What we learned

We learned PyTorch first off. Since we usually didn't use it, it was a big struggle to actually figure it out. But once we did it was relatively straight forward. We also learned how to use ViT models and how to create and upload datasets to it. Connecting the generator model and the classifier model was a challenge but we accomplished that through research.

What's next for RareSynth

We want and hope to expand the model to 3D in hopes that it can generate more complex structures. But this is completely out of scope for us for now due to the lack of time. We hope it can generate blood veins or even more complicated parts like organs. We also hope to have this detection and practice system availble for many more types of health problems to hopefully raise confidence in aspiring doctors and also so that the general public can be more familiar with how such issues come about and how they look. Along with this, we want to have an information segment that explains the issue generated in the image and gives a brief, but accurate, summary of the symptoms and the next steps a patient should do if they were to unfortunately have the disease.