Inspiration
The McMed Hacks workshop series included a diverse range of methods and strategies in medical image analysis. The members of our group were particularly interested in exploring image segmentation through a project. When we heard about the Alpha Tau Grand Challenge, it seemed to tick all the boxes for what we wanted to work on. It was a complex and stimulating problem with readily available data in a subject we were interested in.
What it does
Our model takes CT images and performs instance segmentation, creating masks for these seeds. Then, it identifies the location of these seeds from these masks by finding vectors for the centre position and orientation. Our goal was to create a coarse-to-fine model for this challenge where the first phase of the model focuses on generating the segmentation masks for the seeds using Mask-RCNN and the second phase predicts the center and orientation from the set of predicted 2D segmentation masks using a 3DCNN model.
3D images ---> RCNN(for segmenting seeds) ----> stacked masks ----> 3DCNN----> two output layers: one for center and other for orientation
How we built it
For the mask R-CNN, we drew inspiration from the McMed Hacks week 5 workshop where the goal of the model was similar. However, we had to make modifications since the mask R-CNN is trained on 2D images, while we were working with 3D. Then, we employed 3D CNN and MLP techniques we learned.
Challenges we ran into
• Figuring out where to kick-start our project, and the steps needed to reach our end-goal • Planning the architecture and flow of the project • Transferring 3D images provided on AlphaTau to 2D in order to train the data
Accomplishments that we're proud of
Our model has a neat and detailed architecture. Training the first half of our model, we were able to design two parts of the architecture. It was our first time dealing with 3D models and many of our first hackathons, and we are proud of figuring a solution to a complex problem in a small timeframe.
What we learned
• How Mask R-CNNs work, their importance in medical imaging, and how to use convert 3D images on AlphaTau to 2D • We used two models: - One deep learning model Mask RCNN is to segment seed and differentiate instances of seeds. - Another machine learning model is to identify the center and orientation of seeds.
What's next for Alpha Tau Grand Challenge
We want to build on our project and work on some of its shortcomings. Training the model end-to-end and use of Continous Learning will tackle the issue of Catastrophic Forgetting. In the future scope we can perform End-to-End training of the model and tune in the hyperparameters.
Log in or sign up for Devpost to join the conversation.