Inspiration
U-net is a well-known network for biomedical image segmentation. This summer our teammates were using U-net for medical image segmentation so we just naturally thought of using U-net on AlphaTau!
How we built it
We preprocessed the data by cropping the raw images. The cropping was based on the raw image's largest connected component. Then used PyTorch to implement U-net. The resulting masks were semantic segmentations at the moment. Then applied Cluster method on masks and were able to label each seed. Then pipelined it for centroid + orientations calculations.
Challenges we ran into
While cropping the image, we binarized the original image first as a kind of augmentation, then used findContours() function to generate a list of optional contours. We picked out the largest bounding box from the contours, which retains the largest boundary of the organ. But sometimes the bounding box may miss some seeds since these seeds exceed the boundary of the organ.
Limited GPUs, it's hard to compete with others as everyone wants more training resources!
Seeds in the image can be hard to recognize because they are very small. We can't really trust 'accuracy' so we used DiceScore.
Calculating orientation of the seeds. We went for some algorithms online and decided to use the distance between the furthest 2 points in a connected component (seed)
We tried to do 3D modeling on the brain by CT images. We used the measuring module in Skimage to extract vertices and faces from raw images and applied Poly3DCollection to generate meshes. We learned the Hounsfield Unit of mouth is around 40 so we set the threshold to 40. But it seems that the result does not make sense. We might lack some essential information (metadata). Usually, when we want to do 3D modeling by CT image we need DICOM format data which contains a thickness of slice and interval between different slices etc., but we just have NumPy files, there is no DICOM.
Accomplishments that we're proud of
We applied clustering methods such as DBSCAN and KMeans to classify all non-zero coordinates in the raw image. We plotted clusters in 3D space, each cluster referring to a seed, where different seeds were denoted by different colors. The visualization was able to clearly show the seeds in 3d. While training, we found the resolution and the size of the raw images are too large to process. In order to decrease the expense during training, we cropped out the region of interest(ROI) from the whole CT image. After we trained with the original training dataset, we re-trained our model with these new cropped images. We got a significant increase in Dice Score.
What we learned
We got to learn so much! We implemented U-net and learned that accuracy may not be the best approach in some scenarios. We also learned how to calculate orientations and centroids of connected components (seeds). There are definitely a lot more! Last but not least, GatherTown is such an amazing online platform for us to chill!!
What's next for AlphaTau Seed Segmentation Using U-net
Better Region of Interests. So far our ROI takes the assumptions that the mouth/brain area is all gray. However, when browsing the data, there were some samples that have mouth/brain areas in darker gray and then wrap by the lighter gray areas. This will highly impact our ROI and affected prediction results.
Optimizing our UNet architecture by applying the attention mechanism, which will allow for the network to put more weight on the features of the skip connection that will be relevant. It allows for the direct connection to focus on a particular part of the input, rather than feeding in every feature.
Applies Encoder/Decoder for U-net.
Log in or sign up for Devpost to join the conversation.