Introduction
Splice2Deep endeavors to create a Deep-learning based model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. As described in the paper's abstract, the accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. The paper's simplicity allows us room for experimentation beyond the scope of the paper. Could we turn this from a binary classification task into a prediction task? How might varying the architecture change accuracy? Importantly, we are also curious as to the paper's replicability. Achieving such high accuracy with a simple model seems too good to be true. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.
Challenges
The hardest part of the project we’ve encountered so far is inferring from some of the underspecifications in the paper. Many of the figures in the publication suggested a rather simple architecture, especially with respect to the components of each convolutional network. However, by accessing the model summaries of the models saved in their accompanying repository, we were able to cross-reference with the paper in order to more accurately recreate the architecture used in the paper.
Another challenge was that the data we thought we’d be able to use from their repository is not accessible as they no longer are paying for the appropriate amount of storage through Git LFS. This caused a setback in data collection, as we thought we would have their data as a starting point. We instead turned to randomly generating data in order to test the baseline functionality of our models.
Insights
While we don’t have concrete results, we are relatively confident that our architecture matches what was used in the paper, and that the preprocessing we have implemented will allow it to be trained on real data once we can find a good source. The models are able to achieve zero error on a given set of random training data as expected, but it obviously does not generalize well to other samples.
Plan
We are slightly behind on the project especially with respect to data due to the genomes the paper used being locked up in the cloud. Currently, we need to dedicate more time to collecting data for the same organisms used in the paper as well as novel ones in order to evaluate the generalizability of their architecture.
We are thinking of changing some of the architecture, and necessarily, the preprocessing of the data. We found that the parallel structure of their convolution layers in particular to be odd, and also realized that the data could be embedded one-hot traditionally then use 1D convolution instead of embedding them across rows just to use 2D convolution. If we think that something like an LSTM or transformer would perform better than convolution, we may just skip to that as it would likely provide a more interesting alternative method.