Private user

Private user posted an update —

Analysis of Splice2Deep

Group Members:

Jonas Ladenson, Peter Popescu, Annika Singh, and Kenneth Zhang

Introduction:

Splice2Deep endeavors to create a deep learning model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. The accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Methodology:

The data needed for this project are genomes of any species with annotated splice sites to validate the model. We specifically used FASTA files for the base nucleotide sequence and GTF or GFF3 files for annotated information about intron and exon boundaries, also known as splice sites. All data was from ENSEMBL. The GTF/GFF3 files were processed into lists of exon start and stop locations we could use to index into the FASTA file. This was specifically done for the 5 species analyzed in the original Splice2Deep paper: Thale Cress, Japonica Rice, Fruit Fly, C. Elegans, and Human. It was additionally tested on the Camel and Indian Cobra for generalizability. Preprocessing includes removing characters that aren’t A, C, T, or G from the data and creating embeddings/one-hot encodings representing either nucleotides or trinucleotides. The Deep Splicer paper used a slightly different set of organisms that are stored as text files pre-separated by splice site type: donor, acceptor, and other (i.e., not a splice site).

As specified in the Splice2Deep paper, we take raw DNA sequences as input and embed them into either single or trinucleotide one-hot vectors. The vectors are one-hot column-wise such that they create 2D ‘images’ that are then input to a series of convolutional layers, a ReLU layer, pooling layers, fully connected layers, and a softmax layer. This is done separately for three separate convolutional models: the models that go over the whole window and the intron window use the single nucleotide encodings, while the model for the exon window uses the trinucleotide encodings. The result of all three models is finally fed into an MLP binary classifier which predicts whether or not the center of a window of base pairs is a splice site or not. For our final results, each model was trained on 20,000 sequences split equally between positive and negative examples using the hyperparameters detailed in the Splice2Deep paper.

Results:

The original Splice2Deep paper boasts impressive results, consistently achieving above 90% accuracy across donor and acceptor models for different species. In contrast, we were unable to classify splice sites with higher than chance accuracy when using complete splice site information. This came despite making a number of improvements to our approach, including optimizing our preprocessing and utilizing better hardware to increase the amount of training data. We achieved similarly poor results using Splice2Deep’s pretrained models, furthermore indicating that the processing and annotation of our data was our performance bottleneck. After DL Day, we trained our models using data from Deep Splicer, another splice site prediction paper, and achieved much better results. These can be seen in the images we uploaded to DevPost. We suspect that this data is significantly more processed than ours, as the public data in their repository is pre-separated by splice site type. It also appeared to be generally more sanitized when compared to the exon data we processed from ENSEMBL’s GTF files. This was disappointing as it suggested that the paper we investigated was not fully transparent about their model’s capabilities, and was also unable to solve splice site annotation in general, just a specific case.

Challenges:

One challenge was inferring from some of the underspecifications in the paper. Many of the figures in the publication suggested a rather simple architecture, especially with respect to the components of each convolutional network. However, by accessing the model summaries of the models saved in their accompanying repository, we were able to cross-reference with the paper in order to more accurately recreate the architecture used in the paper.

Another challenge was data. As a starting point for our project, we thought we would be able to use data from the authors’ repository. However, this was not accessible as they no longer pay for the appropriate amount of storage through Git LFS. Moreover, the Splice2Deep authors said very little about their preprocessing. Taking inspiration from other similar papers and ENSEMBL (where the Splice2Deep authors got their data from), we thus downloaded GTF and FASTA files for each organism studied in the original paper. Then, we extracted exon start and stop locations from each exon in the GTF file (these ARE where the gene should be spliced). However, this did not end up being a good strategy as it did not allow the model to predict with any accuracy. As discussed in results, we believe that the authors were not transparent about performing significantly more preprocessing to simplify and group the data specifically into donor splice sites.

Reflection:

The project turned out to be less than perfect as it revealed certain shortcomings of the paper. The model’s initially low accuracy is most likely a reflection of the more raw data we used in comparison to the original paper. If we had more time to do the project over again, we would change the preprocessing to label based on spliced sites within a fixed smaller sequence rather than the whole sequence and change the architecture to utilize more sequential layers as well as parallel layers. There are many such data processing practices we have seen detailed in other papers that seem to be well accepted which would have likely improved our model’s performance. Additionally, It would be much more impressive to have a model which had the power to label splice sites generally, rather than detect only a certain type. This was done by the paper that our cleaned-up is from by having their model predict which of the three types of splice sites–including false–a given sequence was. Our biggest takeaway from this project is that the processing of data can greatly affect the efficacy and generalizability of training and detailing the precise augmentations done to a given dataset is vital to the integrity of reproducible research.