Splice2Deep is a 2020 research paper published by Somayah Albaradei et al. It uses a simple architecture. It takes raw DNA sequences as input and passes them through a sequence encoding layer, convolutional layers, a ReLU layer, pooling layers, fully connected layers, and then a softmax layer. This results is finally fed into a binary classifier which predicts whether or not a sequence of base pairs constitues a splice site or not. The results appear impressive, with accuracies consistently above 90%.

However, Splice2Deep could be improved. It uses a CNN-based architecture, while other authors have tried to use different approaches, for example architectures which resemble language models. We are also curious about the replicability of Splice2Deep's results, given its relatively simple architecture. Thus, we will endeavor to recreate the Splice2Deep architecture and test it ourselves.

Splice2Deep Deep Splicer

Built With

Share this project:

Updates

Private user

Private user posted an update —

Analysis of Splice2Deep

Group Members:

Jonas Ladenson, Peter Popescu, Annika Singh, and Kenneth Zhang

Introduction:

Splice2Deep endeavors to create a deep learning model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. The accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Methodology:

The data needed for this project are genomes of any species with annotated splice sites to validate the model. We specifically used FASTA files for the base nucleotide sequence and GTF or GFF3 files for annotated information about intron and exon boundaries, also known as splice sites. All data was from ENSEMBL. The GTF/GFF3 files were processed into lists of exon start and stop locations we could use to index into the FASTA file. This was specifically done for the 5 species analyzed in the original Splice2Deep paper: Thale Cress, Japonica Rice, Fruit Fly, C. Elegans, and Human. It was additionally tested on the Camel and Indian Cobra for generalizability. Preprocessing includes removing characters that aren’t A, C, T, or G from the data and creating embeddings/one-hot encodings representing either nucleotides or trinucleotides. The Deep Splicer paper used a slightly different set of organisms that are stored as text files pre-separated by splice site type: donor, acceptor, and other (i.e., not a splice site).

As specified in the Splice2Deep paper, we take raw DNA sequences as input and embed them into either single or trinucleotide one-hot vectors. The vectors are one-hot column-wise such that they create 2D ‘images’ that are then input to a series of convolutional layers, a ReLU layer, pooling layers, fully connected layers, and a softmax layer. This is done separately for three separate convolutional models: the models that go over the whole window and the intron window use the single nucleotide encodings, while the model for the exon window uses the trinucleotide encodings. The result of all three models is finally fed into an MLP binary classifier which predicts whether or not the center of a window of base pairs is a splice site or not. For our final results, each model was trained on 20,000 sequences split equally between positive and negative examples using the hyperparameters detailed in the Splice2Deep paper.

Results:

The original Splice2Deep paper boasts impressive results, consistently achieving above 90% accuracy across donor and acceptor models for different species. In contrast, we were unable to classify splice sites with higher than chance accuracy when using complete splice site information. This came despite making a number of improvements to our approach, including optimizing our preprocessing and utilizing better hardware to increase the amount of training data. We achieved similarly poor results using Splice2Deep’s pretrained models, furthermore indicating that the processing and annotation of our data was our performance bottleneck. After DL Day, we trained our models using data from Deep Splicer, another splice site prediction paper, and achieved much better results. These can be seen in the images we uploaded to DevPost. We suspect that this data is significantly more processed than ours, as the public data in their repository is pre-separated by splice site type. It also appeared to be generally more sanitized when compared to the exon data we processed from ENSEMBL’s GTF files. This was disappointing as it suggested that the paper we investigated was not fully transparent about their model’s capabilities, and was also unable to solve splice site annotation in general, just a specific case.

Challenges:

One challenge was inferring from some of the underspecifications in the paper. Many of the figures in the publication suggested a rather simple architecture, especially with respect to the components of each convolutional network. However, by accessing the model summaries of the models saved in their accompanying repository, we were able to cross-reference with the paper in order to more accurately recreate the architecture used in the paper.

Another challenge was data. As a starting point for our project, we thought we would be able to use data from the authors’ repository. However, this was not accessible as they no longer pay for the appropriate amount of storage through Git LFS. Moreover, the Splice2Deep authors said very little about their preprocessing. Taking inspiration from other similar papers and ENSEMBL (where the Splice2Deep authors got their data from), we thus downloaded GTF and FASTA files for each organism studied in the original paper. Then, we extracted exon start and stop locations from each exon in the GTF file (these ARE where the gene should be spliced). However, this did not end up being a good strategy as it did not allow the model to predict with any accuracy. As discussed in results, we believe that the authors were not transparent about performing significantly more preprocessing to simplify and group the data specifically into donor splice sites.

Reflection:

The project turned out to be less than perfect as it revealed certain shortcomings of the paper. The model’s initially low accuracy is most likely a reflection of the more raw data we used in comparison to the original paper. If we had more time to do the project over again, we would change the preprocessing to label based on spliced sites within a fixed smaller sequence rather than the whole sequence and change the architecture to utilize more sequential layers as well as parallel layers. There are many such data processing practices we have seen detailed in other papers that seem to be well accepted which would have likely improved our model’s performance. Additionally, It would be much more impressive to have a model which had the power to label splice sites generally, rather than detect only a certain type. This was done by the paper that our cleaned-up is from by having their model predict which of the three types of splice sites–including false–a given sequence was. Our biggest takeaway from this project is that the processing of data can greatly affect the efficacy and generalizability of training and detailing the precise augmentations done to a given dataset is vital to the integrity of reproducible research.

posted an update —

Final Write-up / Reflection

Group Members:

Peter Popescu Jonas Ladenson Annika Singh Kenneth Zhang

Introduction:

Splice2Deep endeavors to create a deep learning model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. The accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Methodology:

The data needed for this project are genomes of any species with annotated splice sites to validate the model. We specifically used FASTA files for the base nucleotide sequence and GTF or GFF3 files for annotated information about intron and exon boundaries, also known as splice sites. All data was from ENSEMBL. The GTF/GFF3 files were processed into lists of exon start and stop locations we could use to index into the FASTA file. This was specifically done for the 5 species analyzed in the original Splice2Deep paper: Thale Cress, Japonica Rice, Fruit Fly, C. Elegans, and Human. It was additionally tested on the Camel and Indian Cobra for generalizability. Preprocessing includes removing characters that aren’t A, C, T, or G from the data and creating embeddings/one-hot encodings representing either nucleotides or trinucleotides. The Deep Splicer paper used a slightly different set of organisms that are stored as text files pre-separated by splice site type: donor, acceptor, and other (i.e., not a splice site).

As specified in the Splice2Deep paper, we take raw DNA sequences as input and embed them into either single or trinucleotide one-hot vectors. The vectors are one-hot column-wise such that they create 2D ‘images’ that are then input to a series of convolutional layers, a ReLU layer, pooling layers, fully connected layers, and a softmax layer. This is done separately for three separate convolutional models: the models that go over the whole window and the intron window use the single nucleotide encodings, while the model for the exon window uses the trinucleotide encodings. The result of all three models is finally fed into an MLP binary classifier which predicts whether or not the center of a window of base pairs is a splice site or not. For our final results, each model was trained on 20,000 sequences split equally between positive and negative examples using the hyperparameters detailed in the Splice2Deep paper.

Results:

The original Splice2Deep paper boasts impressive results, consistently achieving above 90% accuracy across donor and acceptor models for different species. In contrast, we were unable to classify splice sites with higher than chance accuracy when using complete splice site information. This came despite making a number of improvements to our approach, including optimizing our preprocessing and utilizing better hardware to increase the amount of training data. We achieved similarly poor results using Splice2Deep’s pretrained models, furthermore indicating that the processing and annotation of our data was our performance bottleneck. After DL Day, we trained our models using data from Deep Splicer, another splice site prediction paper, and achieved much better results. These can be seen in the images we uploaded to DevPost. We suspect that this data is significantly more processed than ours, as the public data in their repository is pre-separated by splice site type. It also appeared to be generally more sanitized when compared to the exon data we processed from ENSEMBL’s GTF files. This was disappointing as it suggested that the paper we investigated was not fully transparent about their model’s capabilities, and was also unable to solve splice site annotation in general, just a specific case.

Challenges:

One challenge was inferring from some of the underspecifications in the paper. Many of the figures in the publication suggested a rather simple architecture, especially with respect to the components of each convolutional network. However, by accessing the model summaries of the models saved in their accompanying repository, we were able to cross-reference with the paper in order to more accurately recreate the architecture used in the paper. Another challenge was data. As a starting point for our project, we thought we would be able to use data from the authors’ repository. However, this was not accessible as they no longer pay for the appropriate amount of storage through Git LFS. Moreover, the Splice2Deep authors said very little about their preprocessing. Taking inspiration from other similar papers and ENSEMBL (where the Splice2Deep authors got their data from), we thus downloaded GTF and FASTA files for each organism studied in the original paper. Then, we extracted exon start and stop locations from each exon in the GTF file (these ARE where the gene should be spliced). However, this did not end up being a good strategy as it did not allow the model to predict with any accuracy. As discussed in results, we believe that the authors were not transparent about performing significantly more preprocessing to simplify and group the data specifically into donor splice sites.

Reflection:

The project turned out to be less than perfect as it revealed certain shortcomings of the paper. The model’s initially low accuracy is most likely a reflection of the more raw data we used in comparison to the original paper. If we had more time to do the project over again, we would change the preprocessing to label based on spliced sites within a fixed smaller sequence rather than the whole sequence and change the architecture to utilize more sequential layers as well as parallel layers. There are many such data processing practices we have seen detailed in other papers that seem to be well accepted which would have likely improved our model’s performance. Additionally, It would be much more impressive to have a model which had the power to label splice sites generally, rather than detect only a certain type. This was done by the paper that our cleaned-up data is from by having their model predict which of the three types of splice sites–including false–a given sequence was. Our biggest takeaway from this project is that the processing of data can greatly affect the efficacy and generalizability of training and detailing the precise augmentations done to a given dataset is vital to the integrity of reproducible research.

Private user

Private user posted an update —

Introduction

Splice2Deep endeavors to create a Deep-learning based model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. As described in the paper's abstract, the accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. The paper's simplicity allows us room for experimentation beyond the scope of the paper. Could we turn this from a binary classification task into a prediction task? How might varying the architecture change accuracy? Importantly, we are also curious as to the paper's replicability. Achieving such high accuracy with a simple model seems too good to be true. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Challenges

The hardest part of the project we’ve encountered so far is inferring from some of the underspecifications in the paper. Many of the figures in the publication suggested a rather simple architecture, especially with respect to the components of each convolutional network. However, by accessing the model summaries of the models saved in their accompanying repository, we were able to cross-reference with the paper in order to more accurately recreate the architecture used in the paper.

Another challenge was that the data we thought we’d be able to use from their repository is not accessible as they no longer are paying for the appropriate amount of storage through Git LFS. This caused a setback in data collection, as we thought we would have their data as a starting point. We instead turned to randomly generating data in order to test the baseline functionality of our models.

Insights

While we don’t have concrete results, we are relatively confident that our architecture matches what was used in the paper, and that the preprocessing we have implemented will allow it to be trained on real data once we can find a good source. The models are able to achieve zero error on a given set of random training data as expected, but it obviously does not generalize well to other samples.

Plan

We are slightly behind on the project especially with respect to data due to the genomes the paper used being locked up in the cloud. Currently, we need to dedicate more time to collecting data for the same organisms used in the paper as well as novel ones in order to evaluate the generalizability of their architecture.

We are thinking of changing some of the architecture, and necessarily, the preprocessing of the data. We found that the parallel structure of their convolution layers in particular to be odd, and also realized that the data could be embedded one-hot traditionally then use 1D convolution instead of embedding them across rows just to use 2D convolution. If we think that something like an LSTM or transformer would perform better than convolution, we may just skip to that as it would likely provide a more interesting alternative method.

Private user

Private user posted an update —

People

  • Jonas Ladenson: jladenso
  • Peter Popescu: ppopescu
  • Annika Singh: asing122
  • Kenneth Zhang: kczhang

Introduction

Splice2Deep endeavors to create a Deep-learning based model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. As described in the paper's abstract, the accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. The paper's simplicity allows us room for experimentation beyond the scope of the paper. Could we turn this from a binary classification task into a prediction task? How might varying the architecture change accuracy? Importantly, we are also curious as to the paper's replicability. Achieving such high accuracy with a simple model seems too good to be true. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Related Work

DeepSS is a deep learning model that predicts splice sites in DNA sequences. It was more accurate than other methods at the time as it could capture higher-level features from the sequences. DeepSS can also identify important sequence motifs that regulate splice sites. This information can help us understand how genes are expressed and regulated. DeepSS uses a convolutional neural network (CNN) to learn relevant features from DNA sequences. The CNN is trained on a dataset of known splice sites and non-splice sites in Homo sapiens and Caenorhabditis elegans. Once trained, the CNN can be used to predict splice sites in new sequences.

Existing Implementations:

https://github.com/SomayahAlbaradei/Splice_Deep

Data

The data needed for this project is a genome (of any species) with annotated splice sites to validate the model. When building / testing our model, we would therefore initially like to use the data provided by the Splice2Deep research paper itself. The data is stored through Git LFS in the above repository and will allow us to replicate the paper and test if its results are accurate. Across positive and negative examples for donor and acceptor sites for the six species in the paper, the data totals around 2GB. Their preprocessing includes removing characters that aren’t A, C, T, or G from the data and creating embeddings/one-hot encodings representing either nucleotides or trinucleotides.

However, we do not believe that this data is sufficient by itself, since it was not provided independently of the paper, and does not allow us to test our model or improvements on additional species from those discussed in the paper. These species are humans, Thale cress (a plant in the mustard family), Japonica rice, the common fruit fly, and the worm C. Elegans. Options to widen the data available to us include ENSEMBL, UC Santa Cruz’ Genome Browser, GENCODE, and the National Center for Biotechnology Information’s RefSeq. Of these, we will choose to use ENSEMBL, and specifically use datasets for the Arabian Camel and the Indian Cobra. These were found at this site, and were chosen because they are eukaryotic animals, unlike most of the data in the paper. Their full genomes are each around 500MB. We should not need to do much preprocessing beyond the specific encodings done by the researchers.

Methods

Splice2Deep takes raw DNA sequences as input and embeds them into either single or trinucleotide one-hot vectors. The vectors are one-hot column-wise such that they create a 2D ‘image’ that is convolutional layers, a ReLU layer, pooling layers, fully connected layers, and then a softmax layer. This is done separately for three separate models: the ones that go over the whole window and the upstream window (towards 5’) use the single nucleotide encodings, while the model for the downstream window (towards 3’) uses the trinucleotide encodings. The result of all three models is finally fed into a binary classifier which predicts whether or not a sequence of base pairs constitutes a splice site or not. Given the relatively simple architecture of each model and small dataset size of 500mb, it should be possible to train the models locally.

Currently, the most difficult part of implementing the existing implementation at the Github link above is setting up an environment that will run the versions of the libraries on which it was implemented. We were attempting to run the pre-trained models from their repository as a baseline before training the models on a similar architecture ourselves, but this may not be worth doing as theoretically this should have the same results as reported in the paper.

Metrics

We have an accuracy metric that constitutes the “success” of our model, quantifying the correctness of the binary label outputs. Where T, F represent true and false for the actual label, respectively, and P, N represent positive and negative for the output the model predicts, respectively. The accuracy is measured by $\frac{TP+TN}{TP+FP+TN+FN}$, and error is 1-accuracy. We plan to run experiments by testing the model on the data the paper used in addition to the others mentioned in the data section then comparing our error rates with those described in the paper. The authors quantified their success in terms of error rate, finding that Slice2Deep reduced the average error rates for acceptor and donor SS by 41.97% and 28.51% respectively compared to previous methods, and achieved accuracies above 95%.

  • Base goal: Reimplement the architecture in the paper
  • Target goal: Improve the convolutional architecture with more layers and filters.
  • Stretch goal: Implement a different architecture that may work better for this sequential data, e.g. RNN or transformer

Ethics

The major stakeholders in this problem are biologists and researchers who would care about finding splice sites to accurately delineate boundaries between exons and introns in a gene’s DNA sequence. This is crucial for understanding gene expression regulation, disease mechanism, genome annotation, and developing targeted therapies. False positives in identifying splice sites could lead to incorrectly including non-functional or irrelevant sequences in gene annotations, potentially leading to ineffective treatments or problems in later experiments. False negatives could result in missing genuine splice sites, impeding accurate gene annotation and potentially overlooking important regulatory elements or disease-causing mutations.

We plan to quantify error as 1 - accuracy, where accuracy is defined as TP+TNTP+FP+TN+FN. This metric implies that every DNA sequence labeling is weighted equally, whether or not they have a splice site. This means that errors are weighted equally, meaning that an extra false positive and false negative have the same effect proportionally on the error and is not relative to any other metric. However, this could be problematic because identifying a non-existent splice site could be a worse or better outcome than not identifying a splice site depending on the application.

Division of Labor

  • Data acquisition: Jonas
  • Implementing existing architecture: Peter
  • Designing experiments: Annika
  • Implementing new architecture: Kenny

posted an update —

Project Check-In #2

People

Jonas Ladenson: jladenso Peter Popescu: ppopescu Annika Singh: asingh122 Kenneth Zhang: kczhang

Introduction

Splice2Deep endeavours to create a Deep-learning based model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. As described in the paper's abstract, the accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species.

We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. The paper's simplicity allows us room for experimentation beyond the scope of the paper. Could we turn this from a binary classification task into a prediction task? How might varying the architecture change accuracy? Importantly, we are also curious as to the paper's replicability. Achieving such high accuracies with a simple model seems too good to be true. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Related Work