Private user

Private user posted an update —

People

  • Jonas Ladenson: jladenso
  • Peter Popescu: ppopescu
  • Annika Singh: asing122
  • Kenneth Zhang: kczhang

Introduction

Splice2Deep endeavors to create a Deep-learning based model to identify splice sites in genomes. This is a binary classification task which takes a sequence of base pairs and predicts whether they represent a splice site or not. As described in the paper's abstract, the accurate identification of intron/exon boundaries is important for the correct annotation of genes with multiple exons. Approaching this problem with a model, rather than identifying boundaries analytically, is useful because it allows generalization to multiple species. We chose this paper for a few reasons. It is relatively simple, and is different from many other DL models in genetics, which behave more similarly to language models than CNNs. The paper's simplicity allows us room for experimentation beyond the scope of the paper. Could we turn this from a binary classification task into a prediction task? How might varying the architecture change accuracy? Importantly, we are also curious as to the paper's replicability. Achieving such high accuracy with a simple model seems too good to be true. We are curious whether this will hold up when we test the model, and if we evaluate it on different species.

Related Work

DeepSS is a deep learning model that predicts splice sites in DNA sequences. It was more accurate than other methods at the time as it could capture higher-level features from the sequences. DeepSS can also identify important sequence motifs that regulate splice sites. This information can help us understand how genes are expressed and regulated. DeepSS uses a convolutional neural network (CNN) to learn relevant features from DNA sequences. The CNN is trained on a dataset of known splice sites and non-splice sites in Homo sapiens and Caenorhabditis elegans. Once trained, the CNN can be used to predict splice sites in new sequences.

Existing Implementations:

https://github.com/SomayahAlbaradei/Splice_Deep

Data

The data needed for this project is a genome (of any species) with annotated splice sites to validate the model. When building / testing our model, we would therefore initially like to use the data provided by the Splice2Deep research paper itself. The data is stored through Git LFS in the above repository and will allow us to replicate the paper and test if its results are accurate. Across positive and negative examples for donor and acceptor sites for the six species in the paper, the data totals around 2GB. Their preprocessing includes removing characters that aren’t A, C, T, or G from the data and creating embeddings/one-hot encodings representing either nucleotides or trinucleotides.

However, we do not believe that this data is sufficient by itself, since it was not provided independently of the paper, and does not allow us to test our model or improvements on additional species from those discussed in the paper. These species are humans, Thale cress (a plant in the mustard family), Japonica rice, the common fruit fly, and the worm C. Elegans. Options to widen the data available to us include ENSEMBL, UC Santa Cruz’ Genome Browser, GENCODE, and the National Center for Biotechnology Information’s RefSeq. Of these, we will choose to use ENSEMBL, and specifically use datasets for the Arabian Camel and the Indian Cobra. These were found at this site, and were chosen because they are eukaryotic animals, unlike most of the data in the paper. Their full genomes are each around 500MB. We should not need to do much preprocessing beyond the specific encodings done by the researchers.

Methods

Splice2Deep takes raw DNA sequences as input and embeds them into either single or trinucleotide one-hot vectors. The vectors are one-hot column-wise such that they create a 2D ‘image’ that is convolutional layers, a ReLU layer, pooling layers, fully connected layers, and then a softmax layer. This is done separately for three separate models: the ones that go over the whole window and the upstream window (towards 5’) use the single nucleotide encodings, while the model for the downstream window (towards 3’) uses the trinucleotide encodings. The result of all three models is finally fed into a binary classifier which predicts whether or not a sequence of base pairs constitutes a splice site or not. Given the relatively simple architecture of each model and small dataset size of 500mb, it should be possible to train the models locally.

Currently, the most difficult part of implementing the existing implementation at the Github link above is setting up an environment that will run the versions of the libraries on which it was implemented. We were attempting to run the pre-trained models from their repository as a baseline before training the models on a similar architecture ourselves, but this may not be worth doing as theoretically this should have the same results as reported in the paper.

Metrics

We have an accuracy metric that constitutes the “success” of our model, quantifying the correctness of the binary label outputs. Where T, F represent true and false for the actual label, respectively, and P, N represent positive and negative for the output the model predicts, respectively. The accuracy is measured by $\frac{TP+TN}{TP+FP+TN+FN}$, and error is 1-accuracy. We plan to run experiments by testing the model on the data the paper used in addition to the others mentioned in the data section then comparing our error rates with those described in the paper. The authors quantified their success in terms of error rate, finding that Slice2Deep reduced the average error rates for acceptor and donor SS by 41.97% and 28.51% respectively compared to previous methods, and achieved accuracies above 95%.

  • Base goal: Reimplement the architecture in the paper
  • Target goal: Improve the convolutional architecture with more layers and filters.
  • Stretch goal: Implement a different architecture that may work better for this sequential data, e.g. RNN or transformer

Ethics

The major stakeholders in this problem are biologists and researchers who would care about finding splice sites to accurately delineate boundaries between exons and introns in a gene’s DNA sequence. This is crucial for understanding gene expression regulation, disease mechanism, genome annotation, and developing targeted therapies. False positives in identifying splice sites could lead to incorrectly including non-functional or irrelevant sequences in gene annotations, potentially leading to ineffective treatments or problems in later experiments. False negatives could result in missing genuine splice sites, impeding accurate gene annotation and potentially overlooking important regulatory elements or disease-causing mutations.

We plan to quantify error as 1 - accuracy, where accuracy is defined as TP+TNTP+FP+TN+FN. This metric implies that every DNA sequence labeling is weighted equally, whether or not they have a splice site. This means that errors are weighted equally, meaning that an extra false positive and false negative have the same effect proportionally on the error and is not relative to any other metric. However, this could be problematic because identifying a non-existent splice site could be a worse or better outcome than not identifying a splice site depending on the application.

Division of Labor

  • Data acquisition: Jonas
  • Implementing existing architecture: Peter
  • Designing experiments: Annika
  • Implementing new architecture: Kenny