bulkyBERT

Cecile Meier-Scherling posted an update — Apr 28, 2023 11:01 AM EDT

Introduction

In this project, we want to develop a deep learning model based on a transformer model architecture to learn the temporal and spatial patterns of gene regulation. Any regulatory architecture requires non-linearities in order to be able to exhibit interesting functions, which makes deep neural networks a great tool to model them. The purpose of our model is to learn the temporal patterns in the gene expression data and predict the next time point given a query expression profile. We will use attention to make inferences on which genes inform the prediction of downstream expression patterns.

Challenges

Our original plan was to use a Convolutional Neural Network (CNN) architecture since we are modeling sequential data. The reason why we considered CNNs was that the type of data sets available for time series gene expressions are very small both in terms of sampling frequency and number of replications. However, the problem we are trying to solve, i.e. identifying causal relationships between genes, calls for something like an attention mechanism which simultaneously considers all genes an their mutual importance. Since we made a switch from the CNN to the transformer as out model, we have spent a little more time than originally anticipated. Other than that we have not encountered any issues.

Insights

At this point we are still in the process of finalizing the model. We realize that we will need to train one transformer per gene that takes into account all other genes to predict one gene. This way we hope to recover the generating gene network in the attention scores of the transformer model. We hope to find that we can leverage patterns in gene expression to forecast future gene expression values. Additionally, we hope to use the attention values learned by the transformer to visualize gene expression patterns in a network format.

Plan

We have finished the simulation of our time-series gene expression data. We have also finalized the code for the self-attention and transformer. Lastly, we need to complete the training of the transformer model. In particular, we need to work out how to format the input data and implement the loss function as MSE loss between the true input of the gene that we want to predict and the predicted sequence of the gene.

We decided to change the model from a CNN to a transformer, because we realized transformers more easily relate all variables in a sequence to each other without the spatial proximity constraints of convolution. With positional embedings we can still preserve the sequential information.