Deliverables:
GITHUB: https://github.com/rcrawfo1-brown/DL-TCN-NeuralDecoding PRESENTATION:https://docs.google.com/presentation/d/1aE4LZqD8_WMdcQiF1n_1LZRdrAcPraG8jAXRQZD6BFc/edit?usp=sharing FINAL PAPER: https://docs.google.com/document/d/1yHRlwM7xPe8IlJWvKMScRvUTnDpUH0vjj9tX-yUrgCY/edit?usp=sharing
Brain computer interfaces have the potential to revolutionize assistive technology options for people with paralysis and other severe motor impairments. Currently, most decoding paradigms predominantly make use of fast-time spiking information to inform decoders, however there is evidence to believe that lower frequency local field potentials (LFP) may contain relevant information and improve decoder stability. This paper presents a new methodology for incorporating slow time and fast time LFP data in order to improve decoding accuracy and stability in two use cases. To incorporate LFP temporal dynamics we employed a mixed network model, combining a feature extracting temporal convolutional network encoder with an LSTM decoder in order to effectively incorporate time series information. We evaluated this decoder offline using K-fold validation.
Similar work in our field of Brain Computer Interfaces has been done in attempting to use new variational autoencoder methods to infer latent dynamics from simultaneously recorded, single-trial, high-dimensional neural spiking data. This group of researchers implements the hypothesis that a driven nonlinear dynamical system provides a reasonable model of many neural processes. The Latent Factor Analysis via Dynamical Systems (LFADS) model is an instantiation of a variational auto-encoder (VAE) extended to sequences, which consists of two components, a decoder or generator and an encoder. This VAE model is used for inferring latent dynamics from single-trial, high-dimensional neural spike trains but can be extended to other kinds of neural data.
We will be using neural data gathered in the BrainGate2 clinical trial from participant T11, a 37-year old man with tetraplegia. Data is recorded through two Blackrock Utah microelectrode arrays placed in the motor cortex. Neural data is recorded as a voltage signal per channel, with 96 channels per array at a sampling rate of 30 kHz. In order to shorten processing time and ensure that the model learns and makes use of multiscale data, we will downsample all data to 1 kHz and then further separate the data into multiple bands of LFP features sampled at different rates to create an consistently sized matrix with different bands of information.
Using a pre-existing library created by the Locus Lab at Carnegie Mellon University, we will create a TCN network to predict velocity outputs based on LFP neural data and LFP data in conjunction with previously calculated spiking data. We will compare offline cursor control from velocity outputs based on (1) LFP features and (2) LFP features in conjunction with previously calculated spiking features to just previously calculated spiking features. To compare offline performance, we will use angle error (AE), defined as the angle between the cursor-to-target vector and the directional vector estimated by the decoder (best = 0°; max = 180°). The goal is that LFP features, in combination with spiking features, will aid long-term decoding than just spiking features alone as LFP features have more stable neural tunings than spiking data.
Our dataset contains sensitive human intracortical neural recordings of people with paralysis. Because of its sensitive nature, when curating the dataset, we deidentified the participant's personal information and ensured proper encryption when handling and extracting neural features. We also kept in mind that this is a single participant study so we are not attempting to draw strong conclusions that try to generalize to other people of tetraplegia until further investigated.
The main stakeholder in our problem is the end user who uses the BCI system. In a clinical trial, safety of the user is the utmost priority over any other considerations such as efficacy. Under no circumstances should the participant be harmed, especially when they are in a more vulnerable state of paralysis. While the main goal of our project is to build a more robust decoder that improves the user’s experience during cursor control when using the BCI for long-term use, we should keep in mind that our design should not compromise user’s autonomy and agency.
Literature review: RC, SLE Data curation and preprocessing: EP, RC Model architecture: SLE Metric and analysis: EP, RC Write up/presentation: EP, RC, SLE
Check in 3:
Brain computer interfaces have the potential to revolutionize assistive technology options for people with paralysis and other severe motor impairments. Currently, most decoding paradigms predominantly make use of fast-time spiking information to inform decoders, however there is evidence to believe that lower frequency local field potentials (LFP) may contain relevant information and improve decoder stability. This paper presents a new methodology for incorporating slow time and fast time LFP data in order to improve decoding accuracy and stability in two use cases. To incorporate LFP temporal dynamics we employed a mixed network model, combining a feature extracting temporal convolutional network encoder with an LSTM decoder in order to effectively incorporate time series information. We evaluated this decoder offline using K-fold validation. Challenges: What has been the hardest part of the project you’ve encountered so far? Some data preprocessing and loading challenges. Challenges switching for tensorflow to pytorch. Since this is a new approach we need to extract and test the data in a novel way so there’s some difficulties associated with that. Insights: Are there any concrete results you can show at this point? How is your model performing compared with expectations? No concrete results at this point, we’re working on visualizations. But our model is training on one block of data, today we’re working on scaling it to multiple sessions of data. We plan to use a confusion matrix for showing our discrete gesture decoding model and trajectories for our kinematic decoding model. Once we have visualizations Plan: Are you on track with your project? What do you need to dedicate more time to? What are you thinking of changing, if anything? We’re a little bit behind, we need to work on getting visualizations and training on multiple days worth of data at a time. We also need to try using a larger window of data, as our LSTM used a larger window of data for its predictions and it’s unideal to predict off of just one time step. This should be simple with the Offline LFP data we have calculated and just requires adjusting window sizing .
Final:
The field of brain computer interfacing (BCI) most generally translates time-series and image data recorded from the brain into actions on a computer, prosthetic, or other device. One of the central pushes in the field is to use BCIs to enable people with severe motor disabilities to control a computer for communication. Research by the BrainGate lab at Brown University and others has shown the potential for intracortical brain computer interfaces (iBCIs) to restore communication, movement, and emulate able-bodied device control for people with tetraplegia or other motor impairments. iBCIs use existing neural subspaces to control computers or other devices directly [1, 2]. Given iBCIs require little to no voluntary movement control, they have great potential to revolutionize assistive technologies (ATs) for people with locked-in syndrome and dysarthria. While great progress has been made over the last 30 years to translate iBCIs into ATs, iBCI ATs are still not widely used clinically as an AT. One key issue facing the field is the problem of neural instability. Due to various factors, neural signals across modality are not stable, and can change from session to session or even over the course of one session. This means that in order to effectively use BCI ATs, frequent recalibration is necessary on the part of the patient, increasing mental burden to use the device. Currently, most decoding paradigms predominantly make use of fast-time spiking information to inform decoders, however it has also been shown that local field potentials (LFPs) can be used to produce a more stable decoder than spiking activity, though a decoder with LFPs alone is less accurate than that of spiking data [3, 4]. Local field potentials are filtered neural time series voltages, producing a waveform in a specific frequency range for each channel you record from. In our implementation we were unable to fully debug our pipelines in order to get a working model on neural data. We were able to get a working toy model and we have next steps for fixing our model going forward. Overall we attempted a novel pipeline, and though we had limited success we learned a lot about using hyperparameter optimization, programming in pytorch, and the pitfalls of different DL models.
Log in or sign up for Devpost to join the conversation.