Neural decoding with spiking neural networks

GIF
Spiking neuron model dynamics (leaky integrate-and-fire).
GIF
Visualization of the experimental spiking data (130 channels were reshaped into 13x10 grid). Spike times were used as inputs in our network.
Architecture of our spiking neural network. We also included recurrent weights between the hidden layer units (no shown in figure).
Average mean squared error loss between the output neurons membrane voltage and the true x and y components of the monkeys finger velocity.
Example from the training dataset. Top.) Membrane potential of output neurons. Bottom.) Target finger velocity (x and y components).
Example from the test dataset. Top.) Membrane potential of output neurons. Bottom.) Target finger velocity (x and y components).

Inspiration

As the number of companies developing brain-computer interfaces continues to grow (e.g., Neuralink and Synchron), it is critical to investigate low latency and energy-efficient neural decoding methods. We wanted to show that spiking neural networks (SNNs), which can be implemented on neuromorphic chips, can decode such signals and fill this gap.

Additionally, SNNs are closer to the underlying neuron biophysics. Therefore, we hoped that by using realistic neural dynamics, we could learn more useful temporal representations in the data compared to traditional decoding or artificial neural network methods which rely on averaging the number of spikes over time.

What it does

Using an SNN with leaky integrate-and-fire dynamics we decode the finger velocity of a primate completing a reaching task from multielectrode array neural recordings.

Neural spike time recordings -> SNN model -> Predict output finger velocity in real time

Two output neurons were trained in a supervised fashion so that their subthreshold membrane potentials matched the temporal dynamics of the monkey's finger velocity (x and y components of the finger motion after projecting the 3D finger position onto a 2D screen in front of the monkey).

With minimal hyperparameter tuning, we achieved decent performance, with the output neurons capturing the timing and general trend of the finger velocity.

How we built it

The neural data can be found at link

The spiking neural network was built using PyTorch. The equations for the LIF dynamics were discretized and simulated forwards in time using an exponential integration method. A surrogate gradient method was used to allow gradients to flow freely through the discontinuous activation used for spiking.

For the SNN architecture, we had 130 units in the input layer corresponding to the channels of the multielectrode array, 50 units in the hidden layer that also had recurrent connections, and 2 units in the output layer corresponding to the x and y component of the predicted finger velocity.

For a tutorial on surrogate gradient descent

Challenges we ran into

Took longer than expected to understand the formatting of the spike timing and finger velocity data
Trying to tune the hyperparameters to improve performance was hard given the time constraint of the hackathon
We tried a more complicated neuron model (LIF with adaptation), but performance didn't improve. However, this might change with a more rigorous grid search over the hyperparameters

Accomplishments that we're proud of

We were able to get a basic spiking neural network running and achieve decent performance on the test set, something that we're quite happy with given the timeline

What we learned

We all learned a lot about spiking neural networks and PyTorch!

What's next for Neural decoding with spiking neural networks project

Parameter tuning to improve decoding performance
Different machine learning methods could be implemented using SNNs (e.g., generate a latent representation with a variational autoencoder and then train a readout layer)
Ultimately we hope to see SNNs being used in Brain-Computer Interfaces