Inspiration
Gradient descent, the basis of most neural networks, is inefficient. I sought to make a neural network which learns in a way that is more similar to the way the human brain learns. https://en.wikipedia.org/wiki/Hebbian_theory is a theory in neuroscience which proposes a mechanism by which the brain learns - by associating stimuli that occur frequently together. Colloquially, "Neurons that fire together, wire together," NFTWT, I designed Knifty around this principle. There are lots of implementations of Hebbian learning for neural networks out there, but I wanted to challenge myself to see if I could do it without using floats or division. I don't claim this is super original, just was something I want to try to get to work.
What it does
Knifty is a data classifier, similar to a self-organized map. A data set, in the form of a set of n-dimensional vectors, is fed into the network. The net then seeks to partition the data.
Traditional NNs work on gradient descent - you feed in a stimulus, activate the network, get the output Y, compare that to some expected output Y', then try to reduce that error (Y-Y') by adjusting the weights of the network based on the derivative of the activation function. The main problem with this is a) it's computationally inefficient and b) the deeper your NN gets, the more "dilute" the error signal gets from back-propagation.
It's kinda similar to the k-means algorithm included in Theano, which I found after I submitted this, so yeah, there's that, but this is different :P
Knifty works by checking to see which neurons participated in a network. If a neuron in Layer 2 participates in getting a neuron in Layer 3 to activate, then the weight of that connection is increased. Otherwise, the weight is decreased. In effect, neurons that fire together, wire together. The sensitivity of the neuron also adjusts over time, such that the longer period of time a neuron fails to get activated, the more sensitive to activation it becomes. Conversely, frequently activated neurons are de-sensitized. This is modeled on the biological concept of upregulation and downregulation.
All operations are done with integers, and the only operations are addition and multiplication. There is no division, logarithm, sigmoid, or derivatives. This leads to massive gains in computational efficiency.
How I built it
Pretty much just mashed out some python code. I did the design phase on a whiteboard with my friend.
Challenges I ran into
The hardest part was planning out how the network would piece together.
Accomplishments that I'm proud of
I had absolutely no idea that this would actually work. It was pretty much a wild idea kicking around in my mind.
What I learned
Sometimes crazy ideas actually work.
What's next for KNIFTY - A Hebbian Classifier without Gradient Descent
Right now, I'm merely testing on simple, generated clustering datasets. I would like to start feeding in real-life data, such as demographic data, images, and EEG data. I also want to write it into a compiled language such as C for the performance gains.
Log in or sign up for Devpost to join the conversation.