Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for NeuronSpark

NeuronSpark

Inspiration

Modern language models are dense Transformer stacks that spend identical compute on every token — whether it is a function word or a hard reasoning step. Biological cortex works differently: spiking neurons fire only when needed, and recurrent state carries memory across time. We asked whether a language model can be built entirely from spiking neurons — not a Transformer with spiking activations, but a pure-SNN core — and still learn language at scale.

What it does

NeuronSpark is a pure spiking neural network language model with selective state-space dynamics. The full residual stream, the FFN, and the associative memory layers are all implemented with spiking neuron dynamics trained end-to-end on next-token prediction. Each token is processed by an adaptive number of internal micro-steps: easy tokens terminate early, hard tokens spend more compute — the routing decision is causal and made by the network itself. An attention-style memory layer provides associative recall on top of the spiking recurrent backbone.

How we built it

• PLIF-style spiking neurons with surrogate-gradient training on the standard LM objective.
• Per-block selective state-space recurrence; every recurrent path has a PyTorch eager reference and a fused production kernel, verified bit-for-bit for forward/backward/state parity.
• Adaptive per-token compute (K-routing) under a fixed parameter budget, with dormant/active controlled ablations for clean attribution.

Challenges we faced

• BF16 quantization froze the threshold dynamics; fixed by promoting precision-sensitive tensors to FP32 before optimizer construction.
• Keeping the spiking recurrence fully parallel without breaking causality (single pass, one state update per token).
• Maintaining loss viability while the network learns to differentiate token difficulty.

What we learned

Sample-efficient training of a pure-SNN language model is viable, and the key engineering is not the neuron model itself but the precision protocol, the recurrence kernel, and the causal routing decision.

Built With

• cuda
• deepspeed
• python
• pytorch
• spiking-neural-networks
• triton