Perforated NeuroShrink

Inspiration

Modern neural networks continue to grow larger and more expensive, even though many of their connections contribute little to final predictions. This inefficiency limits deployment on edge devices and increases energy and infrastructure costs. Our inspiration came from biological neurons, which do not use dense, always-active connections. Instead, they rely on selective dendritic branches that activate only when needed. We wanted to explore whether this principle could be translated into practical neural network optimization using PyTorch.

What it does

NeuroShrink explores dendritic optimization to reduce neural network size and computation while preserving full trainability. We replace standard dense convolution layers with dendritic-style sparse convolutions, enabling selective computation. Beyond static sparsity, we introduce an adaptive dendritic mechanism where dendritic branch utilization dynamically responds to activation strength during inference, enabling input-aware computation.

How we built it

We built NeuroShrink entirely in PyTorch, training all models from scratch. A baseline ResNet-18 was implemented first, then standard convolution layers were replaced with dendritic-style sparse convolutions using grouped operations and learnable masks. We further added an adaptive dendritic mechanism that dynamically scales dendritic branch usage based on activation strength during inference. The entire pipeline is reproducible and benchmarked consistently.

Challenges we ran into

The main challenge was integrating dendritic layers into an existing architecture without breaking training stability or reproducibility. Designing adaptive behavior that remains simple, interpretable, and computationally lightweight under tight time constraints was also non-trivial.

Accomplishments that we're proud of

We achieved a 98.43% reduction in parameters and compute compared to a standard ResNet-18, while maintaining a clean and fully trainable architecture. We also successfully demonstrated adaptive, input-aware dendritic computation without relying on pretrained models or additional datasets.

What we learned

We learned that large efficiency gains are possible without changing model architecture or data. Biological inspiration, when applied carefully, can lead to practical and measurable improvements in modern neural networks.

What's next for Perforated NeuroShrink

Future work includes training-time adaptive dendrites, integration with larger vision and transformer models, and deployment-focused evaluation on edge and low-power hardware.

Built With

• python
• pytorch
• torchvision