Attention Spiking Neural Networks
Внимательные импульсные нейронные сети
2023-01-31
SCID: 54.1/duyf7hb9
Discuss with AI
ImageNet-1Kblock dynamical isometryevent-based action recognitionmulti-dimensional attentionspiking neural networks
Figures from the paper
Abstract (AI)
Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.
Key Findings
1
Attention weights optimize membrane potentials and regulate spiking responses, enabling vanilla SNNs to achieve sparser firing, higher accuracy, and improved energy efficiency simultaneously.
2
Compared with Res-ANN-104, the performance gaps are -0.95 and +0.21 percentage points, while energy efficiency reaches 31.8× and 7.4× for single-step and four-step models.
3
On ImageNet-1K, single-step and four-step Res-SNN-104 achieve top-1 accuracies of 75.92% and 77.08%, respectively, reported as state-of-the-art SNN results.
4
The paper introduces a multidimensional attention module for SNNs that separately or jointly models temporal, channel, and spatial attention.
5
Theoretical analysis shows that block dynamical isometry can resolve spiking degradation or gradient vanishing commonly encountered in SNNs; a spiking-response visualization method further analyzes efficiency.
Research Object
spiking neural networks (SNNs)
Research Subject
the effects of multi-dimensional attention mechanisms on SNN spiking responses, accuracy, sparsity, energy efficiency, and training dynamics
Publication Details
Publication Date
2023-01-31
Journal
Publisher
ISSN
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks2020
Attention mechanisms in computer vision: A survey2022
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks2019