Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks
Обучение с использованием суррогатного градиента в спайковых нейронных сетях: перенос возможностей градиентной оптимизации в спайковые нейронные сети
2019-11-01
SCID: 54.1/nfwq4fp4
Discuss with AI
data-driven learningneuromorphic processorsspiking neural networkssurrogate gradient learningsynaptic plasticity
Figures from the paper
Abstract (AI)
Spiking neural networks (SNNs) are nature's versatile solution to fault-tolerant, energy-efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking NN processors have attempted to emulate biological NNs. These developments have created an imminent need for methods and tools that enable such systems to solve real-world signal processing problems. Like conventional NNs, SNNs can be trained on real, domain-specific data; however, their training requires the overcoming of a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training SNNs and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. Accordingly, it gives an overview of existing approaches and provides an introduction to surrogate gradient (SG) methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.
Key Findings
1
Spiking neural networks offer fault-tolerant and energy-efficient signal processing, motivating their implementation on neuromorphic hardware.
2
Surrogate gradient methods are presented as a flexible and efficient approach for overcoming the nondifferentiability challenges of spiking neural networks.
3
The article surveys existing SNN training approaches and provides a step-by-step introduction to surrogate gradient learning.
4
The article systematically explains the main obstacles in SNN training, including concepts related to synaptic plasticity and data-driven learning.
5
Training SNNs on real, domain-specific data is challenging because their binary spike outputs and dynamical behavior complicate gradient-based optimization.
Research Object
spiking neural networks (SNNs)
Research Subject
training SNNs using surrogate-gradient methods, including challenges arising from their binary and dynamical nature
Publication Details
Publication Date
2019-11-01
Journal
Publisher
ISSN
Cited by
1864
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by5
Training Spiking Neural Networks Using Lessons From Deep Learning2023
The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks2020
A Spiking Neural Network With Adaptive Graph Convolution and LSTM for EEG-Based Brain-Computer Interfaces2023
Federated Learning With Spiking Neural Networks2021
Attention Spiking Neural Networks2023