Improving neural networks by preventing co-adaptation of feature detectors
Улучшение нейронных сетей путем предотвращения соадаптации детекторов признаков
2012-07-03
SCID: 54.1/wmhtcehz
Discuss with AI
dropoutfeedforward neural networksoverfitting reductionpreventing co-adaptation of feature detectorsspeech and object recognition benchmarks
Figures from the paper
Abstract (AI)
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
Key Findings
1
Dropout prevents complex co-adaptations where a feature detector is only useful in combination with specific other detectors
2
Dropout yields large improvements on many benchmark tasks and sets new records for speech and object recognition
3
Randomly omitting (dropping out) half of feature detectors during training greatly reduces overfitting in large feedforward neural networks trained on small datasets
4
With dropout, each neuron learns features that are broadly useful across many internal contexts, improving generalization
Research Object
Large feedforward neural network trained on small training sets (with dropout applied to feature detectors)
Research Subject
Reducing overfitting by preventing co-adaptation of feature detectors via random omission (dropout), leading to improved generalization and benchmark performance
Publication Details
Publication Date
2012-07-03
Journal
Publisher
ISSN
Cited by
8652
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Cited by20
Deep Residual Learning for Image Recognition2016
ImageNet classification with deep convolutional neural networks2017
Adam: A Method for Stochastic Optimization2014
Distilling the Knowledge in a Neural Network2015
Convolutional Neural Networks for Sentence Classification2014
Non-local Neural Networks2018
Object Detection With Deep Learning: A Review2019
Convolutional neural networks: an overview and application in radiology2018
Neural Architectures for Named Entity Recognition2016
Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks2016
Deep learning applications and challenges in big data analytics2015
Knowledge Graph Embedding: A Survey of Approaches and Applications2017
Accelerating t-SNE using tree-based algorithms2014
Deep Learning for IoT Big Data and Streaming Analytics: A Survey2018
Deep learning in optical metrology: a review2022
Deep learning for AI2021
A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs2016
Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN2016
FS-Net: A Flow Sequence Network For Encrypted Traffic Classification2019
Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness2020