Improving neural networks by preventing co-adaptation of feature detectors

Geoffrey E. Hinton; Ilya Sutskever; Alex Krizhevsky; Ruslan Salakhutdinov; Nitish Srivastava

doi:10.48550/arxiv.1207.0580

Improving neural networks by preventing co-adaptation of feature detectors

Улучшение нейронных сетей путем предотвращения соадаптации детекторов признаков

Geoffrey E. Hinton, Ilya Sutskever, Alex Krizhevsky, Ruslan Salakhutdinov, Nitish Srivastava

2012-07-03

SCID: 54.1/wmhtcehz Discuss with AI

dropoutfeedforward neural networksoverfitting reductionpreventing co-adaptation of feature detectorsspeech and object recognition benchmarks

Abstract (AI)

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.

Key Findings

Dropout prevents complex co-adaptations where a feature detector is only useful in combination with specific other detectors

Dropout yields large improvements on many benchmark tasks and sets new records for speech and object recognition

Randomly omitting (dropping out) half of feature detectors during training greatly reduces overfitting in large feedforward neural networks trained on small datasets

With dropout, each neuron learns features that are broadly useful across many internal contexts, improving generalization

Research Object

Large feedforward neural network trained on small training sets (with dropout applied to feature detectors)

Research Subject

Reducing overfitting by preventing co-adaptation of feature detectors via random omission (dropout), leading to improved generalization and benchmark performance

Publication Details

DOI

10.48550/arxiv.1207.0580

Publication Date

2012-07-03

Journal

Publisher

ISSN

Cited by

6639

Open access PDF

https://arxiv.org/pdf/1207.0580

Access Type

Author Information

Authors

Geoffrey E. Hinton

Ilya Sutskever

Alex Krizhevsky

Ruslan Salakhutdinov

Nitish Srivastava

Download PDF

Make a presentation

References available in scid.ai3

Gradient-based learning applied to document recognition1998 Learning Multiple Layers of Features from Tiny Images2024 Reducing the Dimensionality of Data with Neural Networks2006

Cited by20

ImageNet classification with deep convolutional neural networks2017 Distilling the Knowledge in a Neural Network2015 Convolutional Neural Networks for Sentence Classification2014 Non-local Neural Networks2018 Object Detection With Deep Learning: A Review2019 Convolutional neural networks: an overview and application in radiology2018 Neural Architectures for Named Entity Recognition2016 Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks2016 Knowledge Graph Embedding: A Survey of Approaches and Applications2017 Deep learning applications and challenges in big data analytics2015 Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN2016 A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs2016 Deep learning in optical metrology: a review2022 Deep learning for AI2021 FS-Net: A Flow Sequence Network For Encrypted Traffic Classification2019 End-to-end text-dependent speaker verification2016 Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects2021 FlowPic: A Generic Representation for Encrypted Traffic Classification and Applications Identification2021 Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness2020 Deep Residual Learning for Image Recognition2016