Reducing the Dimensionality of Data with Neural Networks

Geoffrey E. Hinton; Ruslan Salakhutdinov

doi:10.1126/science.1127647

Reducing the Dimensionality of Data with Neural Networks

Снижение размерности данных с помощью нейронных сетей

Geoffrey E. Hinton, Ruslan Salakhutdinov

2006-07-27

SCID: 54.1/3q9byxxu Discuss with AI

autoencoderdimensionality reductionmultilayer neural networkprincipal components analysis (PCA)weight initialization

Abstract (AI)

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such "autoencoder" networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Key Findings

An effective weight-initialization method is described that enables deep autoencoders to learn useful low-dimensional codes.

Deep autoencoder–learned codes outperform principal components analysis for dimensionality reduction according to the abstract.

Gradient descent fine-tuning of autoencoder weights is effective only when initial weights are near a good solution.

Multilayer neural networks with a small central layer can convert high-dimensional data into low-dimensional codes by reconstructing inputs.

Research Object

Deep autoencoder neural network trained to reconstruct high-dimensional input vectors with a small central (bottleneck) layer

Research Subject

Learning low-dimensional codes (dimensionality reduction) via effective weight initialization and gradient-descent fine-tuning to produce representations superior to principal component analysis

Publication Details

DOI

10.1126/science.1127647

Publication Date

2006-07-27

Journal

Publisher

ISSN

Access Type

Author Information

Authors

Geoffrey E. Hinton

Ruslan Salakhutdinov

Download PDF

Make a presentation

References available in scid.ai1

Neural networks and physical systems with emergent collective computational abilities.1982

Cited by20

Artificial intelligence: A powerful paradigm for scientific research2021 A Unifying Review of Deep and Shallow Anomaly Detection2021 State of the Art in Defect Detection Based on Machine Vision2021 Deep learning in optical metrology: a review2022 Deep learning for AI2021 Deep Learning for Time Series Anomaly Detection: A Survey2024 Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects2021 A review of vibration-based damage detection in civil structures: From traditional methods to Machine Learning and Deep Learning applications2020 Digital Twin: Values, Challenges and Enablers From a Modeling Perspective2020 Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities2020 Deep Learning for Generic Object Detection: A Survey2019 Recent advances and applications of machine learning in solid-state materials science2019 Object Detection With Deep Learning: A Review2019 Speech Recognition Using Deep Neural Networks: A Systematic Review2019 Review of Deep Learning Algorithms and Architectures2019 Deep Learning for Computer Vision: A Brief Review2018 Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017 Least Squares Generative Adversarial Networks2017 Image-to-Image Translation with Conditional Adversarial Networks2017 A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines2016