Representation Learning with Contrastive Predictive Coding
Обучение представлений с помощью контрастивного предиктивного кодирования
2018-07-10
SCID: 54.1/af5k2gjr
Discuss with AI
Contrastive Predictive Codingautoregressive modelsnegative samplingprobabilistic contrastive lossunsupervised representation learning
Figures from the paper
Abstract (AI)
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Key Findings
1
A probabilistic contrastive loss encourages representations to retain information maximally useful for predicting future samples while remaining tractable through negative sampling.
2
Contrastive Predictive Coding is proposed as a universal unsupervised method for learning useful representations from high-dimensional data.
3
The approach achieves strong representation-learning performance across four domains: speech, images, text, and reinforcement learning in 3D environments.
4
The method predicts future observations in latent space using powerful autoregressive models rather than predicting directly in the original data space.
Research Object
Contrastive Predictive Coding representations learned from high-dimensional data across speech, images, text, and 3D reinforcement-learning environments
Research Subject
Learning useful latent representations by predicting future samples with autoregressive models and a probabilistic contrastive loss
Publication Details
Publication Date
2018-07-10
Journal
Publisher
ISSN
Cited by
14624
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
ImageNet classification with deep convolutional neural networks2017
Adam: A Method for Stochastic Optimization2014
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
Sequence to Sequence Learning with Neural Networks2014
FaceNet: A unified embedding for face recognition and clustering2015
Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups2012
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation2014
Kaldi Speech Recognition Toolkit2024
Cited by15
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units2021
On the Opportunities and Risks of Foundation Models2021
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing2022
An Empirical Study of Training Self-Supervised Vision Transformers2021
Toward Causal Representation Learning2021
Self-Supervised Speech Representation Learning: A Review2022
ETC: Encoding Long and Structured Inputs in Transformers2020
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech2021
SSAST: Self-Supervised Audio Spectrogram Transformer2022
A comprehensive survey of deep learning for time series forecasting: architectural diversity and open challenges2025
Vision-language models for medical report generation and visual question answering: a review2024
3D Infomax improves GNNs for Molecular Property Prediction2021
Speaker Normalization for Self-Supervised Speech Emotion Recognition2022
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A Survey2025
SeDR: Segment Representation Learning for Long Documents Dense Retrieval2022