Distributed Representations of Words and Phrases and their Compositionality
Распределённые представления слов и фраз и их композиционность
2013-10-16
SCID: 54.1/jmr6chh2
Discuss with AI
continuous Skip-gramdistributed word representationsnegative samplingphrase representationssubsampling frequent words
Figures from the paper
Abstract (AI)
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
Key Findings
1
A phrase-detection method enables learning high-quality vector representations for millions of multiword phrases.
2
Negative sampling is introduced as a simple alternative to hierarchical softmax for training word representations.
3
Subsampling frequent words substantially accelerates training and produces more regular word representations.
4
The authors identify that standard word vectors ignore word order and cannot adequately represent idiomatic phrases compositionally.
5
The paper extends the continuous Skip-gram model to improve distributed vector quality and training efficiency.
Research Object
distributed vector representations of words and multi-word phrases learned from text
Research Subject
the syntactic and semantic compositionality of word and phrase representations, including phrase meaning, word-order limitations, representation quality, and training efficiency
Publication Details
Publication Date
2013-10-16
Journal
Publisher
ISSN
Cited by
24533
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by20
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
"Why Should I Trust You?": Explaining the Predictions of Any Classifier2016
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
A survey on Image Data Augmentation for Deep Learning2019
Convolutional Neural Networks for Sentence Classification2014
Enriching Word Vectors with Subword Information2017
A Survey of Large Language Models2026
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)2018
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation2014
Hierarchical Attention Networks for Document Classification2016
Neural Architectures for Named Entity Recognition2016
Knowledge Graph Embedding by Translating on Hyperplanes2014
Learning Entity and Relation Embeddings for Knowledge Graph Completion2015
Knowledge Graph Embedding: A Survey of Approaches and Applications2017
CatBoost for big data: an interdisciplinary review2020
Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors2014
REALM: Retrieval-Augmented Language Model Pre-Training2020
A Unifying Review of Deep and Shallow Anomaly Detection2021
Siamese Recurrent Architectures for Learning Sentence Similarity2016
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation2018