Enriching Word Vectors with Subword Information

Обогащение векторов слов информацией о субсловах
Tomáš Mikolov, Édouard Grave, Armand Joulin, Piotr Bojanowski
2017-12-01

character n-gramsmorphological word representationsskipgram modelsubword informationword similarity and analogy tasks
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.
1
Across nine languages, the representations are evaluated on word similarity and analogy tasks.
2
Character n-gram representations incorporate word morphology and better address large vocabularies and rare words than whole-word vectors.
3
Compared with recently proposed morphological representations, the method achieves state-of-the-art performance on these tasks.
4
The method trains quickly on large corpora and can generate representations for words absent from the training data.
5
The paper enriches skipgram word vectors by representing each word as the sum of vectors for its character n-grams.

Continuous word vector representations augmented with character n-gram (subword) components

the effectiveness of subword-based word representations for morphological modeling, including performance on word similarity and analogy tasks and representation of unseen words

Publication Details
Publication Date
2017-12-01
Journal
Publisher
ISSN
Cited by
11057
Access Type
Author Information
Authors
Tomáš Mikolov
Édouard Grave
Armand Joulin
Piotr Bojanowski
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%