Multimodal Distributional Semantics
Мультимодальная дистрибутивная семантика
2014-01-23
SCID: 54.1/t22hnr8g
Discuss with AI
distributional semantic modelsmultimodal distributional semanticsperceptual groundingtext-image integrationvisual words
Figures from the paper
Abstract (AI)
Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete visual words in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.
Key Findings
1
A flexible architecture integrates text-based and image-based distributional information into unified word representations.
2
Empirical tests show that the integrated multimodal model outperforms a purely text-based model on semantic evaluations.
3
The proposed approach automatically identifies discrete visual words in images and incorporates their co-occurrence with textual words.
4
Traditional distributional semantic models derive word meaning solely from textual co-occurrence, limiting their perceptual grounding.
5
Visual information provides semantic content that is somewhat complementary to information captured from text alone.
Research Object
Distributional semantic model extended with image-based visual-word co-occurrences
Research Subject
the improvement and complementary semantic information provided by perceptual grounding through integrated text- and image-based distributional representations
Publication Details
Publication Date
2014-01-23
Journal
Publisher
ISSN
Cited by
910
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai3
Cited by3
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge2017
Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors2014
The Discriminative Lexicon: A Unified Computational Model for the Lexicon and Lexical Processing in Comprehension and Production Grounded Not in (De)Composition but in Linear Discriminative Learning2019