Multimodal Distributional Semantics

Мультимодальная дистрибутивная семантика
Nam K. Tran, Marco Baroni, Elia Bruni, N. K. Tran
2014-01-23

distributional semantic modelsmultimodal distributional semanticsperceptual groundingtext-image integrationvisual words
Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete “visual words” in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.
1
A flexible architecture integrates text-based and image-based distributional information into unified word representations.
2
Empirical tests show that the integrated multimodal model outperforms a purely text-based model on semantic evaluations.
3
The proposed approach automatically identifies discrete visual words in images and incorporates their co-occurrence with textual words.
4
Traditional distributional semantic models derive word meaning solely from textual co-occurrence, limiting their perceptual grounding.
5
Visual information provides semantic content that is somewhat complementary to information captured from text alone.

Distributional semantic model extended with image-based visual-word co-occurrences

the improvement and complementary semantic information provided by perceptual grounding through integrated text- and image-based distributional representations

Publication Details
Publication Date
2014-01-23
Journal
Publisher
ISSN
Cited by
910
Access Type
Author Information
Authors
Nam K. Tran
Marco Baroni
Elia Bruni
N. K. Tran
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%