Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Мультимодальное компактное билинейное объединение для ответов на визуальные вопросы и визуального заземления
2016-01-01
SCID: 54.1/9acsqfsa
Discuss with AI
Visual7W datasetmultimodal compact bilinear poolingmultimodal feature fusionvisual groundingvisual question answering
Figures from the paper
Abstract (AI)
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years.However, tasks such as visual question answering require combining these vector representations with each other.Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors.As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features.We extensively evaluate MCB on the visual question answering and grounding tasks.We consistently show the benefit of MCB over ablations without MCB.For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation.This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
Key Findings
1
MCB consistently improves performance over ablated models without multimodal compact bilinear pooling on visual question answering and visual grounding tasks.
2
The MCB-based model achieves state-of-the-art performance on the Visual7W dataset and the VQA challenge.
3
The method addresses the infeasible dimensionality of explicit outer products while retaining richer multimodal interactions than element-wise operations or concatenation.
4
The paper proposes Multimodal Compact Bilinear pooling (MCB) to efficiently approximate expressive outer-product interactions between visual and textual feature vectors.
5
The proposed visual question answering architecture applies MCB twice: for spatial attention prediction and for fusing attended visual features with question features.
Research Object
multimodal visual question answering and visual grounding systems
Research Subject
the effectiveness of Multimodal Compact Bilinear pooling for expressive combination of visual and textual representations
Publication Details
Publication Date
2016-01-01
Journal
Publisher
ISSN
Cited by
1696
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
Deep Residual Learning for Image Recognition2016
Very Deep Convolutional Networks for Large-Scale Image Recognition2014
Adam: A Method for Stochastic Optimization2014
ImageNet: A large-scale hierarchical image database2009
Sequence to Sequence Learning with Neural Networks2014
Bilinear CNN Models for Fine-Grained Visual Recognition2015
Data Driven VRP: A Neural Network Model to Learn Hidden Preferences for VRP2021