Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

Не просто предполагайте; смотрите и отвечайте: преодоление априорных представлений в визуальном ответе на вопросы
Aniruddha Kembhavi, Dhruv Batra, Devi Parikh, Aishwarya Agrawal
2018-06-01

Changing PriorsGrounded Visual Question AnsweringStacked Attention NetworksVQA-CP datasetsVisual Question Answering
A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model - Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.
1
Existing VQA models suffer significant performance degradation on VQA-CP compared with the original VQA evaluation setting.
2
GVQA disentangles visual concept recognition from question-specific answer-space identification, reducing reliance on training-set priors and improving cross-distribution generalization.
3
GVQA significantly outperforms SAN on both VQA-CP versions and surpasses MCB in several cases, while offering greater transparency and complementary strengths on standard VQA datasets.
4
VQA models rely heavily on superficial answer priors, revealing inadequate image grounding when evaluated under changing answer distributions.
5
VQA-CP v1 and VQA-CP v2 introduce train/test splits with different answer priors for each question type, directly testing robustness to prior shifts.

Visual Question Answering (VQA) models evaluated on VQA-CP and VQA datasets under changing answer-prior distributions

The models’ reliance on answer priors versus image grounding, including cross-distribution generalization and interpretability

Publication Details
Publication Date
2018-06-01
Journal
Publisher
ISSN
Cited by
623
Access Type
Author Information
Authors
Aniruddha Kembhavi
Dhruv Batra
Devi Parikh
Aishwarya Agrawal
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%