Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Не просто предполагайте; смотрите и отвечайте: преодоление априорных представлений в визуальном ответе на вопросы
2018-06-01
SCID: 54.1/zhm4eqwp
Discuss with AI
Changing PriorsGrounded Visual Question AnsweringStacked Attention NetworksVQA-CP datasetsVisual Question Answering
Figures from the paper
Abstract (AI)
A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model - Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.
Key Findings
1
Existing VQA models suffer significant performance degradation on VQA-CP compared with the original VQA evaluation setting.
2
GVQA disentangles visual concept recognition from question-specific answer-space identification, reducing reliance on training-set priors and improving cross-distribution generalization.
3
GVQA significantly outperforms SAN on both VQA-CP versions and surpasses MCB in several cases, while offering greater transparency and complementary strengths on standard VQA datasets.
4
VQA models rely heavily on superficial answer priors, revealing inadequate image grounding when evaluated under changing answer distributions.
5
VQA-CP v1 and VQA-CP v2 introduce train/test splits with different answer priors for each question type, directly testing robustness to prior shifts.
Research Object
Visual Question Answering (VQA) models evaluated on VQA-CP and VQA datasets under changing answer-prior distributions
Research Subject
The models’ reliance on answer priors versus image grounding, including cross-distribution generalization and interpretability
Publication Details
Publication Date
2018-06-01
Journal
Publisher
ISSN
Cited by
623
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest