A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering
Симметричная схема плотного поиска с двойным кодированием для визуального ответа на вопросы, требующие привлечения знаний
2023-07-18
SCID: 54.1/8x2stmdm
Discuss with AI
OK-VQAdense retrievaliterative knowledge distillationknowledge-intensive visual question answeringsymmetric dual encoding
Figures from the paper
Abstract (AI)
Knowledge-Intensive Visual Question Answering (KI-VQA) refers to answering a question about an image whose answer does not lie in the image. This paper presents a new pipeline for KI-VQA tasks, consisting of a retriever and a reader. First, we introduce DEDR, a symmetric dual encoding dense retrieval framework in which documents and queries are encoded into a shared embedding space using uni-modal (textual) and multi-modal encoders. We introduce an iterative knowledge distillation approach that bridges the gap between the representation spaces in these two encoders. Extensive evaluation on two well-established KI-VQA datasets, i.e., OK-VQA and FVQA, suggests that DEDR outperforms state-of-the-art baselines by 11.6% and 30.9% on OK-VQA and FVQA, respectively.
Key Findings
1
An iterative knowledge distillation method bridges representation-space differences between the unimodal textual and multimodal encoders.
2
DEDR outperforms state-of-the-art baselines by 11.6% on OK-VQA and 30.9% on FVQA.
3
DEDR uses symmetric dual encoding, mapping documents and image-based queries into a shared embedding space through textual and multimodal encoders.
4
Experiments on OK-VQA and FVQA demonstrate the effectiveness of the proposed retrieval framework for KI-VQA.
5
The paper introduces a retriever-reader pipeline specifically designed for knowledge-intensive visual question answering.
Research Object
Knowledge-intensive visual question answering (KI-VQA) tasks involving questions about images whose answers are not contained in the images
Research Subject
Retrieval and answering performance in KI-VQA, particularly the effectiveness of symmetric dual-encoding dense retrieval and iterative knowledge distillation for aligning textual and multimodal representations
Publication Details
Publication Date
2023-07-18
Journal
Publisher
ISSN
Cited by
27
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai4
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Dense Passage Retrieval for Open-Domain Question Answering2020
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021