A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering

Симметричная схема плотного поиска с двойным кодированием для визуального ответа на вопросы, требующие привлечения знаний
Hamed Zamani, Alireza Salemi, Juan Altmayer Pizzorno
2023-07-18

OK-VQAdense retrievaliterative knowledge distillationknowledge-intensive visual question answeringsymmetric dual encoding
Knowledge-Intensive Visual Question Answering (KI-VQA) refers to answering a question about an image whose answer does not lie in the image. This paper presents a new pipeline for KI-VQA tasks, consisting of a retriever and a reader. First, we introduce DEDR, a symmetric dual encoding dense retrieval framework in which documents and queries are encoded into a shared embedding space using uni-modal (textual) and multi-modal encoders. We introduce an iterative knowledge distillation approach that bridges the gap between the representation spaces in these two encoders. Extensive evaluation on two well-established KI-VQA datasets, i.e., OK-VQA and FVQA, suggests that DEDR outperforms state-of-the-art baselines by 11.6% and 30.9% on OK-VQA and FVQA, respectively.
1
An iterative knowledge distillation method bridges representation-space differences between the unimodal textual and multimodal encoders.
2
DEDR outperforms state-of-the-art baselines by 11.6% on OK-VQA and 30.9% on FVQA.
3
DEDR uses symmetric dual encoding, mapping documents and image-based queries into a shared embedding space through textual and multimodal encoders.
4
Experiments on OK-VQA and FVQA demonstrate the effectiveness of the proposed retrieval framework for KI-VQA.
5
The paper introduces a retriever-reader pipeline specifically designed for knowledge-intensive visual question answering.

Knowledge-intensive visual question answering (KI-VQA) tasks involving questions about images whose answers are not contained in the images

Retrieval and answering performance in KI-VQA, particularly the effectiveness of symmetric dual-encoding dense retrieval and iterative knowledge distillation for aligning textual and multimodal representations

Publication Details
Publication Date
2023-07-18
Journal
Publisher
ISSN
Cited by
27
Access Type
Author Information
Authors
Hamed Zamani
Alireza Salemi
Juan Altmayer Pizzorno
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%