Flamingo: a Visual Language Model for Few-Shot Learning

Flamingo: визуальная языковая модель для обучения по малому числу примеров
Oriol Vinyals, Andrew Zisserman, Jacob Menick, Serkan Cabi, Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Karen Simonyan, Gong, Zhitao, Brock, Andrew, Alayrac, Jean-Baptiste, Barr, Iain, Ricardo Barreira
2022-04-29

Few-shot learningIn-context learningMultimodal web corporaVisual language modelsVisual question answering
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
1
A single Flamingo model achieves state-of-the-art few-shot performance on open-ended, captioning, and multiple-choice visual question-answering tasks through task-specific prompting.
2
Flamingo is a family of visual language models designed to adapt rapidly to novel multimodal tasks using only a few annotated examples.
3
Flamingo outperforms models fine-tuned on thousands of times more task-specific data across numerous benchmarks.
4
The architecture bridges pretrained vision-only and language-only models and processes arbitrarily interleaved visual and textual sequences, including images and videos.
5
Training on large-scale multimodal web corpora enables Flamingo to perform in-context few-shot learning across diverse vision-language tasks.

Flamingo family of Visual Language Models (VLMs)

In-context few-shot adaptation and performance across image- and video-based vision-language tasks

Publication Details
Publication Date
2022-04-29
Journal
Publisher
ISSN
Cited by
1555
Access Type
Author Information
Authors
Oriol Vinyals
Andrew Zisserman
Jacob Menick
Serkan Cabi
Jean-Baptiste Alayrac
Jeff Donahue
Pauline Luc
Antoine Miech
Iain Barr
Yana Hasson
Karel Lenc
Arthur Mensch
Katie Millican
Malcolm Reynolds
Roman Ring
Eliza Rutherford
Tengda Han
Zhitao Gong
Sina Samangooei
Marianne Monteiro
Sebastian Borgeaud
Andrew Brock
Aida Nematzadeh
Sahand Sharifzadeh
Mikołaj Bińkowski
Ricardo Barreira
Karen Simonyan
Gong, Zhitao
Brock, Andrew
Alayrac, Jean-Baptiste
Barr, Iain
Ricardo Barreira
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%