Video Swin Transformer

Video Swin Transformer
Yue Cao, Zheng Zhang, Stephen Ching-Feng Lin, Han Hu, Zheng Zhang, Ze Liu, Yixuan Wei, Ning Jia, Jia Ning
2022-06-01

Kinetics-400Kinetics-600Something-Something v2Swin TransformerVideo Swin Transformeraction recognitionlocality inductive biaspre-trained image modelsspatial-temporal localityspeed-accuracy trade-off
The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-l accuracy on Kinetics-400 and 85.9 top-l accuracy on Kinetics-600 with ~20× less pre-training data and ~3× smaller model size) and temporal modeling (69.6 top-l accuracy on Something-Something v2).
1
Achieves 85.9% top-1 on Kinetics-600 using approximately 20× less pre-training data and about 3× smaller model size.
2
Achieves state-of-the-art accuracy across video recognition benchmarks, including 84.9% top-1 on Kinetics-400.
3
Achieves strong temporal modeling performance with 69.6% top-1 accuracy on Something-Something v2.
4
Adapted Swin Transformer from images to videos realizes locality while leveraging pre-trained image models.
5
Introducing locality inductive bias into video Transformers yields a better speed-accuracy trade-off than prior models that use global self-attention.

Video Swin Transformer architecture for video recognition

Effect of introducing locality inductive bias (local windowed self-attention adapted from Swin Transformer) on speed-accuracy trade-off and temporal/spatial modeling performance in video recognition benchmarks

Publication Details
Publication Date
2022-06-01
Journal
Publisher
ISSN
Cited by
2283
Access Type
Author Information
Authors
Yue Cao
Zheng Zhang
Stephen Ching-Feng Lin
Han Hu
Zheng Zhang
Ze Liu
Yixuan Wei
Ning Jia
Jia Ning
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%