Kinetics-400Kinetics-600Something-Something v2Swin TransformerVideo Swin Transformeraction recognitionlocality inductive biaspre-trained image modelsspatial-temporal localityspeed-accuracy trade-off
Figures from the paper
Abstract (AI)
The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-l accuracy on Kinetics-400 and 85.9 top-l accuracy on Kinetics-600 with ~20× less pre-training data and ~3× smaller model size) and temporal modeling (69.6 top-l accuracy on Something-Something v2).
Key Findings
1
Achieves 85.9% top-1 on Kinetics-600 using approximately 20× less pre-training data and about 3× smaller model size.
2
Achieves state-of-the-art accuracy across video recognition benchmarks, including 84.9% top-1 on Kinetics-400.
3
Achieves strong temporal modeling performance with 69.6% top-1 accuracy on Something-Something v2.
4
Adapted Swin Transformer from images to videos realizes locality while leveraging pre-trained image models.
5
Introducing locality inductive bias into video Transformers yields a better speed-accuracy trade-off than prior models that use global self-attention.
Research Object
Video Swin Transformer architecture for video recognition
Research Subject
Effect of introducing locality inductive bias (local windowed self-attention adapted from Swin Transformer) on speed-accuracy trade-off and temporal/spatial modeling performance in video recognition benchmarks
Publication Details
Publication Date
2022-06-01
Journal
Publisher
ISSN
Cited by
2283
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai11
Deep Residual Learning for Image Recognition2016
ImageNet classification with deep convolutional neural networks2017
Very Deep Convolutional Networks for Large-Scale Image Recognition2014
Adam: A Method for Stochastic Optimization2014
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
Gradient-based learning applied to document recognition1998
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale2020
Non-local Neural Networks2018
Randaugment: Practical automated data augmentation with a reduced search space2020