Scaling Vision Transformers
Масштабирование Vision Transformer
2022-06-01
SCID: 54.1/2ea79bq5
Discuss with AI
ImageNet top-1 accuracy 90.45%Vision Transformerdata scalingmodel scalingscaling laws
Figures from the paper
Abstract (AI)
Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class.
Key Findings
1
A ViT model with two billion parameters was successfully trained and achieved a new ImageNet state-of-the-art: 90.45% top-1 accuracy.
2
Refinements to ViT architecture and training reduced memory consumption and increased accuracy of resulting models.
3
The large ViT model demonstrates strong few-shot transfer, achieving 84.86% top-1 accuracy on ImageNet with only 10 examples per class.
4
Vision Transformers (ViT) scaling relationships between error rate, data, and compute are characterized by scaling ViT models and data both up and down.
Research Object
Vision Transformer (ViT) models and their scaling (model and dataset scale across parameter count and data size)
Research Subject
Relationships between error rate, data, and compute as ViT models and training data are scaled; architectural and training refinements affecting memory consumption and accuracy
Publication Details
Publication Date
2022-06-01
Journal
Publisher
ISSN
Cited by
1093
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai10
Deep Residual Learning for Image Recognition2016
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
ImageNet: A large-scale hierarchical image database2009
Learning Multiple Layers of Features from Tiny Images2024
Emerging Properties in Self-Supervised Vision Transformers2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
Scaling Laws for Neural Language Models2020
Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (Short Paper)2023
Qualitative Spatial Reasoning over Questions (Short Paper)2022
Cited by5
Scalable Diffusion Models with Transformers2023
Swin Transformer V2: Scaling Up Capacity and Resolution2022
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models2023
Multimodal Foundation Models: From Specialists to General-Purpose Assistants2024
GenTron: Diffusion Transformers for Image and Video Generation2024