Transformer in Transformer
Transformer in Transformer
2021-02-27
SCID: 54.1/spmdfd7e
Discuss with AI
ImageNet top-1 accuracyTransformer iN Transformer (TNT)local patch attentionvisual transformersvisual words and visual sentences
Figures from the paper
Abstract (AI)
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4$\times$4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.
Key Findings
1
Achieves 81.5% top-1 accuracy on ImageNet, about 1.7% higher than state-of-the-art visual transformer with similar computational cost.
2
Aggregates features of words and sentences to enhance representation ability with negligible additional computational cost for inner-patch attention.
3
Divides standard local patches (e.g., 16×16) into smaller sub-patches (e.g., 4×4) so that 'words' attention is computed inside each 'sentence'.
4
Proposes Transformer in Transformer (TNT) architecture that models attention both between local patches (sentences) and within patches (words).
5
Provides PyTorch and MindSpore implementations publicly available in the referenced repositories.
Research Object
Transformer iN Transformer (TNT) visual transformer architecture for image representation
Research Subject
Modeling and aggregating multi-scale intra-patch (word) and inter-patch (sentence) attention to improve image feature representation and classification performance
Publication Details
Publication Date
2021-02-27
Journal
Publisher
ISSN
Cited by
964
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai9
ImageNet classification with deep convolutional neural networks2017
Rethinking the Inception Architecture for Computer Vision2016
PyTorch: An Imperative Style, High-Performance Deep Learning Library2019
Learning Multiple Layers of Features from Tiny Images2024
Non-local Neural Networks2018
mixup: Beyond Empirical Risk Minimization2017
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers2021
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
Pre-Trained Image Processing Transformer2021
Cited by10
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
Attention mechanisms in computer vision: A survey2022
PVT v2: Improved baselines with pyramid vision transformer2022
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification2021
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows2022
Vision Transformer with Deformable Attention2022
CMT: Convolutional Neural Networks Meet Vision Transformers2022
Review of large vision models and visual prompt engineering2023
Large Language Models: A Structured Taxonomy and Review of Challenges, Limitations, Solutions, and Future Directions2025