Transformers in Time Series: A Survey
Трансформеры во временных рядах: обзор
2023-08-01
SCID: 54.1/m5xw3g92
Discuss with AI
Transformersanomaly detectionforecastinglong-range dependenciestime series
Figures from the paper
Abstract (AI)
Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance.
Key Findings
1
Empirical analyses performed include robust analysis, model size analysis, and seasonal-trend decomposition to study Transformer performance on time series.
2
The paper systematically reviews adaptations and structural modifications of Transformer architectures specifically made to address time series challenges.
3
The review identifies both strengths and limitations of Transformers in time series and provides suggested future research directions and guidance.
4
Time series Transformers are categorized and analyzed across key application tasks: forecasting, anomaly detection, and classification.
5
Transformers' ability to capture long-range dependencies and interactions makes them especially attractive and effective for time series modeling.
Research Object
Transformer models applied to time series data
Research Subject
Architectural adaptations, application categorizations (forecasting, anomaly detection, classification), empirical performance analyses (robustness, model size, seasonal-trend decomposition), strengths and limitations of Transformers for time series modeling
Publication Details
Publication Date
2023-08-01
Journal
Publisher
ISSN
Cited by
1979
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai15
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale2020
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting2021
Are Transformers Effective for Time Series Forecasting?2023
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
Self-Attention with Relative Position Representations2018
Pre-Trained Image Processing Transformer2021
Time-series forecasting with deep learning: a survey2021
A Review on Outlier/Anomaly Detection in Time Series Data2021
Efficient Transformers: A Survey2022
Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines2021
Learning Graph Structures With Transformer for Multivariate Time-Series Anomaly Detection in IoT2021
Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (Short Paper)2023
Time Series Anomaly Detection Leveraging MSE Feedback with AutoEncoder and RNN2024
Qualitative Spatial Reasoning over Questions (Short Paper)2022