Are Transformers Effective for Time Series Forecasting?
Эффективны ли трансформеры для прогнозирования временных рядов?
2023-06-26
SCID: 54.1/m237uv6e
Discuss with AI
LTSF-LinearTransformerslong-term time series forecastingpositional encodingself-attention temporal information loss
Figures from the paper
Abstract (AI)
Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss. To validate our claim, we introduce a set of embarrassingly simple one-layer linear models named LTSF-Linear for comparison. Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin. Moreover, we conduct comprehensive empirical studies to explore the impacts of various design elements of LTSF models on their temporal relation extraction capability. We hope this surprising finding opens up new research directions for the LTSF task. We also advocate revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future.
Key Findings
1
A set of simple one-layer linear models (LTSF-Linear) was introduced as a baseline for long-term time series forecasting.
2
Comprehensive empirical studies identify that various design elements of LTSF models affect their ability to extract temporal relations.
3
On nine real-life datasets, LTSF-Linear outperforms existing sophisticated Transformer-based LTSF models in all cases, often by a large margin.
4
The results call into question the validity of Transformer-based solutions for LTSF and suggest revisiting their use for other time series tasks (e.g., anomaly detection).
5
Transformer-based LTSF models can lose temporal information due to the permutation-invariant self-attention mechanism, despite positional encodings and token embeddings.
Research Object
Long-term time series forecasting (LTSF) models and datasets
Research Subject
Effectiveness of Transformer-based models versus simple one-layer linear models (LTSF-Linear) at extracting temporal relations and achieving forecasting performance in long-term time series forecasting, including impacts of model design elements on temporal relation extraction
Publication Details
Publication Date
2023-06-26
Journal
Publisher
ISSN
Cited by
4242
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Greedy function approximation: A gradient boosting machine.2001
Neural Machine Translation by Jointly Learning to Align and Translate2014
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting2021
Transformers in Time Series: A Survey2023
Stock Price Prediction Using the ARIMA Model2014
Qualitative Spatial Reasoning over Questions (Short Paper)2022
Cited by4
Transformers in Time Series: A Survey2023
Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey2023
A comprehensive survey of deep learning for time series forecasting: architectural diversity and open challenges2025
An Enhanced Air Quality Forecasting Method Integrating Feature Mode Decomposition and Multilayer Plasticity Echo State Network2026