Are Transformers Effective for Time Series Forecasting?

It’s true that a lot of progress has been made in this field with very well-designed models attempting to overcome the limitations of Vanilla Transformers in forecasting tasks.

One of these limitations is the quadratic time/complexity of the original self-attention. To deal with it, most models use specialized self-attention blocks.

For example, Informer [1] uses ProbSparse Attention to decrease complexity to O(LlogL), while FEDFormer [2] uses Frequency Enhanced Blocks and Frequency Enhanced Self-Attentions to perform representation learning in the frequency domain, ultimately reducing complexity to linear. My personal favorite is PyraFormer [3], which uses Hierarchical Self-Attention to describe temporal dependencies through multiresolutions (aka Temporal Hierarchies, see my other Medium Posts), reducing complexity to O(L).

Other limitations, including bias towards overly smooth solutions and rotationally invariant learning, are inherited from the limitations of deep learning models for tabular data in general. These are discussed in more detail in [4].

Still, the most important limitation is that self-attention is permutation-invariant (anti-order) while time series observations have strong temporal relationships [5]. For semantic applications like NLP, this is not a big issue as rearranging some words doesn’t greatly change the overall semantic meaning. However, for time series, even with multiple positional encoding techniques to preserve temporal information, it’s not enough. Positional encoding only encodes ordering information to some extent.

To address this limitation, several models use specialized architectures in order to preserve some of the temporal information. LogTrans [6] uses convolution self-attention to preserve local context, Non-Stationary-Transformer [7] uses De-Stationary-Attention to discover temporal dependencies through exploiting stationarity, AutoFormer [8] applies seasonal-trend decomposition, etc.

The questions that arise are the following:

• Is it worth adding all these complex blocks to deal with the limitations of the Vanilla Transformers?
• Are these models robust enough to work in a production environment with different real-world settings?
• Are the reported accuracy improvements in long-term forecasting a product of the adjusted self-attention or from other components (eg Direct Multistep Forecasting Strategy?)[5]

Another issue is model evaluation. Are these models properly benchmarked before claiming SOTA performance on some datasets? For example, according to [9], a non-trainable model with 0 parameters outperforms the AutoFormer transformer on multiple tasks. Then, in [5], researchers use very simple sets of linear models that implement Direct Multistep Forecasting to outperform most established transformers.

It’s clear that something is missing from the puzzle. Something that will unlock the full potential of Transformers for forecasting.

On the other hand, I’m certain that with careful tuning, these models can work and show improvements on the given forecasting task. We know for a fact that experts in the field of Operational Forecasting at Scale like Amazon use them effectively in production [10].

Additionally, during the International Symposium on Forecasting (ISF) on 2022, another group of experts, the Zalando forecasting team presented their Transformer model used in production with excellent results. Unfortunately, there’s no publication yet, but I’m pretty sure this will change soon :).

So are these models worth using for forecasting purposes?

For sure they are not the go-to model as they are for NLP tasks. There are multiple, established and battle-tested models that have been widely used in a production environment. These models are currently preferred over Transformers, and for a good reason. We still do not have enough evidence to suggest that Transformers offer significant benefits considering their implementation costs.

However, under the right circumstances, and especially given the appropriate expertise to effectively implement and maintain them, they can be effective.

To get started on the field [12] gives a nice survey of Transformers for time series forecasting and [13] is a great walkthrough on how to implement Transformers for forecasting using Hugging Face.

I’m looking forward to the future of Transformers and Time Series Forecasting.