City Research Online

Abstract

The prevalence of multivariate time series data across several disciplines fosters a demand and, subsequently, significant growth in the research and advancement of multivariate time series analysis. Drawing inspiration from a popular natural language processing model, the Transformer, we propose the Spatio-Temporal Transformer with Relative Embeddings (STTRE) to address multivariate time series forecasting. This work primarily focuses on developing a Transformer-based framework that can fully exploit the spatio-temporal nature of a multivariate time series by incorporating several of the Transformer’s key components, but with augmentations that allow them to excel in multivariate time series forecasting. Current Transformer-based models for multivariate time series often neglect the data’s spatial component(s) and utilize absolute position embeddings as their only means to detect the data’s temporal component(s), which we show is flawed for time series applications. The lack of emphasis on fully exploiting the spatiotemporality of the data can incur subpar results in terms of accuracy. We redesign relative position representations, which we rename to relative embeddings, to unveil a new method for detecting latent spatial, temporal, and spatiotemporal dependencies more effectively than previous Transformer-based models. We couple these relative embeddings with a restructuring of the Transformer’s primary sequence learning mechanism, multi-head attention, in a way that allows for full utilization of relative embeddings, thus achieving up to a 24% improvement in accuracy over other state-of-the-art multivariate time series models on a comprehensive selection of publicly available multivariate time series forecasting datasets.

Publication Type:	Article
Additional Information:	This is an open access article distributed under the terms of the Creative Commons CC-BY license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Publisher Keywords:	Multivariate Time Series, Transformer, Forecasting, Attention, Embeddings, Spatio-Temporal
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science T Technology
Departments:	School of Science & Technology > Engineering
SWORD Depositor:	Symplectic Administrator

Export

Downloads