Meta-learning and Data Augmentation for Stress Testing Forecasting Models

Meta-learning and Data Augmentation for Stress Testing Forecasting Models⋆ Ricardo In´acio1,2[0009−0008−6435−5245], Vitor Cerqueira1,2[0000−0002−9694−8423], Mar´ılia Barandas3[0000−0002−9445−4809] and Carlos Soares1,2,3[0000−0003−4549−8917] 1Faculdade de Engenharia da Universidade do Porto, Porto, Portugal [email protected] {csoares,vcerqueira}@fe.up.pt 2Laboratory for Artificial Intelligence and Computer Science (LIACC), Portugal 3Fraunhofer Portugal AICOS, Portugal [email protected] Abstract. The effectiveness of univariate forecasting models is often hampered by conditions that cause them stress. A model is considered to be under stress if it shows a negative behaviour, such as higher-thanusual errors or increased uncertainty. Understanding the factors that cause stress to forecasting models is important to improve their reliability, transparency, and utility. This paper addresses this problem by contributing with a novel framework called MAST (Meta-learning and data Augmentation for Stress Testing). The proposed approach aims to model and characterize stress in univariate time series forecasting models, focusing on conditions where they exhibit large errors. In particular, MAST is a meta-learning approach that predicts the probability that a given model will perform poorly on a given time series based on a set of statistical time series features. MAST also encompasses a novel data augmentation technique based on oversampling to improve the metadata concerning stress. We conducted experiments using three benchmark datasets that contain a total of 49.794 time series to validate the performance of MAST. The results suggest that the proposed approach is able to identify conditions that lead to large errors. The method and experiments are publicly available in a repository. Keywords: Time series ·Forecasting ·Stress testing ·Meta-learning ·Data augmentation ⋆This work was partially funded by projects AISym4Med (101095387) supported by Horizon Europe Cluster 1: Health, ConnectedHealth (n.º46858), supported by Competitiveness and Internationalisation Operational Programme (POCI) and Lisbon Regional Operational Programme (LISBOA 2020), under the PORTUGAL 2020 Partnership Agreement, through the European Regional Development Fund (ERDF) and NextGenAI - Center for Responsible AI (2022-C05i0102-02), supported by IAPMEI, and also by FCT plurianual funding for 2020-2023 of LIACC (UIDB/00027/2020 UIDP/00027/2020) arXiv:2406.17008v1 [cs.LG] 24 Jun 2024 2 R. In´acio et al. 1 Introduction Time series forecasting is a relevant problem in various application domains, such as industry, healthcare, or finance. Accurate forecasts help reduce uncertainty about the future and foster a data-driven decision-making within organizations [18]. However, the effective application of forecasting models is sometimes hampered by stress conditions that reduce their performance and reliability. In machine learning, stress denotes conditions where a model exhibits a negative behaviour, such as higher-than-usual errors or increased uncertainty. Stress factors result in unreliable predictions, causing end-users to distrust models. In the context of forecasting, stress can stem from factors such as i) data difficulty problems, for instance out-of-distribution samples or missing data, or ii) an inadequate inductive bias where the model fails to capture relevant patterns in specific time series. Stress testing aims to evaluate the reliability of machine learning approaches by identifying, modelling, or simulating challenging scenarios. This process provides actionable insights for improving models and decision-making by end-users. Figure 1 illustrates the forecasting performance of a model across several univariate time series. It shows the distribution of SMAPE (symmetric mean absolute percentage error) incurred by a forecasting model across 1428 time series. The distribution is right-skewed, where the long-tail denotes time series where the model is under stress and exhibits larger errors. We will provide further details on this figure in Section 4. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 SMAPE 0 100 200 300 400 500 600 Frequency Distribution of SMAPE Fig. 1. Forecasting performance of a model across several univariate time series according to SMAPE. Meta-learning and Data Augmentation for Stress Testing Forecasting Models 3 The goal of this paper is to stress test a given forecasting model. Particularly, identify the conditions that lead to a large error in a given time series. Understanding these factors is an essential step to improve the understanding of the inner workings of forecasting models. Ultimately, this knowledge promotes a more reliable and transparent use of forecasting systems by enabling end-users to adapt their decisions under the identified stress conditions. We accomplish this goal by proposing a new approach for modelling stress in univariate time series forecasting models called MAST (short for Meta-learning and data Augmentation for Stress Testing). Meta-learning involves using machine learning to improve the performance of machine learning models, with tasks such as algorithm selection [6] or dynamic model combination [9]. MAST uses meta-learning to model the occurrence of large errors based on structural properties of time series (e.g., seasonal strength). These structural properties are computed using time series feature extraction methods [28, 4] and include statistics such as the average value or standard deviation. Large errors (relative to the typical performance of a model) represent a small fraction of the dataset (c.f. Figure 1). A limited number of samples complicates the process of modelling these instances. In this context, we frame the above meta-learning problem as an imbalanced learning task. Imbalanced domain learning is the field of machine learning devoted to modelling datasets where the target distribution is skewed, which means most observations belong to a single class. However, the minority class is often the most relevant, which, in our cases, represents the cases where a model incurs a large error. Resampling techniques are typically used to tackle class imbalance in binary classification problems. We explore resampling methods (e.g., SMOTE [10]) to generate synthetic samples concerning large errors. We hypothesize that the augmented dataset improves the performance of the meta-learning model. We validate the proposed approach in three benchmark datasets that comprise a total of 49.794 time series. The experiments suggest that our method is able to identify the conditions that lead to large errors by a forecasting model. Using Shapley values [23], we provide concrete examples about the conditions where a given model performs poorly. The proposed method and experiments are available online 4. In summary, the contributions of this paper are the following: –A novel methodology for stress testing forecasting models based on metalearning and data augmentation called MAST; –A data augmentation technique based on resampling strategies that creates synthetic observations concerning large forecasting errors; –A set of extensive experiments that validate the proposed method in benchmark datasets, including usage examples based on Shapley values. 4https://github.com/ricardoinaciopt/mast 4 R. In´acio et al. 2 Literature Review This section lays out the foundations of our research. We start by defining the problem of univariate time series forecasting (Section 2.1), focusing on machine learning approaches trained in a global manner. Then, we describe several methods for data augmentation in Section 2.2. We highlight practices designed for time series and approaches tailored for tackling class imbalance. We also list several frameworks for time series feature extraction (Section 2.3). Finally, we briefly overview the relevance of stress testing and its scope within responsible artificial intelligence (Section 2.4). 2.1 Problem definition Univariate time series forecasting. A univariate time series is a time-ordered sequence of observations. It can be defined as Y={y1, y2, . . . , yt}, where yi∈R is the numeric value of Ycollected at time iand tis the length of Y. In this work, we aim at predicting the value of the next hobservations of a given univariate time series, yt+1, . . . , yt+h, based on historical data. We address the forecasting task using machine learning approaches and an auto-regressive methodology. We use sliding windows based on time delay embedding to reconstruct a time series Yfor supervised learning. Each observation of a time series is modelled as a function of its past plags [5]. Essentially, we build a dataset D={< Xi, yi>}t i=p+1 where yirepresents the i-th observation and Xi∈Rpis the i-th corresponding set of plags: Xi={yi−1, yi−2, . . . , yi−p}. The goal is to build a regression model fwith the form yi=f(Xi). Global Forecasting Models. We focus on forecasting problems that involve a collection of multiple univariate time series: Y={Y1, Y2, . . . , Yn}, where Yjis the j-th time series and nis the number of time series in the collection. Local forecasting methods build an individual model for each time series in a collection Y. Classical techniques, such as ARIMA [17], tend to follow this approach. In contrast, recent machine learning methods applied to forecasting tend to follow a global approach and build a model for a given collection using the historical data of all available time series. Global forecasting models have outperformed local ones in benchmark datasets and competitions [15]. Using several time series to train a forecasting model has been shown to lead to better forecasting performance [15, 8]. In practice, this is achieved by transforming each time series using the approach described in Section 2.1. Then, a global model is trained on the concatenation of the resulting datasets D= {D1, D2, . . . , Dn}, where Djis the dataset built for time series Yj. A global forecasting approach often requires appropriate data preprocessing steps such as transformations, which we detail in Section 4. 2.2 Data Augmentation Data augmentation techniques are useful when a large or diverse enough dataset is not available for training a model [3]. For time series, there are various methods Meta-learning and Data Augmentation for Stress Testing Forecasting Models 5 to create synthetic time series. These range from simple methods such as jittering or flipping the original time series [31], to more sophisticated approaches such as seasonal decomposition [12] or generative models [20]. Most time series data augmentation techniques aim to create synthetic time series that preserve the properties of the original ones. There are also approaches that attempt to create time series samples in parts of the data space with limited examples. For instance, Cerqueira et al. [7] leverage resampling techniques such as SMOTE [10] to increase the sample size of particular time series within a collection Y.SMOTE randomly selects instances from the minority class (in the case of Cerqueira et al. [7], individual time series in a collection), and interpolates them based on their nearest neighbours to create new samples. While resampling methods are typically used to mitigate class imbalance issues in classification, Cerqueira et al. [7] apply them in forecasting problems to improve the global-local trade-off [19]. Overall, the ultimate goal of existing time series data augmentation techniques is to obtain an enriched dataset that leads to better forecasting performance [8]. In this work, we leverage these methods to improve meta-learning and stress testing a forecasting model. We explore oversampling strategies, to generate synthetic samples in observations where a forecasting model shows a large error, increasing the overall representation of such instances. We focus specifically on SMOTE and ADASYN as they are well-established methods in the literature of imbalanced domain learning [27]. 2.3 Time Series Feature Extraction Extracting features from time series has been shown to improve performance in different tasks, such as forecasting [21] or classification [4]. Several frameworks have been proposed for this task. Examples include the works by Prudˆencio and Ludermir [29], Lemke et al. [21], Montero-Manso et al. [28], Barandas et al. [4], or Christ et al. [11]. In the context of time series forecasting, feature-based modelling is typically used in meta-learning approaches for model selection and combination. This process involves summarising each time series in a collection into a set of features. Then, a metamodel is trained to predict the most appropriate learning algorithm based on these features. For a new query time series, this metamodel predicts which model, or set of models, should be used for forecasting. In the benchmark M4 forecasting competition, Montero-Manso et al. [28] applied this approach and ranked second place, outperforming several classical and machine learning approaches. Besides forecasting, time series feature extraction is a common approach for time series classification tasks. In these, the label associated with a given time series is modelled based on a set of features [4]. In this work, we use the tsfeatures framework for time series feature extraction [28]. As mentioned above, Montero-Manso et al. [28] developed a competitive feature-based forecasting approach using this framework. Overall, tsfeatures contains a set of 42 features, including entropy, linearity, lumpiness, or trend. We 6 R. In´acio et al. refer the reader to the work by Montero-Manso et al. [28] for a comprehensive description of these. 2.4 Stress Testing Building reliable and robust predictive models is a key step towards the responsible use of machine learning [14]. This task is challenging because real-world data is susceptible to stress conditions that undermine the performance of models. These conditions can be caused by issues such as out-of-distribution data, missing values, or an inadequate inductive bias concerning some parts of the input space. Identifying the conditions in which a model is under stress and is prone to under-perform is a crucial aspect for supporting the adoption of machine learning systems, especially in sensitive domains such as healthcare. Stress testing is an emerging topic that addresses this problem [13]. The goal of stress testing is to evaluate the reliability of machine learning approaches by identifying, modelling and simulating challenging scenarios. Stress testing was recently pioneered in the context of computer vision in the work by Cunha et al. [13]. They propose an extension of generative adversarial networks called GASTeN (Generative Adversarial Stress Testing Network) that aims to create synthetic samples with two key properties: i) realistic and ii) challenging, where a given classifier serves a prediction with low confidence. Their framework enables the simulation of a diverse set of conditions where a model is not reliable, which provides useful insights for end-users. While they focus on generating samples that are challenging for a given model, we adopt a meta-learning approach to model and identify the weaknesses of a model. To our knowledge, this is the first work that explores meta-learning for stress testing. Similar to us, Roˇzanec et al. [30] also address the problem of identifying poor forecasts made by a global model, leveraging anomaly detection methods to this effect. However, they focus on managing the global-local trade-off and improving forecasting performance. 3 Stress Testing Forecasting Models using Meta-learning This section formalizes MAST, the proposed approach for modelling the stress based on meta-learning. We address univariate time series forecasting problems using datasets involving multiple time series. We tackle this task using an autoregressive approach, as defined in Section 2.1. In this work, MAST involves using meta-learning for modelling the performance of a forecasting model based on time series features and data augmentation. The workflow of MAST, which is illustrated in Figure 2, is split into two stages: i) a development stage and; ii) an inference stage. Meta-learning and Data Augmentation for Stress Testing Forecasting Models 7 Fig. 2. Workflow behind MAST, which is split into a development stage and an inference stage. In the development stage, we conduct performance estimation, feature extraction, data augmentation, and meta-learning. Then, the resulting metamodel is applied during the inference stage to predict whether a forecasting model will incur a large error. 3.1 Development stage In the development stage, we aim at building a metamodel that predicts the probability that a time series will induce a large error on a forecasting model. This goal is split into four steps: 1. Performance estimation: Conducting a training plus evaluation cycle to estimate the performance of a forecasting model using a collection of time series; 2. Feature extraction: Using the training data from step 1., summarize each time series into a set of features; 3. Data augmentation: Create synthetic time series samples to enrich the metadataset concerning large errors; 4. Meta-learning: Build a metamodel that predicts the performance of a forecasting model based on time series features. Performance estimation. We conduct a procedure to estimate the performance of a forecasting model on each time series in the available collection Y. Let Ddenote the dataset built from Y, which is prepared for auto-regression. We split Dinto training and testing sets, Dtrain and Dtest, and fit a model fusing Dtrain. The performance of fis evaluated on Dtest, resulting in error scores according to SMAPE metric. This leads to a set of performance estimates 8 R. In´acio et al. E={e1, e2, . . . , en}, where eiis the performance estimate of ffor time series Yi. Feature extraction and meta-dataset. We summarise the training set of each time series Y∈ Y using feature extraction. Let gdenote a feature extraction function that transforms a time series Yiinto a set of features Zi= {zi,1, zi,2, . . . , zi,m}, where mis the number of features and zi,k is the k-th feature of Yi. This transformation is written as Zi=g(Yi). There are several approaches in the literature for the function g, including TSFEL [4], tsfeatures [28], or catch22 [22]. These involve several statistical operations, such as the mean or standard deviation of the input time series. While the methodology is applicable to any of these approaches, we focus on tsfeatures [28], as we mentioned in Section 2.3. The results from the performance estimation, E, and feature extraction Z are concatenated into a meta-dataset D′={< Zi, ei>}n i=1. Each element in D′ includes the features of a given time series, along with the error score of a given forecasting model fon that data. Note that the features are extracted using the training data, while the error is computed on the test data. This process preserves the temporal order of data and avoids data leakage. Our goal is to use the meta-dataset to develop a metamodel f′that classifies where the forecasting model fwill incur a large error in a given time series. We define a binary variable for large errors based on the error scores E, which is determined as follows: bi=(1 if ei> τ, 0 otherwise.(1) For a given time series, bitakes the value of 1 if eiis above τ, which represents the threshold above which an error is considered large. The threshold τcan be defined by domain expertise or in a data-driven way, e.g. using percentiles. In effect, the meta-dataset D′={< Zi, bi>}n i=1 involves a binary classification task. The goal is to model upcoming large errors (b) based on time series features (Z) extracted from historical data. Data augmentation. We work under the assumption that large errors are rare. Thus, the distribution of bis imbalanced, with b= 1 being the minority class. The meta-class imbalance poses an additional challenge for learning an adequate model for large errors. We tackle this issue by framing the task as an imbalanced domain learning problem [27], and use resampling approaches for data augmentation. We employ an oversampling algorithm (e.g. SMOTE [10]) to create new (Zi,synthetic,bi,synthetic) samples concerning large errors (b= 1). The idea is to simulate conditions that lead to large errors based on the existing data. Meta-learning The created synthetic samples are represented in a dataset D′ synthetic, which is concatenated with the original dataset D′to augment it. Meta-learning and Data Augmentation for Stress Testing Forecasting Models 9 Subsequently, a metamodel f′is trained based on the combined meta-dataset D′∪ D′ synthetic. The output of the metamodel ˆ bi=f′(Zi) provides information about the probability that the forecasting model fwill incur a large error on Yi. 3.2 Inference stage In the inference stage, we use the metamodel f′to predict the probability that fwill incur a large error for a query time series Yquery. We conduct this process as follows: 1. Feature extraction: Transform Yquery into a set of features Zquery 2. Compute the probability of a large error using the metamodel: ˆ b=f′(Zquery) The information about the probability of a large error can be used in various ways, for example deferring the forecasts by not accepting the predictions made by f. Another possibility is to use the metamodel output for a descriptive analysis by characterizing the feature space in conditions with high probability of a large error. 4 Experiments We carried out extensive experiments to evaluate the proposed method. The experiments focus on the following research questions: – RQ1: How does the base forecasting model compare with benchmark approaches? – RQ2: Is the metamodel able to predict large errors by a given forecasting model? – RQ3: What is the impact of data augmentation using resampling in the performance of the metamodel? – RQ4: How does the performance of the metamodel vary with different values of τ? Besides these, we also present an example of how the results of the metamodel can be used to understand the conditions where a forecasting model is expected to perform poorly. 4.1 Data We use the following datasets in the experiments: M3 Monthly [24], M4 Monthly [25], and Tourism Monthly [2]. These datasets were part of different forecasting competitions and represent long-standing univariate time series forecasting benchmarks. In the interest of conciseness, we focus on time series with a monthly sampling frequency. Table 1 provides a brief summary of the data. Both the forecasting horizon (h) and the number of lags (p) are set to 12, which represents one seasonal cycle (one year) of data. 16 R. In´acio et al. 26. Makridakis, S., Spiliotis, E., Assimakopoulos, V.: M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38(4), 1346– 1364 (2022) 27. Moniz, N., Cerqueira, V.: Automated imbalanced classification via meta-learning. Expert Systems with Applications 178, 115011 (2021) 28. Montero-Manso, P., Athanasopoulos, G., Hyndman, R.J., Talagala, T.S.: Fforma: Feature-based forecast model averaging. International Journal of Forecasting 36(1), 86–92 (2020) 29. Prudˆencio, R.B., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137 (2004) 30. Roˇzanec, J., Trajkova, E., Kenda, K., Fortuna, B., Mladeni´c, D.: Explaining bad forecasts in global time series models. Applied Sciences 11(19) (2021) 31. Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., Xu, H.: Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002.12478 (2020)