SimilarityTS: Toolkit for the evaluation of similarity for multivariate time series

Original software publication SimilarityTS: Toolkit for the evaluation of similarity for multivariate time series Alejandro Fernández-Montes a,∗ , Damián Fernández-Cerero a, Felipe Escalera-González a, Agnieszka Jakóbik c, Belén Bermejo b, Carlos Juiz b aDepartment of Computer Languages and Systems, University of Sevilla, 41012 Sevilla, Spain bDepartment of Computer Science, University of the Balearic Islands, 07122 Palma, Spain cCracow University of Technology, Warszawska st 24, 31-155 Cracow, Poland A R T I C L E I N F O Dataset link: https://github.com/alejandrofdez -us/similarity-ts-cli/tree/main/data_samples Keywords: Time series Similarity Multivariate Toolkit Python A B S T R A C T This paper presents SimilarityTS, a toolkit for the evaluation of similarity for multivariate time series. This software enables the comparison and assessment of the similarity between time series in a standard way, using the most relevant metrics and figures usually employed in the literature, such as the Kullback–Leibler divergence, Dynamic Time Warping, DTW paths, PCA, and t-SNE representations, among others. This toolkit provides a better way to compare research results and facilitates the repeatability of experiments. Code metadata Current code version 1.0.6 Permanent link to code/repository used for this code version https://github.com/ElsevierSoftwareX/SOFTX-D-23-00469 Permanent link to Reproducible Capsule Legal Code License MIT license Code versioning system used git Software code languages, tools, and services used Python Compilation requirements, operating environments & dependencies Python 3.9, dtaidistance, matplotlib, pandas, scikit-learn, tqdm, natsort, chardet If available Link to developer documentation/manual https://github.com/alejandrofdez-us/similarity-ts/blob/main/README.md Support email for questions [email protected] [email protected] [email protected] Software metadata Current software version 1.0.1. Permanent link to executables of this version https://github.com/alejandrofdez-us/similarity-ts-cli Permanent link to Reproducible Capsule Legal Software License MIT license Computing platforms/Operating Systems Linux, OS X, Microsoft Windows, Unix-like Installation requirements & dependencies similarity-ts, tqdm If available, link to user manual — if formally published include a reference to the publication in the reference list https://github.com/alejandrofdez-us/similarity-ts-cli/blob/main/README.md Support email for questions [email protected] [email protected] [email protected] ∗Corresponding author. E-mail address: [email protected] (Alejandro Fernández-Montes). https://doi.org/10.1016/j.softx.2023.101527 Received 20 July 2023; Received in revised form 12 September 2023; Accepted 12 September 2023 Available online 28 September 2023 2352-7110/ SoftwareX 24 (2023) 101527 2 Alejandro Fernández-Montes et al. 1. Introduction Time series play a significant role in multiple real-world problems, spanning economics, meteorology, engineering, and various other disciplines. Many of the challenges faced by researchers in these fields, such as the prediction and generation of time series through machine learning models, involve a comprehensive comparative analysis between time series to assess their similarity. Analogous challenges in fields such as image generation or production estimation typically employ one of the following approaches for evaluation: (a) analysis grounded in absolute numeric differences, or (b) subjective assessments through human evaluation. However, to effectively evaluate the performance of unsupervised machine learning models through multivariate time series similarity analysis, the aforementioned approaches fall short. We need to factor in several concepts such as: (a) correlation between variables; (b) trends, the long-term general direction of the time series; (c) seasonality, the regular and predictable patterns repeated at fixed intervals; (d) cycles, patterns that are not repeated at fixed intervals; and (e) noise, i.e., random variability. This work presents SimilarityTS, a toolkit for the Evaluation of Similarity for multivariate time series. This software toolkit enables the research community to compare and assess the similarity between datasets of multivariate time series. SimilarityTS assembles a standard set of well-established metrics and visualisation techniques and fosters consistency when comparing research results and facilitates the repeatability of experiments. Further details on how this toolkit facilitates a more robust and harmonised evaluative framework will be elaborated in Section 2. This paper has been structured as follows: Section 2presents the motivation behind the proposal of SimilarityTS, as well as a brief analysis of the state of the art regarding the evaluation of similarity between time series. Section 3describes the architecture and capabilities of the software, Section 4includes a group of examples of usage and results provided by the toolkit, and Section 5draws the final conclusions and future work. 2. Motivation Numerous research works require a deep analysis of the similarity of multivariate time series to evaluate the quality of their proposals. Although SimilarityTS is a general-purpose toolkit, for the sake of brevity, we present a short review of the literature regarding only one particular topic: the generation of multivariate time series datasets through semi- and unsupervised machine learning models. These models are used for a wide range of applications, such as anomaly detection, class imbalance, data augmentation, and (conditional) prediction. This topic is representative of the current situation and shows the need for a standard evaluation toolkit. The authors of [1] compare four neural network-based generative approaches for time series that belong to the current methodologies in the literature. In this work, they use several techniques to evaluate the quality of the models under consideration: line plots, Dynamic Time Warping Nearest Neighbour, Kullback–Leibler Divergence, Cross- Correlation and Histogram analysis, and a collection of discriminators between the real and synthetic time series. A technique for forecasting an entire process model from historical event data is proposed in [2]. For the evaluation of the quality of the technique, the authors employed a statistical analysis of the features involved, as well as line plots. The authors of [3] propose a hybrid generative adversarial networks for cloud workload prediction. In this work, classic metrics for the evaluation of regression models, such as Moving standard deviation, Mean Absolute Error, and Mean Absolute Percentage Error are used, in addition to line plots. Fig. 1. SimilarityTS programming flowchart. This skimpy analysis clearly shows that there is no standard set of metrics, plots, and tools to evaluate the quality of the models proposed in the literature for the generation of multivariate time series, leading to an inability to fairly compare research results between different authors. SimilarityTS offers a standard set of well-established metrics. We analysed, among others, the techniques employed in the relevant literature in the field, such as Dynamic Time Warping [1,4], the metrics used for the evaluation of classification [5], regression, [2], and unsupervised generative machine learning models in the literature [6], as well as distance measures [7,8] to select the metrics and figures described in Section 3.2.1. 3. Software description The aim of SimilarityTS is to produce analytical and visualisation capabilities with respect to the similarity of multivariate time series. Such insights enable high-quality knowledge-based decisions. To achieve this, the user needs to provide (a) the main time series to be analysed (ts1), that usually represents the baseline or ground truth time series ; and (b) a collection of time series (ts2s) that will be compared to the aforementioned main time series. The toolkit computes metrics and figures that assess the similarity between each of the (ts2s) collection of time series with respect to (ts1). SimilarityTS SoftwareX 24 (2023) 101527 3 Alejandro Fernández-Montes et al. can work with multivariate time series of different length. In such a case, the longer time series, (ts1), will be divided into windows of the size of the shorter ones, (ts2s). Then, the most similar windows will be chosen for comparison. This selection is also configurable for advanced use cases. For a deeper understanding of this process, it can be visualised through the flowchart found in Fig. 1. SimilarityTS is available as: (a) a pip Python package usable by any Python project (such as PyTorch machine learning projects); and (b) a pip command line interface binary. 3.1. Software architecture SimilarityTS is developed following the object-oriented paradigm and uses a layered architecture divided into: •Configuration layer, which establishes the behaviour of the toolkit based on user requirements, such as the metrics and figures to be computed. •Core layer, which performs the initial processing of time series needed by the upper layers and dynamically computes metrics and figures. •Metrics and figures layer, responsible for dynamic creation of the required metrics and figures and supply them to the core layer. This layer provides a plug-in system so that final users can dynamically register their own catalogue of metrics and figures. Thanks to this plug-in system, SimilarityTS provides an easy way to extend and specialise the base functionality. 3.2. Software functionalities Similarity TS provides two categories of results: metrics and figures. The following subsections cover the details of both. 3.2.1. Metrics for the assessment of similarity SimilarityTS computes a set of metrics and figures to evaluate the similarity between time series. To enable multivariate analysis, SimilarityTS computes multivariate implementations for each metric that take into account all the features of the analysis, instead of only computing the univariate analysis between pairs of features. However, for completeness, SimilarityTS also computes the univariate version of each metric to allow isolation of the behaviour of each feature. In the following sections, we describe each of the figures and metrics available in SimilarityTS. 1. Kullback–Leibler divergence: The Kullback–Leibler divergence (also known as KL divergence or relative entropy) is a measure of the difference between two probability distributions. Often used in machine learning and statistics to compare the similarity of two distributions or to contrast how the model fits to data. We used the method described in [9]. The KL divergence can be a useful tool to assess the similarity of two time series. However, it is important to remember that it is sensitive to the specific probability distributions that are used to represent them. 2. Jensen–Shannon divergence: The Jensen–Shannon divergence (JS) is a measure of similarity between two probability distributions. It is a symmetric version of the Kullback–Leibler divergence (KL divergence) and is defined as the average of KL divergences between the distribution 𝑎and the distribution 𝑏, and between the distribution 𝑏and the distribution 𝑎. The JS divergence is a useful tool to compare the similarity of two time series. As it is symmetric, the order of the two distributions under evaluation does not have any influence on the results. Like the KL divergence, the JS divergence is sensitive to the specific probability distributions that are used to represent time series. 3. Kolmogorov–Smirnov test: The Kolmogorov–Smirnov (KS) test is a statistical test used to compare cumulative distribution functions (CDF) between two samples. It is often used to test whether two samples come from the same distribution or to compare the fit of a theoretical distribution to a sample. In the context of time series analysis, the KS test can be used to analyse the similarity of two time series. Both time series must be transformed into a group of feature vectors that represent the distributions. Then, we computed the KS test over the extracted featured vectors of each time series to compute the maximum difference between the CDFs of the two samples. The implementation provided by [10] is used. The KS test is sensitive to specific feature functions that are used to represent time series. In summary, the KL divergence, the KS test, and the JS divergence can be used to measure the similarity of two time series, but they are sensitive to the specific methods used to represent the time series as probability distributions or feature vectors. Choosing the appropriate methods to represent the time series can be important for an accurate comparison of their similarity. 4. Maximum Mean Discrepancy: Maximum mean discrepancy (MMD) is a kernel-based statistical test used to determine whether two given distributions are the same, which is proposed in [11]. In the context of time series analysis, MMD can be used to compare the similarity of two time series. The MMD is calculated as the maximum difference between the mean of the feature vectors in one distribution and the mean of the feature vectors in the other distribution, taken over all possible feature functions. Like the KS test, MMD is sensitive to the specific feature functions that are used to represent the time series. 5. Dynamic Time Warping: Dynamic Time Warping (DTW) is an algorithm that is used to compare time series by aligning them in a way that minimises the distance between them. It is often used in speech recognition and pattern recognition tasks because it enables the comparison of time series that may have different lengths or that may be shifted in time. DTW is not a metric in the strict mathematical sense of the term. A metric is a function that satisfies certain properties, such as being non-negative, symmetric, and satisfying the triangle inequality. DTW does not satisfy these properties [12], so it is not a metric in the traditional sense. However, it is often referred to as a ‘‘distance measure’’ or a ‘‘similarity measure’’ because it quantifies the distance between two time series. In this work, we use the proposal for multidimensional DTW of [13]. 6. Difference of co-variances: The difference of co-variances serve as an indicator to assess how different the relationships between variables of two time series are. It is computed as the average of the row-wise Frobenius norm for the covariance difference matrix. 7. Difference of correlations In the same way, the difference in Pearson’s correlation can be used to assess the similarity between two time series variables. It is computed as the average of the row-wise Frobenius norm for the Pearson correlation difference matrix. 8. Difference of histograms Finally, the difference of histograms can determine how far the ranges of values are between two time series. 3.2.2. Figures for visual assessment 1. Time series plotting: We can use the proposed toolkit to plot the ordinary graphical representation of the time series in a 2D figure with the time represented on the 𝑥axis and the data values on the 𝑦-axis for SoftwareX 24 (2023) 101527 4 Alejandro Fernández-Montes et al. Fig. 2. Example of a 2D time series plot generated by the proposed toolkit. The plot visualises the comparison between two multivariate time series, with time represented on the 𝑥-axis and data values on the 𝑦-axis. Data from the two series are overlapped in the plot to highlight similarities and differences, offering insights into their dynamic relationships. Fig. 3. Delta figure illustrating the differences between the values of a single column from two different time series, grouped by an interval of 50 min. Delta representations aids in the visual comparison of short-, mid-, and long-term patterns in the data, highlighting trends and patterns over the specified interval. (a) the complete multivariate time series; and (b) a plot per column. Each generated figure plots both the first time series and the second time series to easily obtain key insights into the similarities or differences between them. Fig. 2 shows an example of a graphical representation for the comparison of various columns between both time series. To this end, data from both time series are overlapped in the same plot. 2. Deltas: Differences between the values of each column grouped by periods are computed. For example, the differences between the cpu usage every 5 min or every 30 min. These delta figures can be used as a means of comparison between short-, mid-, and long-term patterns of time series. Fig. 3 shows an example of a graphical representation for comparing a column for both time series. 3. DTW path: In addition to the numerical similarity measure, the graphical representation of the DTW path of each column can be useful to better analyse the similarities or differences between the time series columns. Notice that there is no multivariate representation of DTW paths, only single column representations. Fig. 4 shows an example of a graphical representation of the DTW path between columns from both time series. 4. PCA: PCA is a linear-dimensional-reduction technique that aims to find the principal components of a dataset by computing the linear combinations of the original characteristics that explain the most variance in the data. Therefore, PCA is better suited for datasets with a linear structure, whereas t-SNE is better suited for datasets with a nonlinear structure. Fig. 5(a) shows an example of a graphical representation of PCA between all windows of ts1 and each time series in the collection ts2s. 5. T-SNE: T-SNE is a tool to visualise high-dimensional datasets in a 2D or 3D graphical representation proposed by [14], allowing the creation of a single map that reveals the structure of the data at many different scales. T-SNE is a nonlinear technique that aims to preserve the local structure of the data. Fig. 5(b) shows an example of a graphical representation of t-SNE between all windows of ts1 and each time series in the collection ts2s. 4. Illustrative examples 4.1. Command line interface To use the similarity-ts-cli tool, a pip installation is needed by running the following command in your terminal: pip install similarity-ts-cli Some examples of similarity evaluation are shown below. Users can download some test data by running the following command: wget https://github.com/alejandrofdez-us/ ⌋ similarity-ts-cli/raw/main/data_samples.zip && unzip data_samples.zip ↪ ↪ SoftwareX 24 (2023) 101527 5 Alejandro Fernández-Montes et al. Fig. 4. DTW path figure displaying the optimal alignment path between columns from two distinct time series, aiding in a detailed analysis of their similarities or differences. Fig. 5. Dimensionality reduction figures generated by SimilarityTS toolkit using PCA and t-SNE techniques, facilitating a deeper understanding of the underlying patterns and structures in the analysed time series data. •A time series and all time series computing all metrics and figures: similarity-ts-cli -ts1 data_samples/alibaba2018/ts1_ ⌋ machine_usage_days_1_to_8_grouped_ ⌋ 300_seconds.csv -ts2 data_samples/alibaba2018/ts2 -head ↪ ↪ ↪ ↪ Every metric computation and figure generated will be found in the {timestamp} directory. •Two time series computing only the DTW metric and the DTW figure: similarity-ts-cli -ts1 data_samples/alibaba2018/ts1_machine_ ⌋ usage_days_1_to_8_grouped_300_ ⌋ seconds.csv -ts2 data_samples/alibaba2018/ts2/sample_ ⌋ 0.csv -head -m dtw -f dtw ↪ ↪ ↪ ↪ ↪ ↪ For further information on these examples, visit our repository similarity-ts-cli. 4.2. Python package To use the SimilarityTS package, a pip installation is needed by running the following command in your terminal: pip install similarity-ts Users must create a new SimilarityTs object by calling its constructor and passing the following parameters. •ts1 This time series may represent the baseline or ground truth time series as numpy array with shape [length, num_features].ts2s A single or a collection of time series as a numpy array of shape [num_time_series, length, num_features] . •ts1 time series and ts2s time series file(s) must: (a) have the same dimensionality (number of columns), (b) do not include a timestamp column and, (c) include only numeric values. If your data include categorical values, it might be preprocessed to convert them to numerical values. •All ts2s time series must have the same length (number of rows). SoftwareX 24 (2023) 101527 6 Alejandro Fernández-Montes et al. If the ts1 time series is longer (more rows) than the ts2s time series, the ts1 time series will be divided in windows of the same length as the ts2s time series. For each time series in the collection ts2s, the window (*) most similar to the time series ts1 is selected. Finally, metrics and figures are computed. These assess the similarity between each pair of ts2s time series and its most similar associated window ts1. (*) The metric used for selecting the most similar time series window ts1 per each time series file ts2s is pickable. dtw is the default selected metric; however, any of the metrics described in Section 3.2.1 are also available for this purpose. import numpy as np from similarity_ts.similarity_ts import SimilarityTs↪ ts1 =np.random.rand(200,2) ts2s =np.random.rand(5,100,2) similarity_ts =SimilarityTs(ts1, ts2s) for ts2_name, metric_name, computed_metric in similarity_ts.get_metric_computer():↪ print(f'{ts2_name}.{metric_name}: {computed_metric}')↪ Users may also extend the functionality of the toolkit by implementing their own metrics or figures. New metrics and figures may be added to MetricFactory andPlotFactory, respectively, by using the register_metric or register_plot methods. import numpy as np from similarity_ts.similarity_ts import SimilarityTs↪ from similarity_ts.metrics.metric_factory import MetricFactory↪ from similarity_ts.plots.plot_factory import PlotFactory↪ from ed import EuclideanDistance from cc_plot import SimilarityPlotByCorrelation MetricFactory.get_instance() .register_metric(EuclideanDistance)↪ PlotFactory.get_instance() .register_plot(SimilarityPlotByCorrelation)↪ ts1 =np.random.rand(200,2) ts2s =np.random.rand(5,100,2) similarity_ts =SimilarityTs(ts1, ts2s) for ts2_name, metric_name, computed_metric in similarity_ts.get_metric_computer():↪ print(f'{ts2_name}.{metric_name}: {computed_metric}')↪ for ts2_name, plot_name, generated_plots in similarity_ts.get_plot_computer():↪ __save_figures(ts2_name, plot_name, generated_plots)↪ For further information on these examples, visit our repository similarity-ts. 5. Conclusions and future work This paper presented SimilarityTS, a toolkit for the Evaluation of Similarity for multivariate time series. This software toolkit enables the research community to compare and assess the similarity between multivariate time series in a standard way, by means of the most relevant metrics and figures usually employed in the literature, which establishes a standard way to compare research results and facilitates the repeatability of experiments. In future work, we intend to build an extended tool based on SimilarityTS strictly focused on the evaluation of unsupervised generative machine learning models, which would include multiple classification metrics and techniques to discriminate between real and synthetic multivariate time series. Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Alejandro Fernandez Montes reports financial support was provided by Ministerio de Ciencia e Innovación. Alejandro Fernandez Montes reports financial support was provided by Junta de Andalucía (Consejería de Transformación Económica, Industria, Conocimiento y Universidades). Damian Fernandez Cerero reports financial support was provided by Unión Europea NextGenerationEU. Alejandro Fernandez Montes reports financial support was provided by Unión Europea -NextGenerationEU. Data availability The data is available in https://github.com/alejandrofdez-us/simila rity-ts-cli/tree/main/data_samples Acknowledgements Project PID2021-122208OB-I00, PROYEXCEL_00286 and TED2021- 132695B-I00 project, funded by MCIN/AEI / 10.13039/501100011033, by Andalusian Regional Government, and by the European Union - NextGenerationEU. Appendix A. Supplementary data Supplementary material related to this article can be found online at https://doi.org/10.1016/j.softx.2023.101527. References [1] Gatta F, Giampaolo F, Prezioso E, Mei G, Cuomo S, Piccialli F. Neural networks generative models for time series. J King Saud Univ - Comput Inf Sci 2022;34(10, Part A):7920–39. http://dx.doi.org/10.1016/j.jksuci.2022.07.010, URL https:// www.sciencedirect.com/science/article/pii/S1319157822002361. [2] De Smedt J, Yeshchenko A, Polyvyanyy A, De Weerdt J, Mendling J. Process model forecasting and change exploration using time series analysis of event sequence data. Data Knowl Eng 2023;145:102145. [3] Maiyza AI, Korany NO, Banawan K, Hassan HA, Sheta WM. VTGAN: hybrid generative adversarial networks for cloud workload prediction. J Cloud Comput 2023;12(1):97. [4] Li H, Liu J, Yang Z, Liu RW, Wu K, Wan Y. Adaptively constrained dynamic time warping for time series classification and clustering. Inform Sci 2020;534:97–116. http://dx.doi.org/10.1016/j.ins.2020.04.009, URL https: //www.sciencedirect.com/science/article/pii/S0020025520303054. [5] Yoshida S, Chakraborty B. A comparative study of similarity measures for time series classification. In: Otake M, Kurahashi S, Ota Y, Satoh K, Bekki D, editors. New frontiers in artificial intelligence. Cham: Springer International Publishing; 2017, p. 397–408. http://dx.doi.org/10.1007/978-3-319-50953-2_27. [6] Lin W, Yao K, Zeng L, Liu F, Shan C, Hong X. A GAN-based method for time-dependent cloud workload generation. J Parallel Distrib Comput 2022;168:33–44. http://dx.doi.org/10.1016/j.jpdc.2022.05.007, URL https:// www.sciencedirect.com/science/article/pii/S074373152200123X. [7] Li H. Distance measure with improved lower bound for multivariate time series. Physica A 2017;468:622–37. http://dx.doi.org/10.1016/j.physa.2016.10.062, URL https://www.sciencedirect.com/science/article/pii/S037843711630749X. [8] Deza E, Deza MM, Deza MM, Deza E. Encyclopedia of distances. Springer; 2009. [9] Perez-Cruz F. Kullback-Leibler divergence estimation of continuous distributions. In: 2008 IEEE international symposium on information theory. 2008, p. 1666–70. http://dx.doi.org/10.1109/ISIT.2008.4595271. SoftwareX 24 (2023) 101527 7 Alejandro Fernández-Montes et al. [10] Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, et al. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods 2020;17:261–72. http://dx.doi.org/10.1038/s41592-019-0686-2. [11] Gretton A, Borgwardt KM, Rasch MJ, Schölkopf B, Smola A. A kernel two-sample test. J Mach Learn Res 2012;13(1):723–73. [12] Dynamic time warping. Berlin, Heidelberg: Springer Berlin Heidelberg; 2007, p. 69–84. http://dx.doi.org/10.1007/978-3-540-74048-3_4. [13] Meert W, Hendrickx K, Van Craenendonck T, Robberechts P. DTAIDistance. Zenodo; 2020, http://dx.doi.org/10.5281/zenodo.7158824. [14] Van der Maaten L, Hinton G. Visualizing data using t-sne.. J Mach Learn Res 2008;9(11).