Process Performance Indicators for IT Service Management: The PPI Dataset

125 Chapter 8 Process Performance Indicators for IT Service Management: The PPI Dataset Bedilia Estrada-Torresa, Adela del-Río-Ortegaa, Manuel Resinasa, and Antonio Ruiz-Cortésa aUniversity of Seville, Seville, Spain Contents 8.1 Background and Summary ......................................................................125 8.2 Dataset Specification ................................................................................126 8.3 Value of the Data .....................................................................................128 8.4 Data Description ......................................................................................128 8.5 Experimental Design, Materials and Methods .........................................128 8.6 Conclusion ...............................................................................................130 Acknowledgment ..............................................................................................130 References .........................................................................................................130 8.1 Background and Summary In process-oriented organizations, performance measurement is a key factor in determining whether the organization’s objectives in a given area are being achieved, or whether corrective actions are needed. Process performance indicators (PPIs) play a 126 ◾ Research and Evidence in Software Engineering: From Empirical Studies to Open Source Artifacts relevant role in monitoring the business process and identifying possible improvement areas. They allow the evaluation of efficiency and effectiveness of a given process in a quantitative way using data that is generated within the process flow [1–3]. One of the most important elements of PPIs is their measure definition, which provides all the information required to calculate the PPI. In [4], four performance dimensions are described: time, cost, quality and flexibility; derived from the need for processes to be faster, cheaper, better and amenable to changes. These dimensions are reflected in the PPI measure definition. Despite the importance of PPIs, it is unusual to find a clearly defined, fully structured set of key organizational indicators. Usually organizations only have documents with poorly structured information or ambiguously defined indicators with incomplete data [2], or also formal definitions that are difficult for non-technical users to understand, such as algorithms or SQL queries [5]. Over time, several proposals have emerged for the definition and modeling of performance indicators [1, 2, 6, 7], including templates [8, 9] and graphical notations [5, 10], as well as, methodologies [11] and analysis techniques, based amongst others on business process intelligence [12], or the use of PPI definitions in natural language processing [13, 14]. However, a common barrier in process performance research is the lack of public datasets of PPIs defined in real contexts. Organizations are reluctant to make them public because they contain sensitive information for the organization or because they provide a competitive advantage. Reference models such as SCOR [15] or ITIL [16] serve as a guide for the definition of metrics but have the disadvantage of providing definitions at a high level, delegating the task of defining and implementing the indicator customizing it to each organization that follows the reference model. In this chapter, we present the PPI Dataset, a continuously growing public resource containing a set of 102 real performance indicators defined in natural language. These indicators are associated with processes identified in the information technology (IT) service management of several public organizations. The data provided has been previously used in different research approaches, addressing areas as varied as the automatic transformation of natural language PPI definitions into structured PPI definitions that facilitate the automatic calculation of PPIs [13, 14], and the proposal of new indicator management tools and techniques [5]. 8.2 Dataset Specification Subject Natural Language PPI Definitions Specific Subject Area Performance Management Type of Data Spreadsheets Process Performance Indicators for IT Service Management ◾ 127 How Data were Acquired The PPI information was obtained from the SLA of the technical specifications of the RFQ that public organizations provided. The information was filtered and structured manually. As the original information was provided in Spanish, the final list of indicators identified was translated into English. Data Format PPI definitions are provided in CSV format. Indicator lists in English and Spanish are provided separately. Analyzed. Filtered. Parameters for Data Collection We collected the PPIs defined in Service Level Agreements of the technical specifications of the requests for quote that public organizations provided. A total of 102 process performance indicators were identified and defined for several IT service management processes. Description of Data Collection The information from the SLAs was filtered and structured to define the indicators. Furthermore, duplicated PPIs were manually removed. The indicators were classified according to the processes to which they are related. After the review of the domain experts, the final list was created. The definitions were translated to facilitate their use in different contexts. Data Source Location Institution: Applied Software Engineering (ISA) research group. University of Seville. City: Seville Country: Spain Data Accessibility The data is publicly available in a repository. Repository name: The PPI Dataset Direct URL to data: https://github.com/isa-group/ppi-dataset Related Research Article del-Río-Ortega, A., Resinas, M., Durán, A., Bernárdez, B., Ruiz-Cortés, A., Toro, M. VISUAL PPINOT: A Graphical Notation for Process Performance Indicators. Business & Information Systems Engineering. 61, 137–161 (2019). https://doi.org/10.1007/s12599-017-0483-3 van der Aa, H., Leopold, H., del-Río-Ortega, A., Resinas, M., Reijers, H. A. Transforming unstructured natural language descriptions into measurable process performance indicators using Hidden Markov Models. Information Systems. 71, 27–39 (2017). https://doi.org/10.1016/j.is.2017.06.005 van der Aa, H., del-Río-Ortega, A., Resinas, M., Leopold, H., Ruiz-Cortés, A., Mendling, J., Reijers, H. A. (2016) Narrowing the Business-IT Gap in Process Performance Measurement. In: Advanced Information Systems Engineering. CAiSE 2016. Lecture Notes in Computer Science, vol 9694. Springer, Cham. https://doi.org/10.1007/978-3-319-39696-5_33 128 ◾ Research and Evidence in Software Engineering: From Empirical Studies to Open Source Artifacts 8.3 Value of the Data ◾The definitions of indicators contained in the PPI Dataset provide information on the type of performance information that is relevant in commonly used IT service management processes such as those presented in this document (Request for release, Request for change and Incident Management processes). ◾Finding explicit definitions of indicators is rare in real contexts, so the definitions presented can be used as a basis for adapting these indicators to other contexts and/or other similar processes. ◾PPI definitions are very useful in the field of business process management and can be used in future research to define new PPI definition mechanisms such as templates, automatic indicator calculation tools, modeling techniques and tools, approaches based on process performance analysis using process mining, data structure analysis and/or transformation of information structures based on natural language analysis, among many others. 8.4 Data Description The PPI Dataset is publicly available on the Github repository: https://github.com/ isa-group/ppi-dataset The repository contains 102 PPIs identified in the service level agreements (SLA) of the technical specifications of the request for quote (RFQ) that several public organizations provided. These indicators are distributed in different files in CSV format, separating the definitions by language (English and Spanish). ◾ENG_ThePPIDataset.csv ◾SPA_ThePPIDataset.csv Each CSV file contains the following information: ◾Process: The process identifier ◾PPI identifier: A number to identify the PPI ◾PPI name: Short description of the indicator ◾Calculation of the indicator: Detailed description indicating how the indicator should be calculated ◾PPI type: It indicates the type of information or data calculated by the PPI, e.g., time, percentage, etc. 8.5 Experimental Design, Materials and Methods The definition of PPIs is associated with processes from which the information to calculate them is extracted. Since these processes were not clearly defined, Process Performance Indicators for IT Service Management ◾ 129 interviews were conducted with IT staff. With respect to PPIs, the organization provided documentation containing the SLA of the technical specifications of the RFQ. To obtain the PPI Dataset presented, the following was done: ◾From the documents provided, all the information related to the measurement of process performance was extracted. ◾As the information was not homogeneous, it was necessary to preprocess the information provided in order to standardize the types of measures, homogenize the vocabulary and eliminate duplicate records. ◾PPI definitions were translated from Spanish into English. In each application scenario where the dataset information has been used, the treatment of the information has been slightly different, depending on the requirements of each case. Three specific scenarios are shown below, which are included in the section “Dataset Specification – Related research article”. In [13], an approach to automatically align natural language descriptions of PPIs to process models is presented. The approach takes as input values a textual description of the PPI and a related process model. It has three steps: (i) the PPIs are classified according to the type of measure; (ii) the textual description of the PPI is analyzed to extract a set of phrases that relate specifically to the parts of the given process, based on a decision tree; and finally, (iii) the results of the previous steps are combined to generate an alignment between the extracted phrases and the elements of the process model. Similarly, in [14], the data collection was used to automatically translate natural language PPI definitions into definitions according to a structured notation for PPI definition as presented in [8]. The transformation approach (i) analyses a natural language definition of a PPI to identify the parts of that definition that correspond to the parts of the structured definition. To do that, Hidden Markov Models were used. Then (ii) the data required in the structured notation was completed with the information from the previous step to obtain the structured PPI definition. Finally, as a third scenario, in [9], a graphical notation is proposed for the modeling of PPIs. The dataset presented was used as a basis for graphically modeling PPIs using the VisualPPINOT notation. In addition, it was proved that with the basis for automation provided by the tool that implements the notation, it is possible to calculate the indicators without the need to perform translations to an implementation language, like SQL queries, as is traditionally done to calculate the PPI information. Although in the examples presented above only the PPI definitions in English were used, in this dataset they are also provided in Spanish, so that they can be used in future research, such as those related to natural language processing. 130 ◾ Research and Evidence in Software Engineering: From Empirical Studies to Open Source Artifacts 8.6 Conclusion In this chapter, we provide a dataset consisting of 102 PPIs defined in natural language. Those PPIs are related to the IT service management processes of several public organizations in Spain. It is well known that it is difficult to find public datasets that reflect the PPIs used within organizations, as these may reflect sensitive information on the activities carried out within each organization. Therefore, the importance of the dataset provided lies in the fact that it anonymously describes indicators and measures used in real organizations. These PPIs can be used to conduct new research studies or extend existing ones, along lines related to process performance analysis and improvement, natural language processing, or the management of service level agreements, to name a few. Finally, considering the dynamic nature of organizations, we aim at updating this dataset to reflect possible changes and trends in the way their performance is measured. We also plan to enrich the dataset when more information on other PPIs in new or similar processes becomes available. Acknowledgment This work has been partially supported by the European Commission (FEDER), the Spanish Government under projects HORATIO (RTI2018-101204-B-C21) and OPHELIA (RTI2018-101204-B-C22), and the Andalusian R&D&I programs COPAS (P12-TIC-1867), EKIPMENT-PLUS (PR18-FR-2895) and APOLO (US-1264651). References 1. Popova, V., Sharpanskykh, A. Modeling organizational performance indicators. Information Systems. 4, 505–527 (2010). https://doi.org/10.1016/j.is.2009.12.001 2. del-Río-Ortega, A., Resinas, M., Cabanillas, C., Ruiz-Cortés, A. On the definition and design-time analysis of process performance indicators. Information Systems. 4, 470–490 (2013). https://doi.org/10.1016/j.is.2012.11.004 3. del-Río-Ortega, A., Resinas, M., Ruiz-Cortés, A. Business Process Performance Measurement. Encyclopedia of Big Data Technologies. Springer, Cham. (2018). https://doi.org/10.1007/978-3-319-63962-8_99-1 4. Dumas, M., La Rosa, M., Mendling, J., Reijers, H. A. Fundamentals of Business Process Management. (2018) Springer. https://doi.org/10.1007/978-3-662-56509-4 5. del-Río-Ortega, A., Resinas, M., Durán, A., Bernárdez, B., Ruiz-Cortés, A., Toro, M. VISUAL PPINOT: A graphical notation for process performance indicators. Business & Information Systems Engineering. 61, 137–161 (2019). https://doi.org/ 10.1007/s12599-017-0483-3 6. González, O., Casallas, R., Deridder, D. MMC-BPM: A Domain-Specific Language for Business Processes Analysis. In: International Conference on Business Information Systems. BIS 2009. Lecture Notes in Business Information Processing, vol 21. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01190-0_14 Process Performance Indicators for IT Service Management ◾ 131 7. Cuzzocrea, A., Folino, F., Guarascio, M., Pontieri, L. Predictive monitoring of temporally-aggregated performance indicators of business processes against low-level streaming events. Information Systems. 81, 236–266 (2019). https://doi.org/10.1016/ j.is.2018.02.001 8. del-Río-Ortega, A., Resinas, M., Durán, A., Ruiz-Cortés, A. Using templates and linguistic patterns to define process performance indicators. Enterprise Information Systems. 10, 159–192 (2016). https://doi.org/10.1080/17517575.2013.867543 9. Saldivar, J., Vairetti, C., Rodríguez, C., Florian, D., Casati, F., Alarcón, R. Analysis and improvement of business process models using spreadsheets. Information Systems. 57, 1–19 (2016). https://doi.org/10.1016/j.is.2015.10.012 10. Friedenstab, J., Janiesch, C., Matzner, M., Muller, O. Extending BPMN for Business Activity Monitoring. Hawaii International Conference on System Sciences, 4158– 4167. (2012). IEEE 10.1109/HICSS.2012.276 11. Estrada-Torres, B., Piccoli Richetti, P. H., del-Río-Ortega, A., Araujo Baião, F., Resinas, M., Santoro, F. M., Ruiz-Cortés, A. Measuring performance in knowledgeintensive processes. ACM Trans. Internet Technol. 19 (1), 15 (2019). https://doi.org/ 10.1145/3289180 12. Delgado, A., Weber, B., Ruiz, F., Garcia-Rodríguez de Guzmán, I., Piattini, M. An integrated approach based on execution measures for the continuous improvement of business processes realized by services. Information and Software Technology. 56 (2), 134–162 (2014). https://doi.org/10.1016/j.infsof.2013.08.003 13. van der Aa, H., del-Río-Ortega, A., Resinas, M., Leopold, H., Ruiz-Cortés, A., Mendling, J., Reijers, H. A. Narrowing the Business-IT Gap in Process Performance Measurement. In: Advanced Information Systems Engineering. CAiSE 2016. Lecture Notes in Computer Science, vol 9694. Springer, Cham. (2016). https://doi.org/10.1007/ 978-3-319-39696-5_33 14. van der Aa, H., Leopold, H., del-Río-Ortega, A., Resinas, M., Reijers, H. A. Transforming unstructured natural language descriptions into measurable process performance indicators using Hidden Markov Models. Information Systems. 71, 27–39 (2017). https://doi.org/10.1016/j.is.2017.06.005 15. Apics, S.C.C. Supply Chain Operations Reference Model: SCOR Version 11.0. (2015) 16. ITIL Foundation. ITIL 4 Edition. Axelos Books. ISBN: 9780113316076 (2019)