Analyzing Variable Human Actions for Robotic Process Automation

Analyzing Variable Human Actions for Robotic Process Automation A. Mart´ınez-Rojas1(B),A.Jim´enez-Ram´ırez1,J.G.Enr´ıquez1, andH.A.Reijers 2 1Departamento de Lenguajes y Sistemas Inform´aticos, Escuela T´ecnica Superior de Ingenier´ıa Inform´atica, Avenida Reina Mercedes, s/n, 410121 Sevilla, Spain {amrojas,ajramirez,jgenriquez}@us.es 2Department of Information and Computing Sciences, Utrecht University, Princetonplein 5, Utrecht 3584 CC, The Netherlands [email protected] Abstract. Robotic Process Automation (RPA) provides a means to automate mundane and repetitive human tasks. Task Mining approaches can be used to discover the actions that humans take to carry out a particular task. A weakness of such approaches, however, is that they cannot deal well with humans who carry out the same task differently for different cases according to some hidden rule. The logs that are used for Task Mining generally do not contain sufficient data to distinguish the exact drivers behind this variability. In this paper, we propose a new Task Mining framework that has been designed to support engineers who wish to apply RPA to a task that is subject to variable human actions. This framework extracts features from User Interface (UI) Logs that are extended with a new source of data, namely screen captures. The framework invokes Supervised Machine Learning algorithms to generate decision models, which characterize the decisions behind variable human actions in a machine-and-human-readable form. We evaluated the proposed Task Mining framework with a set of synthetic UI Logs. Despite the use of only relatively small logs, our results demonstrate that a high accuracy is generally achieved. Keywords: Robotic Process Automation ·Process discovery ·Task mining ·Decision model discovery 1 Introduction Robotic Process Automation (RPA) is a software technology that facilitates the automation of human tasks, especially when they are structured and repetitive. This research has been supported by the Spanish Ministry of Science, Innovation and Universities under the NICO project (PID2019-105455GB-C31) and the Centro para el Desarrollo Tecnol´ogico Industrial (CDTI) of Spain under the CODICE project (EXP 00130458/IDI-20210319 - P018-20/E09) and by the FPU scholarship program, granted by the Spanish Ministry of Education and Vocational Training (FPU20/05984). f1 f2 ab cd Fig. 1. Proposed framework for variability analysis through interpretable decisions from UI Logs. The current paper focuses on steps 3 and 4. In contrast to other automation approaches (i.e., API-based), RPA works by closely mimicking the way humans interact with computer applications [2]. Some of the typical benefits are that the technology helps to save costs, increases agility, and improves quality [9,18] while its level of intrusiveness is low [33]. Due to these wide range of benefits, industry has adopted this technology on a wide scale in recent years [10]. Most RPA projects start out by observing how human workers perform work that is to be automated. To support this initial RPA analysis, approaches such as Task Mining [1,31] and Robotic Process Mining [22] are highly suitable. What all these techniques have in common is that they operate on UI Logs, i.e., a series of timestamped events (e.g., mouse clicks and keystrokes), obtained by monitoring and recording user interfaces. An open issue concerning discovering the process model behind the UI Log is to disclose the drivers behind the variations that are shown in a process model. These variations indicate that human operators take different decisions for different cases, but it is generally not possible to find rules which explain how these decisions were made. That, however, is crucial knowledge for the engineer who aims to develop the RPA bot. A clear example occurs in the development of a business process outsourcing operation, where it is necessary to work with different systems that, in their turn, are virtualized. In this type of scenario, it is difficult to understand certain operator decisions that depend specifically on the context of the problem being addressed. Although efforts have been made to overcome this problem, they rely on what is observable in the log [3,12,19,32]. This approach has inherent limitations since human work is sometimes based on information that is simply not captured in the log, i.e., it only appears on the screen. Therefore, human decision-making remains hidden in this respect and, as a result, difficult to automate. Against this backdrop, this paper proposes a framework to analyze the variability in human actions automatically. The framework leverages information on the screen to detect factors that influence human decisions – an angle that existing approaches have neglected so far. In previous work, we proposed (1) a tool to monitor the user behavior which generates a UI Log, including one screen capture for each event (cf. step 1 in Fig. 1)[25], and (2) a method to analyze such logs to discover the underlying process model (cf. step 2 in Fig. 1)[15]. This paper significantly extends these contributions by: (1) proposing a novel approach to systematically analyze the screen captures to extract information which is, then, incorporated into the UI Logs (cf. step 3 in Fig. 1) using image-processing techniques [26] (e.g., Optical Character Recognition), (2) presenting a method to discover decision models which explain the variability that is found in the UI Log (cf. step 4 in Fig. 1) using a Machine Learning (ML) approach, and (3) evaluating the approach with synthetics problems of different complexity to demonstrate the efficacy and efficiency of the framework. It should be noted that industrial RPA platforms often do incorporate sophisticated task mining techniques, as well as features for image processing. The point is that these capabilities are not integrated in approaches to analyze task variability. The rest of the paper is organized as follows. Section 2provides the background in topics like behavior monitoring, ML, and image processing. Section 3 introduces a synthetic business case to motivate the proposal. Section 4elabo- rates on the novel method to discover the decision models. Section 5reports the empirical evaluation performed to validate the method. Sect. 6presents a critical discussion. Section 7reviews similar approaches in the literature. Finally, Sect. 8 summarizes the work and describes future research lines. 2 Background The approach that is presented in this paper builds on behavior monitoring techniques, process discovery, Graphical User Interface (GUI) analysis, and ML. For behavioral monitoring, there are several industrial solutions for keylogging1that capture the interaction of a human interacting with a system. In addition, other approaches have been proposed in academia, taking a further step in how to automate certain stages of robotization [3,11,20,24,25]. It should be noted that there are different formats proposed for capturing events, although the most representative for this work is the UI Log from [15] which defines it as an extension of the XES format—standard for event logs in Process Mining—which incorporate attributes like the app name (i.e., the name of the app), event type (i.e., mouse click or keystroke), click type (i.e., left, right, or middle), click coords (i.e., position of the mouse on the screen), the keystroke (i.e., the keys that are typed), and the screenshot (i.e., the screen capture associated to this event path). Using a UI Log, many proposals exist for process discovery, i.e., to automatically or semi-automatically discover the underlying process model that is associated with human behavior [5,6,13,15,19,23]. Moreover, these proposals include functionalities to clean the UI Log from irrelevant information, so that noise in the resulting process model is filtered; and to select variants/cases/activities 1Availabe at: www.spyrix.com and bestxsoftware.com/es/. according to the frequency, length, and other criteria that are useful to identify process candidates to robotize. The resulting process model may contain decision points and separate branches for different process variants. In the field of GUI analysis, approaches exist than can identify the GUI components within an image [28,34]. GUI components are atomic graphical elements with predefined functionality, displayed within a GUI of a software application [28]. Besides locating the element on the screen (i.e., calculating the bounding boxes), they can classify them by the type of element, e.g., image, button, or text. Nevertheless, when dealing with texts, Optical Character Recognition (OCR) techniques are more appropriate. For instance, KerasOCR [17] allows extracting words and their bounding boxes from screen captures. Such an imagebased technique allows for extracting the content and structure of an image in a computer-readable way (cf. Fig. 2). In the ML domain, supervised algorithms exist that focus on finding rules to explain a given dataset, e.g., extracting a decision tree [27]. Datasets are commonly represented in tabular form: each column is an input variable or a label (i.e., what needs to be predicted or classified), and each row is a member of the dataset. In a classification problem, the algorithm tries to find patterns in the input variables that help to explain the labels. Decision trees are an example of classification algorithms that, besides just providing a classification, do so in a human-interpretable way [14]. 3 Running Example This section describes an artificial business case to explain and motivate the problem addressed in this paper. Figure 3depicts an excerpt of the elementary user interfaces for registering a customer in the context of a telecom company. (‘TextView[attached]’, array([ [ 40., 329.], [121., 329.], [121., 346.], [ 40., 346.]]) (‘ButtonView', array([ [ 30., 30.], [115., 30.], [115., 60.], [ 30., 60.]]) (‘ButtonView', array([ [ 30., 30.], [115., 30.], [115., 60.], [ 30., 60.]]) (‘ImageView', array([ [ 120., 330.], [190., 330.], [190., 360.], [ 120., 360.]]) Fig. 2. Example of GUI analysis applied to a sample screen capture. Extracted buttons are in blue, images are in green, and texts are in red. To the sake of readability, some extracted coordinate are not shown. (Color figure online) In this registration process, the human operator checks her email inbox and reviews the pending emails regarding the registration tasks. After opening the email, the operator has to validate that all provided information related to the new customer is correct. More precisely, the customer ID card is expected to be included as an attachment. If it is indeed included (cf. Fig. 3a), all the customer data has to be registered into a CRM system (cf. Fig. 3b). Otherwise, in case the ID card is missing (cf. Fig. 3c), an email has to be sent to the customer requesting such data (cf. Fig. 3d). Regardless which of these two situations occurred (i.e., Variant 1 or Variant 2 of Fig. 3), the operator returns to their inbox to process the next mail. This process must be repeated several times during the day to process the entire queue of emails in the operator’s inbox. Variant 1 Variant 2 a) Open email b) Register customer in CRM c) Open email d) Reply to customer Send Fig. 3. Mockups of motivating example Fig. 4. Excerpt of a UI Log obtained form a keylogger. Fig. 5. Process model discovered from the UI Log. Task Mining techniques can be applied to discover the operator process model [3,15,22]. In essence, these techniques start by monitoring the operator behavior (e.g., with a key logger [25]) and, then, analyze it to extract the relevant activities and cases using image-similarity comparison [15]. This results in a UI Log (cf. Fig. 4) which includes, at a minimum, the time when each event is produced, the activityId, the caseId, the type of event (i.e., mouse click or keystroke), the text which is introduced, the name of the application where the event has occurred, and a screen capture taken just before the event. The process model discovered from such a log would be similar to the one shown in Fig. 5,which correctly depicts that it contains a single decision point after activity “B” (i.e., after seeing the email), where the process branches off into two different variants. Despite the simplicity of this process, disclosing the condition which rules the decision point of this process is challenging, i.e., why is it that the operator choses for decision Id1 or Id2? Existing techniques are unable to find a meaningful correlation between the events and the decision since it is missing from the UI Log – it only appears in the screen captures. 4 Decision Discovery Framework It is clearly challenging to automatically identify the factors behind variable human behavior on the basis of a UI Log that lacks certain key information. The proposed framework includes a first step to enrich the UI Log with features extracted from the screen captures, recorded along with the UI Log (cf. Sect. 4.1). The second step that is carried out by the framework leverages this new information, which is derived from the screen captures, to deliver a decision model. In our opinion, it is crucial that such a model can be interpreted by a machine and also be understood by a human (cf Sect. 4.2). 4.1 Feature Extraction This step transforms the screen captures that are taken for each event in the UI Log into structured information (i.e., features), which can be incorporated back into the log. Since a variety of features can be extracted from a screen capture, the current framework offers a common interface for these extractors (cf. Definition 1) which can be implemented accordingly to the project necessities. Definition 1. AFeature Extractor is a tuple <Name, F unction> that represents a software component named Name,withaFunction that receives an image and returns a list of pairs <key, value>, where the key is the name of a feature that presents a value in the given image. Once implemented, the framework applies these feature extractors to each event in the UI Log, i.e., the Function of each extractor is applied to the screen capture of each event. If the feature extracted is new in the UI Log (i.e., none of the columns of the UI Log have the same name as the key), a new column is appended to the log, and the value is assigned to this event. The rest of the log events have an empty value for this new column. Otherwise, if the feature already exists, the value is assigned to this event at the existing column key. To illustrate this process, we will describe the UI Element Occurrence extractor in more detail. It extracts the occurrences of UI elements in the sense that the output of its Function contains as many keys as different UI elements are found in the screen capture. The value associated with each key is a number greater than 0, which expresses the number of occurrences of this key in the screen capture. The extractor includes the detection and classification of each component to determine which type of UI Element they belong to. For this purpose, each screen capture is processed in three phases by the Function: 1. To detect the UI elements, image-processing techniques are applied to find the elements within the image. In this phase, edge detection algorithms can be used, such as Canny’s algorithm [7], which we applied. Each detected UI element is then cropped to deal with these separately in the next phase. 2. To classify the detected UI elements according to the type of GUI component they belong to, an ML model—previously trained for conducting such a task— is used.2Specifically, we adapted the convolutional neural network proposed in Moran’s work [28], which is able to detect 14 different types of UI elements. 3. To return the number of occurrences for each type of UI element, the detected UI elements in the first phase are first grouped, according to the class determined during the second phase, and then summed. Consequently, after applying the UI Element Occurrence extractor, 14 columns—one for each type of UI element—are appended to the resulting UI Log. They are then available, along with the additional columns from other extractors. Running Example. When considering the screen capture in Fig. 3.a, which relates to our running example, 2 image buttons can be observed (i.e., the ‘envelope’ for returning to the email home page, and the ‘arrow’ for reply), as well as 3 texts. Therefore, the returning list of the Function of the UI Element Occurrence extractor is: {<#ImageButton, 2>, <#TextView,3>}. This results in an UI Log that includes the #ImageButton column, which gets value 2, and the #TextView column, which gets value 3. The values are shown in Fig. 6under the header ‘Occurrence Extractor info’. 2We trained the model with this dataset: https://doi.org/10.5281/zenodo.2530277. Fig. 6. A sample UI Log including features (columns) extracted from the screen captures. Fig. 7. Proposed algorithm for generating labeled datasets 4.2 Decision Model Discovery Once the UI Log is enriched with the extracted features, the decision model discovery takes place. To enable this, first a labeled dataset is generated for each decision point in the process model. Secondly, a classification algorithm is applied to disclose the rules existing in the dataset, which will help to explain the decision in the process model. In the remainder of this section, each of these steps will be explained. Generating the Labeled Dataset. This step processes the enriched UI Log to convert it into a labeled dataset usable by supervised ML algorithms (e.g., classification trees) in the next step. Specifically, one dataset is created for each decision point that appears in the process model. Given a decision point, the objective is to determine which branch is chosen, providing a detailed explanation. Therefore, the label of the resulting dataset is the branch of the decision point. With this aim, Fig. 7illustrates, in an activity diagram, the way of creating an appropriate dataset for each decision point that appears in a process model. First, it receives both the UI Log and the discovered process model and returns a map where the keys are the IDs of the decision point in the process model, and the values are the dataset extracted to each one. The algorithm starts by flattening the UI Log (cf. step 1 in Fig. 7). This operation is done by putting all UI Log events of each case in the same row of the dataset (i.e., the dataset will contain one row for each different case in the UI Log). In this way, the dataset will include columns for an ID (i.e., auto-incremental number starting in 0), the caseID,aTimeStampEnd (i.e., corresponding to the timestamp of the last event of this case), and a TimeStampStart (i.e., corresponding to the timestamp of the first event of this case), and the rest of the UI Log attributes for each activity of this case (i.e., event type, click coordinates, each of the extracted features, etc.). Regarding these latter attributes, the column names are the attribute names prefixing its activityID. For instance, the EventType attribute of activity A will be stored as: EventType A. Then, for each decision point in the process model, the columns of the dataset are filtered in such a way that only the columns related to the activities that precede that decision point are kept (cf. step 2 in Fig. 7). Note that the columns of the events after the decision point relate to events in the future, so they should not be considered when discovering the decision model of that particular decision. Aferwards, the label column is added to the dataset. For each row (i.e., each process case), its value is the branch which is taken for this decision point, (cf. step 3 in Fig. 7). Discovering the Decision Model with Classification Algorithms. As we explained, the labeled dataset generated at this point will be used to train a supervised ML model. This model will classify the label column based on the rest of the dataset columns. There is a wide range of algorithms that can be used for this purpose. For our framework, we use decision trees since both humans and machines can easily interpret them. This kind of model expresses the discovered rules of the classification in the form of a tree: the tree nodes are a column of the dataset (i.e., UI Log attributes) while the tree edges are nonoverlapping conditions as evaluated over the node. Our framework implements four common used algorithms to construct decision trees: CART [8], ID3 [29], C4.5 [30] and CHAID [16]. Although these techniques are known to be similar, they are implemented to explore their behavior in this context. To make the tree even more understable for humans, the framework takes the discovered features and rules to highlight them into the associated screen captures, in this way linking them with their visual information. Fig. 8. Dataset extracted from the UI Log of Fig. 4 15. Jimenez-Ramirez, A., Reijers, H.A., Barba, I., Del Valle, C.: A method to improve the early stages of the robotic process automation lifecycle. In: CAiSE, pp. 446–461 (2019) 16. Kass, G.V.: An exploratory technique for investigating large quantities of categoricaldata.J.R.Statist.Soc.Ser.C(Appl.Statist.)29(2), 119–127 (1980) 17. Keras OCR. https://github.com/faustomorales/keras-ocr. Accessed Mar 2022 18. Lacity, M., Willcocks, L.: What knowledge workers stand to gain from automation. Harv. Bus. Rev. 19, 1–6 (2015) 19. Leno, V., Armas-Cervantes, A., Dumas, M., La Rosa, M., Maggi, F.M.: Discovering process maps from event streams. In: ICSSP, pp. 86–95 (2018) 20. Leno, V., Augusto, A., Dumas, M., La Rosa, M., Maggi, F.M., Polyvyanyy, A.: Identifying candidate routines for robotic process automation from unsegmented UI logs. In: ICPM, pp. 153–160 (2020) 21. Leno, V., Dumas, M., La Rosa, M., Maggi, F.M., Polyvyanyy, A.: Automated discovery of data transformations for robotic process automation. arXiv preprint arXiv:2001.01007 (2020) 22. Leno, V., Polyvyanyy, A., Dumas, M., La Rosa, M., Maggi, F.M.: Robotic process mining: vision and challenges. Bus. Inf. Syst. Eng. 63, 1–14 (2020) 23. Leno, V., Polyvyanyy, A., La Rosa, M., Dumas, M., Maggi, F.M.: Action logger: enabling process mining for robotic process automation. In: CEUR Workshop Proceedings (2019) 24. Leshob, A., Bourgouin, A., Renard, L.: Towards a process analysis approach to adopt robotic process automation. In: ICEBE, pp. 46–53 (2018) 25. L´opez-Carnicer, J.M., del Valle, C., Enr´ıquez, J.G.: Towards an opensource logger for the analysis of RPA projects. In: BPM, pp. 176–184 (2020) 26. Majumder, B.P., Potti, N., Tata, S., Wendt, J.B., Zhao, Q., Najork, M.: Representation learning for information extraction from form-like documents. In: ACL, pp. 6495–6504 (2020) 27. Mitchell, T.: Machine Learning. McGraw Hill, New York (1997) 28. Moran, K., Bernal-C´ardenas, C., Curcio, M., Bonett, R., Poshyvanyk, D.: Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans. Softw. Eng. 46(2), 196–221 (2018) 29. Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986) 30. Quinlan, J.R.: C4. 5: Programs For Machine Learning. Elsevier, Amsterdam (2014) 31. Reinkemeyer, L.: Process Mining in Action: Principles, Use Cases and Outlook. Springer, Switzerland (2020). https://doi.org/10.1007/978-3-030-40172-6 32. Rozinat, A., van der Aalst, W.M.P.: Decision mining in ProM. In: Dustdar, S., Fiadeiro, J.L., Sheth, A.P. (eds.) BPM 2006. LNCS, vol. 4102, pp. 420–425. Springer, Heidelberg (2006). https://doi.org/10.1007/11841760 33 33. Willcocks, L., Lacity, M.: A new approach to automating services. MIT Sloan Manage. Rev. 58(1), 40–49 (2016) 34. Xu, Z., Baojie, X., Guoxin, W.: Canny edge detection based on open CV. In: 2017 13th ICEMI, pp. 53–56 (2017)