Mapping the Landscape: Exploring Large Language Model Applications in Business Process Management

Mapping the Landscape: Exploring Large Language Model Applications in Business Process Management Bedilia Estrada-Torres(B), Adela del-Río-Ortega , and Manuel Resinas SCORE Lab, Universidad de Sevilla, Seville, Spain {iestrada,adeladelrio,resinas}@us.es Abstract. The irruption of large language models (LLMs) during the last year has prompted researchers and practitioners to explore novel scenarios for integrating LLMs, enhancing task execution efficiency across diverse domains. Among these, business process management (BPM) stands out as a fertile ground for leveraging LLM features. As organizations strive to streamline their processes throughout the BPM lifecycle, the potential benefits of incorporating LLMs become increasingly evident. In this sense, over the past year, several approaches have been proposed to incorporate LLMs in BPM-related tasks. Concurrently, research efforts have identified key research directions that can help guide the adoption of LLMs in the BPM lifecycle phases. In this article, we perform a comprehensive literature review to assess the impact and coverage of existing approaches in addressing these research directions. In addition, we deem it particularly relevant to analyze the evaluation criteria followed. By analyzing existing proposals and techniques, we aim to shed light on the most addressed BPM lifecycle phases, pinpointing the research directions they entail and the evaluation criteria utilized. Through this analysis, we provide valuable insights and recommendations to inform future research endeavors. Keywords: LLM ·Business Process Management ·Lifecycle 1 Introduction Large Language Models (LLMs) enable the development of artificial intelligencebased systems capable of analyzing and processing natural language and generating coherent texts. They facilitate, for example, the development of tasks such as translation, summarization, classification and grammatical correction of texts, extraction of relevant information, analysis of opinions, and generation of code based on natural language descriptions [3]. The recent release and ease of access to LLM versions such as ChatGPT [34]and GPT-4 [35] have driven interest in LLMs across numerous users and application contexts [11,37]. From this boom, concerns arise about how LLMs can be integrated into both the development of everyday personal tasks as well as academic [26] and business activities [15]. https://doi.org/10.1007/978-3-031-61007-3_3 Exploring LLM Applications in Business Process Management 23 Regarding business process management (BPM), which concerns us in this study, some authors suggest that LLMs can help improve process management by reducing the effort and expertise required, improving data analysis, or automating tasks [22]. Vidgof et al. [38] extend this analysis and describe the possible applications of LLMs in each of the BPM lifecycle phases. They also propose six research directions to guide the adoption of LLMs in the BPM context. Despite the novelty of LLMs proposals, and in particular, those related to BPM, in this article, we embark on a comprehensive literature review that aims to (i) identify the existing proposals, methods, and techniques, and their main characteristics (e.g., purpose, inputs, outputs), (ii) determine the most addressed BPM lifecycle phases, (iii) assess the coverage of the research directions proposed in [38], (iv) pinpoint the most used LLMs to date, and (v) recognize the evaluation criteria used in each proposal. To the best of our knowledge, this is the first of these analyses. From this information, we extract the current research trends, evaluation criteria, and areas that need more attention, providing valuable insights to inform future research endeavors. The rest of this paper is organized as follows. Section 2introduces preliminary concepts related to this study. The literature review process is described in Sect. 3. Our results are presented in Sect. 4. Finally, Sect. 5concludes the paper. 2 Preliminary Concepts The following subsections present key preliminaries to provide the necessary background for our discussion: the BPM lifecycle and LLMs. 2.1 Business Process Management Lifecycle Several BPM lifecycle definitions can be found in BPM literature. In this article, we focus on the proposal by Dumas et al. [14] that describes six phases. It starts with the identification phase, which consists of distinguishing the processes relevant to the organization, as well as identifying performance measures and objectives. The discovery phase seeks a better understanding of the process and usually results in one or more graphic models to facilitate its comprehensibility. Analysis is the third phase that seeks to identify and evaluate problems and opportunities for process improvement. In the redesign phase, a new process version is proposed based on the previously identified problems and the alternative solutions. The fifth phase, implementation, involves applying the changes derived from the previous phase at the organizational and the systems level (automation). The last phase is monitoring, in which process data is collected and analyzed to identify whether the process is meeting expectations or adjustments are necessary. The result of monitoring can lead to a new discovery phase. 2.2 Large Language Models LLMs are large computational models with billions of parameters that have the ability to understand and generate human language and have exceptional 24 B. Estrada-Torres et al. learning capabilities [11]. LLMs must be trained using enormous amounts of data that are used to learn complex patterns and relationships between words and phrases [3,11,38]. They rely on a (Transformer) architecture to extract meaning from text sequences and understand semantic relationships [3,11]. Multimodal LLMs can use, in addition to texts, other sources of information such as graphics, sounds, and photographs to produce text outputs [3]. Current LLMs have generated significant interest since the recent release of ChatGPT [34]. However, some of them are an evolution of language models such as Bert [13]orGPT-1 [36], released in 2018. There are several LLMs such as GPT-4 [35], a large-scale multimodal model; PaLM 2 [17] that stands out for its mathematical, coding, and reasoning capabilities; Llama 2 [20], a collection of pretrained and fine-tuned models; Mistral [31], an open-source model that stands out for its precise recovery of information on extensive documents; among others. 3 Review Method We conducted a literature review on applications of LLMs related to BPM following a method inspired by the guidelines proposed in [39]and[23]. The search included proposals until December 2023 and seeks to answer the following. –RQ1: What BPM lifecycle phases are addressed by the identified proposals? –RQ2: What are the main input artifacts used by the identified proposals and the output artifacts generated by them? –RQ3: What research directions on LLMs and BPM are being addressed by the identified proposals? –RQ4: Which LLMs are most commonly used in the identified proposals? –RQ5: What are the evaluation criteria used in the identified proposals? 3.1 Inclusion and Exclusion Criteria The literature was reviewed in search of articles that relate LLM proposals to some BPM lifecycle phase, including methods, techniques, or current state analysis (e.g., positioning articles). Only articles in English were considered. We included the arXiv1articles that have not been peer reviewed due to the novelty of the topic and because the objective of our study is to know the initial trends. Articles not exclusively related to BPM but that consider BPM in some way were also included. Unavailable, repeated, and articles with different authors but with the same content were excluded, as well as bachelor and master theses. 3.2 Sources and Search Strategy The search for articles was carried out through three bibliographic reference databases Scopus2,DBLP3and WOS 4. We use “large language models” and 1https://arxiv.org/. 2https://www.scopus.com/. 3https://dblp.org/. 4https://www.webofscience.com/wos/woscc/basic-search. Exploring LLM Applications in Business Process Management 25 Table 1. Search in databases and articles obtained. Databases Query Init. Mod. Rev. Primary studies Scopus (TITLE(business AND process) AND ABS(large AND language AND models) AND ABS(business AND process)) 160 3 2 DBLP “large language models” + “business process” 3 3 2 2 WOS (TI=(business process)) AND (AB=(large language models) AND AB=(business process)) 062 0 0 Google Scholar “large language models” + “business process” - 382 25 17 Total 4507 30 21 Table 2. Classification of articles according to their status and publication type. Publication status Categories Type Num. % Category name Position Phases Total Unpublished 733% arXiv 1 6 7 Journal 3 1 4 Published 14 67% Main Conference-Track 1 4 5 Forums/Workshop/Subconferences 3 2 5 Totals 21 100% - 8 13 21 “business process” as search criteria, both in the titles and abstracts. Including both phrases as mandatory in the titles and abstracts yielded very few results (Table 1,columnInit.). The search was expanded to the Google Scholar database, and the queries were modified to avoid excluding articles relevant to this study (column Query). In total, we obtained 507 articles (column Mod.). The titles and abstracts were reviewed to discard those that did not meet the inclusion criteria. The complete articles were reviewed in cases where there were doubts about their inclusion, resulting in 30 articles (column Rev.). Finally, the complete articles were reviewed to verify the inclusion criteria, to identify the type of contribution, and the BPM lifecycle phase to which it relates, if applicable. We obtained 21 articles, called primary studies as in [23]. All articles obtained were from 2023, probably due to the recent rise of the LLM topic. 4 Results The primary studies were classified according to the type of contribution or their relationship with the BPM lifecycle phases, their publication status, and sources. Although all of the articles used are available online, we made a distinction between those published in journals, conferences, or other forums (Published, 14) and those available at arXiv (Unpublished, 7) because the latter lack prior expert review and approval. Eight primary studies are positioning articles on the facilities provided by LLMs in BPM, of which only one (12,5%) is Unpublished.The remaining 13 primary studies contribute to the BPM lifecycle phases, with 6 (46%) Unpublished articles (Table 2). Next, we describe the two main categories. 26 B. Estrada-Torres et al. 4.1 Positioning Articles We carefully looked at the 8 positioning articles we found to analyze the potential use cases they propose for using LLMs in the BPM lifecycle phases. Regarding the identification phase, the use of LLMs is suggested in [38] for identifying processes from unstructured documentation. For the discovery phase, LLMs are suggested to generate models by mining and analyzing documentation or communication records [38], create software models in general, perform transformations between models [12], and to support the generation of process models centered on data and knowledge of workers [4]. As for the analysis phase, LLMs are recommended for identifying problems and bottlenecks in processes [38], ChatGPT for analyzing and comprehending software systems [12], and prompt engineering to enhance process explainability [9]. LLMs are proposed to redesign information systems and for BPM [15], to automatically extract redesign suggestions [4], although no details are provided; and to find alternatives to errors and suggestions for improvement [38]. Concerning the implementation phase, LLMs are proposed to support the automation and tailoring of tasks [38]and for decision making [4]. Finally, for the monitoring phase, LLMs are suggested to collect information on the process execution and its subsequent analysis [38]. The remaining articles do not address a particular phase: in [30], artificial intelligence is suggested to substitute modelers and code generators; in [18], a workshop is introduced but it does not provide details of its approach; while in [22], it is proposed to integrate process information with LLMs to reduce the effort and experience necessary for knowledge-based management tasks such as process data query language. We examined the positioning papers to identify possible research directions arising from applying LLM to BPM. Out of the 8 positioning primary studies, only in [38] are (six) research directions proposed related to this area: RD1 -The use of LLMs in BPM practice suggests identifying in which BPM tasks LLMs provide greater value. RD2-Usage guidelines for researchers and practitioners to, for example, suggest an LLM to achieve an expected value and collect best practices for creating prompts. RD3 -Creation, release, and maintenance of task variants specific to BPM deals with the mapping of general natural language processing tasks to tasks within BPM. RD4 -Creation, release, and maintenance of data sets and benchmarks refers, for example, whether existing natural language processing datasets and benchmarks are relevant to BPM. RD5-LLM and BPM Artifacts seeks to understand what artifacts are necessary for LLMs to create valuable results. Finally, RD6 -Development and release of LLMs for BPM, seeks to study how LLMs are constructed for specific BPM tasks. In the next section, these six research directions are used as a classification criterion to analyze the 13 articles contributing to the BPM lifecycle phases. 4.2 Articles that Contribute to the BPM Lifecycle Phases This section addresses the research questions using information of Tables 3and 4. Exploring LLM Applications in Business Process Management 27 Table 3. Classification of primary studies related to the BPM lifecycle phases Proposal Purpose Inputs Outpus Status Phases BPM RD Grohs et al. [19] Generate textual BP model descrips. from textual BP descrips. BP description Output structure BP models textual notation and Textual BP description UP Discovery √1 Bellan et al. [6]Extract BP info. from textual BP descrips. BP description [Context info.] BP elements information PDiscovery √1 Zhu et al. [41]Generate textual BP descrips. from tree notations Conditional tree Background Restrictions Textual BP description UP Discovery Analysis √1&2 Licardo et al. [27]Generate BPM models from textual BP descrips. BP descriptions Not detailed Not specified UP Discovery √1 Fill et al. [16]Generate JSON code from textual model descrips BP descriptions Context model Example BP model in JSON code PDiscovery N-E 1 Härer [21]Generate&render models from models descrips. Structured BP model info. Textual and graphical models PDiscovery ×1 Klievtsova et al. [24]Extract BP elements from textual BP descrips. Textual BP descriptions taken from [29] BP elements information PDiscovery √1 Berti et al. [7]Analysis&Process mining from event logs Event logs Process mining information UP Analysis Monitoring √1&5 Caspary et al. [10]BP compliance Event log Bert model PT Model traces pairs Detected anomalies PAnalysis √1 Mustroph et al. [33]BP compliance Event logs BP descriptions Activity matching Compliance output PAnalysis √1&2 Mudgal et al. [32]Process mining from textual information Text-tasks Extraction rules Process mining information PAnalysis ×1 Lins et al. [28]BP Follow-up BPMN XML file Answers to questions UP Analysis √1 Berti et al. [8]Process mining from abstracted BP models Event logs Process mining info. Queries based on prompts UP Monitoring √1&5 To answer RQ1, we identify the BPM lifecycle phases addressed in each of the primary studies (ps). Only for 2 of them we identified two phases, the rest address only one. For the discovery phase (7 ps), most ps generate a textual process model using a particular notation, by generating renderable code (e.g., JSON), using visualization software (e.g., Graphviz), or by generating textual process descriptions from semistructured process information. For the analysis phase, ps identify whether LLMs can follow the process flow based on descriptions or event logs and verify conformance of log traces from a process model. Finally, for the monitoring phase, ps focus on extracting data from processes to identify bottlenecks, throughput times, etc. No ps regarding the other phases were found. Concerning RQ2,theinputs required for the ps vary based on their purpose, including textual or structured process model descriptions or event logs. Optional inputs are enclosed in brackets ([]). The outputs are also diverse and can include process models (textual or renderable code), process descriptions, or quantitative process information, depending on the BPM lifecycle phases addressed. RQ3 refers to the relationship between the ps and the referenced research directions (Table 3,col.RD). We consider that the 13 ps address RD1 by describing practical applications of LLMs in the BPM (lifecycle) tasks. Two ps address RD2 as they present frameworks and algorithms that connect LLMs with BPM tasks and can be considered as guidelines for practitioners. RD5 is addressed by 2 ps, aiming to enhance BPM tasks through prompt engi- 28 B. Estrada-Torres et al. Table 4. Characteristics of the proposals and their evaluations. Proposal Impl. LLM Evaluation Comparison Evaluation F-T Resource sources Internal Ext. Quantitative Qualitative Availability Grohs et al. [19]Prompts by chats GPT-4 6 PET [5] BP desc. BP descs. Measures Measures - [1] R. [25] - Pre-Rec-F1S Pre-Rec-F1S Correctness - - - Input Output Results Bellan et al. [6]Prompts by chats GPT- 3.5T 7 PET [5] BP desc. Different contextual prompts -Pre-Rec-F1S --Prompts* material* Zhu et al. [41]Prompts by chats GPT- 3.5T 100 Cond. Trees With & Without proposal - - Subjective Evaluators opinions -- Licardo et al. [27]Prototype GPT- 3.5T GPT-4 31 BP descriptions LLMs -Accuracy RGED -√*- Fill et al. [16]Prompts by chats GPT-4 6models descriptions - - - - Examples in paper Härer [21]Architecture Prototype GPT-4 Llama 2 2 modeling scenarios LLMs - - Correctness -Source code Klievtsova et al. [24]Prompts by chats GPT-1, GPT-2, GPT-3, GPT-3.5 21 BP descriptions LLMs -KPI similarities --Results Berti et al. [7]pm4py GPT-4 Bard 4 public event logs LLMs -Pre-recall --pm4py library Caspary et al. [10]Python Bert BPM Acad. Init. logs [40] - [2]Pre-Rec-F1S -√Input Output Source Mustroph et al. [33]Prompts by chats GPT-3.5 GPT-4 45 PET [5] BPIC 2020 LLMs -Pre-Rec --Analysis* Event logs* Mudgal et al. [32]Prompts by chats GPT- 3.5T* Log messages (Log extracts) - - - Textual analysis -- Lins et al. [28]Prompts by chats GPT-3.5 PaLM 2 -LLMs - - Evaluation questions (Chats) - Questions Answers In paper Berti et al. [8]pm4py GPT-4 4 public event logs - - - Discussion -Event logs Questions neering and improving task inputs. No proposed solutions related to RD3,RD4, and RD6 were found. To address RQ4, we refer to Table 4(column LLM ), which includes the LLMs used in the evaluation of the ps. Only one ps indicates the specific LLM version (March 2023); see [32]. Seven ps use a single LLM for their evaluations. Of the 21 LLM versions used in the 13 ps,17areGPT versions (7 are GPT-4, and 5 are GPT-3.5). Bard,Llama 2,PaLM 2,andBert are each used only once. To tackle RQ5, we rely on the remaining information in Table 4. Only 4 ps describe an implementation (2 as prototypes and 2 as Python libraries). Eleven ps indicate using datasets (process descriptions or event logs), but only 6 ps reference these datasets. Specific examples were prepared for testing the remaining 5ps.TheComparison columns indicate whether the evaluation was an Internal comparison using measures and specified LLMs or External, comparing results with other articles or replicating experiments like in [19]. The evaluation of the ps can be Quantitative (7 ps) when it includes measures like precision or recall or Qualitative when based on domain experts’ opinions or reviewers’ assessments. Only 2 ps report using fine-tuning (col. F-T), but one does not specify the parameters used (cf. [27]). Finally, the last column of Table 4shows the resources available for each evaluation. In paper indicates that resources are examples included in the articles, and (*) means that resources could not be accessed. Exploring LLM Applications in Business Process Management 29 5 Discussion and Conclusions In this article, we explore the literature to identify and analyze articles relating LLMs to the BPM lifecycle phases (positioning proposals and practical applications). Through our literature study, we explored proposals, methodologies, and techniques while identifying the primary BPM lifecycle phases addressed as well as the input and output artifacts involved. Additionally, we assessed the coverage of research directions previously outlined, pinpointed commonly used LLMs, and recognized the evaluation criteria employed across proposals. From the initial phase of our study, we conclude that the BPM lifecycle phases currently addressed are directly related to text processing or information extraction, such as models or data, i.e., discovery, analysis, and monitoring. However, more attention needs to be devoted to those oriented to identification, implementation, and redesign. Half of the proposals analyzed have not been officially published so far. The recently released LLMs feature significantly improved capabilities compared to their predecessors, which has accelerated their adoption and accessibility. This is likely why research on integrating LLMs into BPM is still in its early stages. Although many proposals use ChatGPT to extract process information, they do not analyze how the quality of prompts may impact the obtained results. Regarding the LLMs used, the tendency to relate LLMs only to GPT versions is striking. To strengthen proposals outcomes, replicating experiments and comparing results among different LLMs is recommended. It is worthwhile to compare proposals with LLMs from different families, including open-source LLMs, which offer greater privacy assurances, especially in sensitive environments. Concerning evaluation, emphasizing reproducibility is crucial, and this requires accessible information sources, configurations, process stages, and outcomes for experiments. Unclear evaluation criteria (e.g. measures) can hinder reproducibility. Half of the proposals use quantitative evaluation, but evaluatordependent qualitative assessments pose challenges in generating and comparing with new experiments. Specific LLM version codes are also crucial to reproducibility, particularly given continuous technological advancements. Acknowledgments. This work has been partially supported by grant PID2021- 126227NB-C21 funded by MCIN/AEI /10.13039/501100011033/FEDER, EU, by grant TED2021-131023B-C22 funded by MCIN/AEI/10.13039/501100011033 and by the European Union “NextGenerationEU"/PRTR. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. van der Aa, H., Di Ciccio, C., Leopold, H., Reijers, H.A.: Extracting declarative process models from natural language. In: Giorgini, P., Weber, B. (eds.) CAiSE 2019. LNCS, vol. 11483, pp. 365–382. Springer, Cham (2019). https://doi.org/10. 1007/978-3-030-21290-2_23 30 B. Estrada-Torres et al. 2. van der Aa, H., Rebmann, A., Leopold, H.: Natural language-based detection of semantic execution anomalies in event logs. Inf. Syst. 102(C) (2021). https://doi. org/10.1016/j.is.2021.101824 3. Amazon Web Services: What are Large Language Models (LLM)? (2024) 4. Beheshti, A., Yang, J., et al.: ProcessGPT: transforming business process management with generative artificial intelligence. In: IEEE ICWS, pp. 731–739 (2023). https://doi.org/10.1109/ICWS60048.2023.00099 5. Bellan, P., van der Aa, H., et al.: PET: an annotated dataset for process extraction from natural language text tasks. In: Cabanillas, C., Garmann-Johnsen, N.F., Koschmider, A. (eds.) BPM 2022. LNBIP, vol. 460, pp. 315–321. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-25383-6_23 6. Bellan, P., et al.: Extracting business process entities and relations from text using pre-trained language models. In: Almeida, J.P.A., Karastoyanova, D., Guizzardi, G., Montali, M., Maggi, F.M., Fonseca, C.M. (eds.) EDOC 2022. LNCS, vol. 13585, pp. 182–199. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-17604- 3_11 7. Berti, A., et al.: Leveraging large language models for process mining (2023) 8. Berti, A., Schuster, D., van der Aalst, W.M.P.: Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study. In: De Weerdt, J., Pufahl, L. (eds.) BPM 2023. LNBIP, vol. 492, pp. 427–439. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-50974-2_32 9. Busch, K., et al.: Just tell me: prompt engineering in business process management. In: van der Aa, H., Bork, D., Proper, H.A., Schmidt, R. (eds.) BPMDS EMMSAD 2023 2023. LNBIP, vol. 479, pp. 3–11. Springer, Cham (2023). https://doi.org/10. 1007/978-3-031-34241-7_1 10. Caspary, J., Rebmann, A., van der Aa, H.: Does this make sense? Machine learningbased detection of semantic anomalies in business processes. In: Di Francescomarino, C., Burattin, A., Janiesch, C., Sadiq, S. (eds.) BPM 2023. LNBIP, vol. 14159, pp. 163–179. Springer, Cham (2023). https://doi.org/10.1007/978-3-031- 41620-0_10 11. Chang, Y., et al.: A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol (2024). https://doi.org/10.1145/3641289 12. Combemale, B., Gray, J., Rumpe, B.: ChatGPT in software modeling. Softw. Syst. Model. 22(3), 777–779 (2023). https://doi.org/10.1007/s10270-023-01106-4 13. Devlin, J., et al.: BERT: Pre-training of deep bidirectional transformers for language understanding (2019) 14. Dumas, M., Rosa, M.L., et al.: Fundamentals of Business Process Management, 2nd edn. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-662-56509-4 15. Feuerriegel, S., Hartmann, J., Janiesch, C., et al.: Generative AI. Bus. Inf. Syst. Eng. 66, 111–126 (2024). https://doi.org/10.1007/s12599-023-00834-7 16. Fill, H., Fettke, P., Köpke, J.: Conceptual modeling and large language models. Enterp. Model. Inf. Syst. Archit. Int. J. Concept. Model. 18, 1–15 (2023) 17. GoogleAI: PaLM 2 (2023). https://ai.google/discover/palm2/ 18. Grisold, T., et al.: Using large language models in business processes (2023). https://bpm2023.sites.uu.nl/program/tutorials/, Tutorials of BPM 19. Grohs, M., Abb, L., Elsayed, N., Rehse, J.R.: Large language models can accomplish business process management tasks. In: De Weerdt, J., Pufahl, L. (eds.) BPM 2023. LNBIP, vol. 492, pp. 453–465. Springer, Cham (2023). https://doi.org/10. 1007/978-3-031-50974-2_34 20. Touvron, H., Martin, L., Stone, K., et al.: Llama 2: Open Foundation and Fine- Tuned Chat Models. Technical report GenAI, Meta (2023)