Towards Securing AI Integrated Business Processes: a Reasoning Framework Informed by a Semantic Model

Towards Securing AI Integrated Business Processes: a Reasoning Framework Informed by a Semantic Model Gal Engelberg1, Avi Shaked2, Nan Messe3, and Pnina Soffer1 1University of Haifa, Haifa, Israel [email protected],[email protected] 2Department of Computer Science, University of Oxford, Oxford, UK [email protected] 3IRIT, Toulouse University, CNRS, INP, UT2, Toulouse, France [email protected] Abstract. The integration of large language models (LLMs) and Artificial Intelligence (AI) into business processes is transforming organizational workflows, by automating LLM-based AI tasks. This incorporation of agentic AI into business processes introduces new security risks that are often overlooked at the business level. Existing frameworks focus on application-level controls, leaving a gap in understanding how AI-related vulnerabilities impact overall process behavior. This research-in-progress paper introduces a novel framework that links business process models with agentic AI components, capturing key security threats, mitigation strategies, and their interdependencies. This framework includes an inference mechanism that uses a semantic model to assess how the integration of AI agents can impact business processes. Key words: AI-Augmented Workflows, Business Process Security, Conceptual Modeling, AI Security, Responsible BPM 1 Introduction We are in an Artificial Intelligence (AI) driven transformation era, as organizations rapidly automate core business processes using recent AI advancements [1, 20]. Technology providers and enterprises embed generative AI into platforms to deploy autonomous agents that streamline workflows, handle queries, improve efficiency, and user satisfaction [15, 7]. Academic and industry research supports this shift, showing how AI can enhance business process management (BPM) [3, 18, 19]. However, this may introduce new security risks. BPM-related security has traditionally focused on infrastructure risks from cloud platforms, Information Technology (IT), and Internet of Things (IoT). Cloud-based BPM poses confidentiality and integrity risks due to multi-tenancy and shared services [16]. IoT integration exposes processes to edge device vulnerabilities, potentially disrupting availability [6]. In IT environments, cyberattacks propagate through interdependent systems, impacting business continuity [5]. 2 Engelberg et al. Existing work often assumes deterministic systems, overlooking emergent risks posed by AI-driven decision-making. AI agents are autonomous systems that combine Large Language Models (LLM) based language processing with tool integration to perform complex tasks without human input, enabling human-like reasoning and communication across domains [10]. Integrating AI agents into business processes introduces unique security risks absent in traditional systems. For example, in AI-driven recruitment, prompt injection attacks may embed hidden instructions in resumes to manipulate evaluations or extract sensitive data as hiring criteria [11]. This example reflects a broader set of challenges related to the reliability, transparency, and governability of AI agent behavior in business contexts [18, 19]. The AI transformation has led to new security frameworks and standards. Examples include OWASP Top 10 for LLM Applications [11], outlining threats such as prompt injection and insecure output handling; OWASP Agentic AI [12], addressing threats arising from AI autonomy; MITRE Atlas [8], cataloging adversarial techniques targeting AI systems; and NIST AI Risk Management Framework [17], offering structured governance of AI risks across the trustworthiness and accountability dimensions. These frameworks provide taxonomies of key threats and control mechanisms in the agentic AI landscape. Regulations as the EU AI Act [4] aim to ensure safe and responsible AI use. These introduce riskbased classifications, imposing stricter requirements on higher risk systems. From a security perspective, the EU AI Act emphasizes resilience to manipulation, reliable data management, transparency of system behavior, and mechanisms for human oversight. Applied to business processes, it underscores the need for careful AI integration, particularly where automated decision affects employees, customers, or regulatory compliance. The various security frameworks address risks at the application or system level [21], leaving the business process level underexplored. This creates gaps in understanding the full business impact of AI-related vulnerabilities. The inability to reason about how AI-specific vulnerabilities impact end-to-end business processes and their security properties limits the analysis of technical risks and their mitigation – as outlined in various security frameworks – with respect to goaloriented regulatory requirements – such as those introduced by the AI Act – and to business objectives. To address this, we propose a framework that provides a foundation for reasoning about the security impact of integrating AI agents into business processes. The paper is structured as follows: Section 2 introduces the framework and the methodology. Section 3 presents preliminary results demonstrating how the semantic model supports reasoning and decision-making for integrating AI agents into business processes. Last, conclusion is provided. 2 Semantic Model Informed Reasoning Framework As a first step towards addressing the outlined challenges, we propose a framework for reasoning about the security impacts of integrating AI agents into business processes. The framework, sketched in Figure 1, consists of two main Towards Securing AI Integrated Business Processes 3 components. The first is a semantic model that codifies descriptions of business processes, AI agents, threats and properties relevant to agentic AI technologies in the context of business processes. The second component is an inference mechanism to reason about the potential effects of integrating agentic AI into the business process. Our research methodology consists of three core steps: (1) identifying relevant concepts from the literature and mapping their dependencies; (2) codifying business processes and their constituent activities alongside their desirable security properties; and (3) analyzing how AI agents may compromise the desirable business process security properties. We plan to evaluate the semantic model using security-related domain ontologies [9, 13], and to assess the usefulness of the inference through interviews with domain experts. Fig. 1. The Proposed Semantic Model Informed Reasoning Framework 3 Demonstrating Reasoning about AI Security Threats in Business Processes We illustrate how the reasoning framework – informed by a semantic model – can support decision-making with respect to incorporating AI agents in a recruitment process, which is a prominent business process in organizations across virtually every domain. Pertinent threats include prompt injection attacks, where a malicious user embeds hidden instructions within their resume, manipulating the AI to provide favorable evaluations irrespective of actual qualifications; and extracting sensitive company information, including confidential hiring criteria or other protected data [11]. We exemplify our approach using a preliminary Object-Process Methodology (OPM) [2] model. OPM was selected for recognizing stateful objects, processes and their relations and hierarchies as key design elements, and for its ability to visually support the envisioned reasoning. However, with an automated reasoning mechanism, other process representations can be used. The preliminary model is available in www.github.com/ASH-SYSTEMS/AI-BPM/tree/main/OPM. Note that the semantic model, which is currently being constructed, is not shown here, but it informs the elements included in the OPM. Fig. 2 shows an excerpt from 4 Engelberg et al. our model, developed in two phases. The top part presents the business process and its security requirements, while the bottom part introduces agentic AI components and their associated threats, supporting reasoning. As described in the top model, the Recruiting For A Position business process includes two specific activities: Receiving Applications and Selecting Top Applications. A third activity is listed simply to denote that the process may include other activities not explicitly discussed here. On the left side, relevant security properties are related to the activities. Response Integrity is a specific form of the known Integrity property, and Intellectual Property Confidentiality is a specific form of the Confidentiality property. We note that while the Intellectual Property Confidentiality property often appears in non AI security properties taxonomies (e.g., [14]), the Response Integrity property is typically absent, and can be seen as an AI-specific flavor of Resource trustfulness. A property has two states: ok, signifying it is satisfied, and not ok if it has the potential to be violated. The security requirements of the process could be determined by assigning properties to activities. For instance, Receiving Applications is assigned the Intellectual Property Confidentiality property, since it is facing external candidates; and Selecting Top Applications is assigned the Response Integrity property, signifying correctness of the selection process. On the right side of the model, the process resources – Human Agent and AI Agent – are listed. The bottom model includes potential specializations, to the level of specific agents. Some relevant components of the Specific AI Agent appear: LLM Model, Function Calling Element,Memory Element. In addition, we outlined threats that are relevant to our example. We are working on integrating an exhaustive list of such threats in the model. Here, we discuss Sensitive Information Disclosure and Prompt Injection, which we analyze as pertinent to the LLM Model component. Two other threats appear, relating to the other AI agent components, yet these will not be further discussed. The threats are linked with their potential adversarial effect of violating specific properties. These componentthreat-property mappings allow for a reasoning mechanism, as outlined in Fig. 1, to determine what are the properties that a specific AI agent exhibits, with respect to both compliance/satisfaction (ok states) and violation (not ok states). These are represented by purple relations, indicating they emerge from other model elements and support the reasoning mechanism. An example of a reasoning query over the presented semantic model, could be: Is an AI agent of the specific AI agent type suitable for performing both activities involved in recruiting for a position? The reasoning mechanism would evaluate if any property violations associated with the AI agent conflicts with the required properties of each activity. The results of such analysis could be communicated with the process owner/designer so that – if needed – a different agent would be used and/or a mitigation could be incorporated. For example, since a Prompt Injection by candidates could lead to the Selecting Top Applications preferring their applications (over others that are more suitable for the position), the desirable Response Integrity of this activity may be compromised if handled exclusively by the Specific AI Agent, and a proper mitigation can be Towards Securing AI Integrated Business Processes 5 in the form of involving a human agent as the co-handler of this activity (e.g., performing lower-level inspection and results verification activities). Fig. 2. Example OPM model 4 Conclusion We discussed and exemplified how the introduction of AI agents into business processes may compromise desirable security properties that are an extension of traditional security objectives. We intend to further develop the underlying semantic model. Specifically, we will incoroporate exhaustive catalogs of agentic AI threats, components, business process security properties and their relations. We also plan to include mitigation patterns. Furthermore, we intend to provide a formal description of the proposed reasoning mechanism and implement an automated version that would allow to verify business process models with respect to AI agents integration, to support decision-makers and process designers. 6 Engelberg et al. References 1. Chrzanowska, E., et al.: Ai-powered digital transformation–organizational perspective. literature review. J. Mod. Sci. 60(6) (2024) 2. Dori, D.: Object-Process Methodology: A Holistic Systems Paradigm. Springer (2002) 3. Dumas, M., et al.: Ai-augmented business process management systems: a research manifesto. ACM Trans. Manag. Inf. Syst. 14(1), 1–19 (2023) 4. European Commission: Eu artificial intelligence act (2024), https://eur-lex. europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206 5. Gonz´alez-Rojas, O., et al.: Quantifying risk propagation within a network of business processes and it services. Bus. Inf. Syst. Eng. 63, 129–143 (2021) 6. Hornsteiner, M., et al.: A reflection on process-oriented industrial iot security management. In: Proc. ICISSP 2025. pp. 242–253 (2025) 7. IBM: Ibm watsonx orchestrate (2024), https://www.ibm.com/products/ watsonx-orchestrate, accessed: 2025-05-14 8. MITRE Corporation: MITRE ATLAS. https://atlas.mitre.org/ (2024), accessed: 2025-05-16 9. Oliveira, ´ I., et al.: An ontology of security from a risk treatment perspective. In: International conference on conceptual modeling. pp. 365–379. Springer (2022) 10. OpenAI: A practical guide to building agents. https://openai.com/research/ agents (2024), accessed: 2025-05-15 11. OWASP Foundation: OWASP Top 10 for Large Language Model Applications. https://owasp.org/ www-project-top-10-for-large-language-model-applications/assets/ PDF/OWASP-Top-10-for-LLMs-v2025.pdf (2025), version 1.1 12. OWASP GenAI Security Project: Agentic ai – threats and mitigations. Online resource, OWASP GenAI Security Project (2025), https://genai.owasp.org/ resource/agentic-ai-threats-and-mitigations/, accessed: 2025-05-16 13. Sales, T.P., et al.: The common ontology of value and risk. In: Proc. ER 2018. pp. 121–135. Springer (2018) 14. Sayar, I., et al.: From what to how: A taxonomy of formalized security properties (2025) 15. ServiceNow: Ai agents studio (2024), https://www.servicenow.com/company/ media/press-room/ai-agents-studio.html, accessed: 2025-05-14 16. Soveizi, N., et al.: Security and privacy concerns in cloud-based scientific and business workflows: A systematic review. Future Generation Computer Systems (2023) 17. Tabassi, E.: Ai rmf 1.0). Tech. Rep. NIST AI 100-1, NIST, Gaithersburg, MD (2023), https://doi.org/10.6028/NIST.AI.100-1, accessed: 2025-05-16 18. Vidgof, M., et al.: Large language models for business process management: Opportunities and challenges. In: Proc. BPM 2023. pp. 107–123. Springer (2023) 19. Vu, H., et al.: Agentic business process management: The past 30 years and practitioners’ future perspectives. arXiv preprint arXiv:2504.03693 (2025) 20. Wornow, M., et al.: Automating the enterprise with foundation models. arXiv preprint arXiv:2405.03710 (2024) 21. Wu, L., et al.: From assistants to adversaries: Exploring the security risks of mobile llm agents (2025), https://arxiv.org/abs/2505.12981