D4.3 – CONFIDENTIAL ORCHESTRATION

CONFIDENTIAL6G has received funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101096435 CONFIDENTIAL COMPUTING AND PRIVACY-PRESERVING TECHNOLOGIES FOR 6G D4.3 – CONFIDENTIAL ORCHESTRATION D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 1 PROJECT INFORMATION Call HORIZON-JU-SNS-2022 Type of Action HORIZON-JU-RIA HORIZON JU Research and Innovation Actions Project start date 01/01/2023 Duration 36 months GA No DELIVERABLE INFORMATION Deliverable WP: WP4 – Confidential Networking Deliverable Task: Task 4.3 – Confidential Orchestration Deliverable Identifier: CONFIDENTIAL6G_D4.3 Deliverable Title: CONFIDENTIAL ORCHESTRATION Editor(s): Jani Suomalainen (VTT) Author(s): Jani Suomalainen, Andrea Dalla Costa, Markku Kylänpää, Muzammal Hoque (VTT), Amit Kumar Srivasvatava (NNF), Eduard Marin (TID), UVC, UCD Reviewer(s): NNF, UVC Contractual Date of Delivery: 30/6/2025 Submission Date: 25/06/2026 Dissemination Level: PU Status: Final Version: 1.0 101096435 D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 2 File Name: CONFIDENTIAL6G_D4.3_confidential_orchestration_v1.0 D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 3 Disclaimer The information and views set out in this deliverable are those of the author(s) and do not necessarily reflect the official opinion of the European Union. Neither the European Union institutions and bodies nor any person acting on their behalf may be held responsible for the use which may be made of the information contained therein. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 4 DOCUMENT LOG VERSION DATE DESCRIPTION OF CHANGE V1.0 25/06/2025 Final version, submitted to EC through SyGMa D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 5 TABLE OF CONTENTS List of Acronyms and Abbreviations .............................................................................. 8 List of Figures .................................................................................................................. 9 List of Tables.................................................................................................................. 10 Executive Summary ........................................................................................................ 11 1 Introduction ............................................................................................................. 12 2 Background on Orchestration ............................................................................... 14 2.1 Orchestration ................................................................................................. 14 2.1.1 Virtualization and Containerization ........................................................ 14 2.1.2 Orchestration Systems ............................................................................ 15 2.1.3 Kubernetes Overview .............................................................................. 16 2.2 Orchestration and Cybersecurity .................................................................. 17 2.2.1 Kubernetes Security ................................................................................ 18 2.2.2 European Research on Security in 6G Orchestration .......................... 19 2.3 Confidential Orchestration ............................................................................. 20 2.3.1 The Confidential Container Project ........................................................ 20 2.3.2 Orchestration and Advanced Cryptography ......................................... 20 2.3.3 Other Hardware-based Confidential Orchestration Approaches ....... 21 3 Requirements for Confidential Orchestration In The 6G Networks ................. 23 3.1 Orchestration Challenges from the 6G Use Cases ..................................... 23 3.2 6G Orchestration Paradigms ......................................................................... 25 3.3 Threat Analysis ................................................................................................ 26 3.3.1 Misbehaving Infrastructure Provider / Operator .................................. 26 3.3.2 Misbehaving Tenants .............................................................................. 28 3.3.3 Side-Channel Threats ............................................................................. 29 3.3.4 ML-Specific Threats ............................................................................... 30 4 Architecture and Technology Enablers for Confidential Orchestration .......... 32 D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 6 4.1 Architecture .................................................................................................... 32 4.2 Infrastructure for Confidentiality .................................................................. 35 4.2.1 Confidential Containers on Cloud Platforms ........................................ 36 4.2.2 Confidential Containers on Edge Platforms ......................................... 40 4.3 Control over Confidential Orchestration...................................................... 43 4.3.1 Intelligence for Orchestration ................................................................ 43 4.3.2 Orchestration Metrics ............................................................................ 44 4.3.3 Remote Attestation and Orchestration ................................................. 45 4.4 Secure Data Sharing Across Multiple Entities ............................................. 48 5 Proof-of-Concepts and Experiments for Confidential Orchestration ............... 52 5.1 Testing and Evaluation ................................................................................... 52 5.1.1 Testing Procedure Development ........................................................... 52 5.1.2 Testing Methodology ............................................................................. 53 5.1.3 A Test Case for Baseline Confidential Orchestration ......................... 54 5.2 Confidential Containers for Federated Learning on Edge and IoT Platforms 56 5.2.1 The Use Case ...........................................................................................56 5.2.2 Test Setup and Methods ........................................................................ 57 5.2.3 Results: Energy and Computing Overhead .......................................... 60 5.3 Trust and Sustainabilility Aware Scheduling on Cloud-Edge Continuum . 61 5.3.1 K8s Scheduling ........................................................................................ 61 5.3.2 Scheduler Customization ....................................................................... 63 5.3.3 Prioritization based on Metrics for Confidential Computing .............. 64 5.4 Side-Channel Attacks on Confidential Containers ..................................... 66 5.4.1 Context .................................................................................................... 66 5.4.2 The Attack ................................................................................................ 67 5.4.3 Results ..................................................................................................... 68 6 Conclusions and Future Work .............................................................................. 69 D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 7 References ..................................................................................................................... 71 D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 8 LIST OF ACRONYMS AND ABBREVIATIONS Acronym/Abbreviation Definition AI Artificial Intelligence CCA Confidential Compute Architecture – from ARM CNI Container Network Interface CNN Convolutional Neural Network CoCo Confidential Container – particularly in a reference to the open-source project CSV China Security Virtualization - a security virtualization technology developed by HYGON, implemented with Chinese cryptographic algorithms CVM Confidential Virtual Machine DLT Distributed Ledger Technology FHE Fully Homomorphic Encryption FL Federated Learning FPGA Field-Programmable Gate Array iDLG Improved Gradient Leakage attack K8s Kubernetes KBS Key Broker Service – from the CoCo Trustee project ML Machine Learning OP-TEE Open Portable Trusted Execution Environment PoC Proof of Concept RATS Remote Attestation Procedures REE Rich Execution Environment SEV Secure Encrypted Virtualization – from AMD SGX Software Guard Extension – form Intel SMPC Secure Multi-Party Computation TCB Trusted Computing Base TDX Trust Domain Extension – from Intel TEE Trusted Execution Environment TLS Transmission Layer Security VM Virtual Machine WP Work Package D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 15 managing large numbers of containers manually would be very complex. Consequently, orchestration tools and systems have been proposed for their management. 2.1.2 ORCHESTRATION SYSTEMS Software orchestration systems manage the complexities of distributed 6G applications. [71] Orchestration platforms automate the deployment, scaling, and operational management of workloads, which can range from containerized applications to virtual machines and legacy software. Kubernetes, originating from Google and maintained by the Cloud Native Computing Foundation [72], has emerged as the most prominent system in the container orchestration landscape. Kubernetes provides an extensive feature set and a broad ecosystem, making it a popular choice for many software deployment cases. However, there are also other platforms for orchestration with distinct characteristics and capabilities. For instance: • Red Hat OpenShift [20] is a hybrid cloud application orchestration system built upon the foundation of Kubernetes. • HashiCorp Nomad [21] presents itself as a compact and versatile orchestrator capable of supporting a wide array of workload types, extending beyond containers to include virtual machines, Java applications, and Windows services, all within a unified platform. • Docker Swarm [22] is Docker's native clustering and orchestration tool, seamlessly integrated into the Docker Engine. It offers a straightforward approach to managing Docker containers across multiple hosts, emphasizing ease of use and quick setup. • Amazon Elastic Container Service (ECS) [23] is a highly scalable and performant container orchestration service provided by Amazon Web Services (AWS). • Azure Container Instances (ACI) [24] is Microsoft Azure's serverless container runtime that provides a simplified way to run containers without the complexities of orchestration. • Google Cloud Run [25] is a serverless platform on Google Cloud that allows users to run stateless containers that automatically scale based on incoming requests. It abstracts away most of the operational burden, including infrastructure management, cluster creation, and scaling tasks. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 16 • Apache Mesos [26] is a powerful cluster management tool that excels at resource isolation and sharing across distributed applications and frameworks. • Rancher [27] is a user-friendly platform designed to simplify the management of Kubernetes clusters across various environments. • Lightweight Kubernetes distributions like K3s [28], k0s [29], and MicroK8s [30] provide simplified Kubernetes implementations suitable for edge computing, IoT, or development environments. 2.1.3 KUBERNETES OVERVIEW Kubernetes (K8s) [40] is a prominent container orchestration system. It is an opensource system designed to automate the deployment, scaling, and management of containerized applications. Kubernetes provides scalability by enabling dynamic reallocation of resources, reliability by automatically restarting failed containers, portability by enabling applications to run consistently in different environments, and efficiency by optimizing resource utilization. Cluster Node Service provider API Node Control plane Controllermanager kube-apiserver scheduler etcd – storage kube-controllermanager kubelet kube-proxy pod pod pod pod container confidential container kubelet kube-proxy Container runtime Container runtime Figure 1. Kubernetes architecture (adapted from [9]) with confidential containers highlighted Key concepts of Kubernetes include: - Pod – smallest deployable unit of computing, one or more containers sharing storage and network resources - Node – physical or virtual machine that runs containers - Cluster – group of nodes working together - Namespaces – mean to organize resources within cluster D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 17 - Workload – application or service running on Kubernetes cluster. Kubernetes provides few workloads for simplifying management of user’s containers and thus easing scaling, availability assurance, and self-healing. The Kubernetes architecture is modular and consists of the control and the data plane, which are illustrated in Figure 1. The control plane manages the state of cluster and makes decisions about how applications should run. The control plane consists of the following components : • kube-apiserver is a front-end that enables interactions with the cluster by exposing the Kubernetes API and validating and processing API requests. • etcd stores cluster data, including configurations, states, and metadata. • kube-scheduler assigns pods onto worker nodes by considering resource requirements, constraints, and other factors to determine the best node for each pod. • kube-controller-manager runs various controller processes that regulate the state of the cluster. • cloud-controller-manager is an optional component that integrates Kubernetes with cloud provider APIs and handles cloud-specific functions, such as managing load balancers and storage volumes. The control plane components typically run on dedicated machines, i.e., on control plane nodes or master nodes. The data plane consists of the worker nodes where applications run. Key components of the data plane include: • kubelet which communicates with the control plane and ensures that containers are running as specified in the pod definitions. • kube-proxy network rules that allow network communication to pods within or from outside of the cluster. • Container runtime is software responsible for running containers. Examples of container runtimes or platforms for container runtimes include Docker, containerd, and CRI-O. 2.2 ORCHESTRATION AND CYBERSECURITY This subsection surveys security solutions and research related to orchestration. It aims to provide high-level view to the current state in all security areas while the next subsection focuses on the related work on confidential computing and orchestration. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 18 2.2.1 KUBERNETES SECURITY Kubernetes has several built-in security features (not related to confidential computing) that can be used to authorize and control its use [15]. To secure the network layer, Kubernetes provides Network Policies [16], a mechanism to control the network traffic between pods, allowing for the implementation of network segmentation and isolation. Network plugins, often based on the Container Network Interface (CNI), are used to implement the underlying network fabric and enforce these policies. There are also several third-party security tools for container vulnerability scanning and management, runtime security monitoring and posture management, and assuring authenticity of containers. Table 1. Conventional security mechanisms for Kubernetes Category Security features/tools Description Kubernetes built-in security Role-based access controls Defining user and group policies and enforcing who can access Kubernetes resources Network policies Limiting communication between pods and external services Pod Security Admission Limiting who can use pods and how much. Replacing Pod Security Policies. Kubernetes Secrets Storing sensitive information 3rd party vulnerability management Clair Open-source project for the static analysis of vulnerabilities in application containers Trivy Open-source tool for scanning container images for vulnerabilities Anchore Platform providing vulnerability scanning and image signing Aqua Security Platform integrating several security functions, including vulnerability scanning, access control, runtime protection, auditing, and security posture (integrity & compliance) management. 3rd party runtime security Falco Open-source anomaly and attack detection within Kubernetes clusters Sysdig Secure commercial threat detection and response platform KubeSec An open-source tool for assessing the security posture of Kubernetes configurations. Container integrity Notary Solution for signing container images, mitigates supply chain risks Connosseur Kuberneters admission controller. Intercepts requests to Kubernetes API server to verify container images before they are deployed or updated. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 19 2.2.2 EUROPEAN RESEARCH ON SECURITY IN 6G ORCHESTRATION The security of orchestration and utilization of trusted computing technologies and privacy enhancing cryptography is an active research topic. For instance, several ongoing European 6G-IA/SNS projects [2] are addressing confidential orchestrations (including remote attested deployment and configuration) and introducing new functions: • DESIRE-6G is developing mutual attestation approach where integrity records are protected with blockchain. They also aim to eliminate infrastructure constraints from deployments with platform agnosticattestation. Similar to C6G, their focus is similarly strongly on confidential (trusted) computing technologies. • iTrust6G combines remote attestation with supply chain vulnerability and AIbased behavioural analysis for orchestration of security • SAFE-6G works in the configuration realm and proposes use of AI to transforming user-originated intents into system configurations that align with trustworthiness levels. • RIGOUROUS addresses resource management over Cloud-edge-IoTcontinuum. They introduce DevSecOps approach with AI-driven security orchestration and intent-based security management. Their orchestration framework aims to addresses network complexity and heterogeneity with decentralized management and they have introduced multi-cluster resource manager to optimize resource allocations across distributed clusters. AIdriven decision engine and threat risk assessor provide contextual insights to prioritize risks and guide orchestration. • PRIVATEER work on privacy related question in decentralization of the 6G architecture and data analytics. Privacy-awareness of orchestration is emphasized to maintain trust in heterogeneous networks. They work also on security orchestration (SOAR), evaluation of end-to-end level of trust based on integrity, confidentiality, and resilience, as well as protection of cyber threat intelligence using FL and FHE. • ACROSS works on security and trust services, including continuous remote attestation, dynamic service management. They automate migration (or orchestration) of containers from compromised to healthy node to ensure resilience. • 6G-BRICKS works on software defined perimeters (SDP) and zero trust concept. They contribute by increasing scalability of security management D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 20 with policy and intent-based configuration and with integration of SDP with service security orchestrators. Common strategic recommendations for security and privacy from ongoing SNSprojects [2] include enhancing decentralization to avoid single points of failure, privacy-preserving analytics to support orchestration, AI-driven automation, realtime trust measurements (in all layers and holistically), decentralized verification, integration of architectures for heterogeneous platforms, and cross-domain collaboration. These principles may also be applicable for confidential orchestration. 2.3 CONFIDENTIAL ORCHESTRATION Confidential orchestration refers to the automated deployment and configuration of applications, functions, and data to different computing platforms for execution in a manner that keeps the application isolated and confidential from the platform provider. Confidential orchestration is an extension of the software orchestration concept, extending it to security and confidentiality-protected applications that are safeguarded using trusted hardware platforms or privacy-enhancing cryptography (enabling technologies addressed more closely in other C6G deliverables). 2.3.1 THE CONFIDENTIAL CONTAINER PROJECT The Confidential Containers (CoCo) project [4][5] enhances the security of containerized applications by leveraging TEEs. It is an open-source project under the Cloud Native Computing Foundation and it has been designed to integrate seamlessly with cloud-native technologies, particularly Kubernetes. CoCo aims to standardize confidential computing at the pod level, supporting multiple TEEs and hardware platforms. CoCo enables containers to run within TEEs, effectively creating a secure enclave for the containerized application [6]. TEEs provide a hardware-backed secure and isolated environment within a processor, ensuring that code and data loaded inside are protected (with respect to confidentiality and integrity) from the other software and applications in the platform. TEEs are supported by major CPU vendors like Intel (with SGX and TDX), AMD (with SEVSNP) and ARM (with TrustZone). 2.3.2 ORCHESTRATION AND ADVANCED CRYPTOGRAPHY In addition to studying TEE-based confidential computations, the C6G project also addresses cryptographic computations based on Fully Homomorphic Encryption (FHE) and Secure Multi-Party Computation (SMPC). FHE and SMPC provide D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 21 alternative approaches to secure applications and functions which are deployed to remote untrustworthy locations. FHE allows computations to be performed on encrypted data without the need for decryption, offering a potential solution to address specific security concerns inherent in containerized and orchestrated environments. By enabling data to remain protected even during processing, FHE presents a paradigm shift in data security, extending beyond encryption at rest and in transit to include encryption in use. Architecturally, FHE involves encrypting data before it is deployed in containers. Computations on this encrypted data are then performed within the containers, with the results remaining encrypted until they are retrieved and decrypted by an authorized entity.[13][14] Potential use cases of FHE in the context of containers also include scanning and verification. Vulnerability scanning of container images typically requires access to the unencrypted image layers. By leveraging FHE, it may become possible to perform these scans on encrypted container images without decrypting them.[18] The widespread implementation of FHE for securing containers and software orchestration currently faces several significant limitations, primarily related to performance overhead and practical challenges. Another practical challenge lies in the complexity of key generation, management, and distribution in FHE systems. To minimize computational overhead, Enkrypt AI has presented a concept for coupling FHE with CoCo to secure critical AI workflows [19]. They applied a riskbased approach to determine which parts of the ML model should be encrypted and what can be run inside confidential containers. 2.3.3 OTHER HARDWARE-BASED CONFIDENTIAL ORCHESTRATION APPROACHES In addition to the CoCo project, there are several alternative hardware-leveraging approaches for orchestrating confidential applications. For instance, Parma [11][12] is a Microsoft-originated technology for deploying confidential containers on the Azure cloud. Parma focuses on providing strong security for containerized applications by using virtual machine-level isolation offered by AMD SEV-SNP. Extensible Orchestration and Protection Framework (eOPF) [3] aims to provide an extensible framework for confidential cloud computing by leveraging Intel SGX enclaves. eOPF is beneficial for enhancing the security of existing applications already leveraging SGX enclaves, especially in scenarios where mitigating sidechannel attacks is a critical requirement. Confidential orchestration approaches also include FPGA application deployment and configuration. For instance, Xiphera D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 22 has developed [1] an FPGA based TEE implementation for hosting critical (AI) applications. FPGA applications were deployed as encrypted packages and the trusted base included TLS server for configuring the application. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 23 3 REQUIREMENTS FOR CONFIDENTIAL ORCHESTRATION IN THE 6G NETWORKS The fundamental objective of bringing confidentiality to orchestration is to increase the security level of software in 6G networks by reducing the threat landscape. Essentially, we aim to facilitate the adoption of a new trust model where service providers and users do not need to trust the service platform providers or platform operators. This is achieved by keeping orchestrated applications isolated from the platform service provider, i.e., preventing the platform provider from reading or modifying orchestrated code or data, while at the same time enabling automated deployment and configurations to third-party infrastructure. This section studies more closely the requirements we aim to address. What is missing from regular orchestration systems that motivate the adoption of the confidential computing paradigm? What are the requirements for confidential orchestration systems, focusing particularly on the needs in 6G? What are the gaps and threats in existing technologies? We then highlight the principle of least privilege and zero-trust as guiding requirement for the architecture defined in the following section. 3.1 ORCHESTRATION CHALLENGES FROM THE 6G USE CASES Telecommunication network architecture and emerging 6G use cases introduce specific challenges for confidential orchestration. 6G networks are expected to be significantly more complex and dynamic than their predecessors. Further, they are envisioned to support a plethora of novel applications ranging from holographic communications and digital twins to autonomous systems and enhanced industrial automation. The scale and dynamism of 6G as well as heterogeneity of components and applications demands automation and orchestration systems capable of autonomous decision-making and proactive adaptation. 6G use cases and their impact on orchestration requirements include, e.g.: • Enhanced mobile broadband requires orchestration to efficiently manage bandwidth requirements and ensure seamless handovers. This may imply dynamic offloading of compute and AI task to the edge to support demanding applications and heterogeneous (space, aerial, terrestrial) networks. Availability of network services requires that the edge devices are D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 24 operational, these services may also handle privacy related information revealing, e.g., user movements and locations. • Ultra-reliable low latency communications, require seamless coverage and self-healing capabilities, which means offloading of services close to users as well as rapid fallback orchestrations. Orchestration solutions provide resilience also in a case of cyberattacks impacting computing capabilities in specific locations. • Massive machine type communications require capabilities to orchestrate services in scalable and zero-touch management manner. • Autonomous vehicles require orchestration where services are tightly coordinated and able to follow the vehicles. Services offloaded to edge can perform low-latency computing-intensive tasks close to vehicles. Orchestrated services are highly integrity and availability critical. Orchestration can extend from edge to user vehicles. • Smart cities use case require orchestration of code supporting heterogeneous devices and real-time analytics over privacy (confidentiality) critical data. • Industrial application requires orchestration of automation and services which are reliability critical and often contain business secrets. Consequently, orchestration platform must provide strong integrity and often confidentiality guarantees. Orchestration can extend from edge to industrial IoT devices. • Extended reality applications require ultra-low high-bandwidth services and will therefore benefit from edge. Some applications, like healthcare, may be very privacy/confidentiality critical, while some, like remote control of unmanned vehicles, is very integrity and availability critical. Consequently, the challenges that the orchestration system should be able to manage include: • Rapid deployment – deployment time must be short particularly in use cases where user is moving. Sometimes deployment cannot be anticipated in advance (e.g. in error recovery case) which require that also unscheduled deployments must be rapid. • Complexity – 6G networks are highly complex due to various applications, technologies, devices and users that all must be supported. Further complexity arises from the cooperation between different operators – private and public – to create seamless end-to-end services. Complex configuration D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 31 Table 5. Security challenges related to orchestration of ML and to use of ML in orchestration Threat Description Mitigations Information exposure Privacy is a challenge when collecting large amounts of heterogeneous data. AI-driven orchestration systems collect private data which may leak. ML models may also be vulnerable and leak data/models. Privacy preserving architectures for learning, such as federated learning Undesirable configurations or abnormal behaviour The distributed and heterogeneous nature of 6G networks complicates the establishment of a common data processing pipeline for ML applications. Complexity may lead to vulnerabilities. AI-driven orchestration system, e.g. intentbased orchestration, may make unexpected decisions that are difficult to understand. AI systems are also susceptible for attack such as poisoning and evasion. Explainable AI, supply chain security, investing resources on security architecture design and verification Attacks against automation components Closed-loop network automation driven by AI may introduce security threats such as DoS attacks, man-in-the-middle attacks, and deception attacks System and API security Attacks against AI models Adversaries can target 6G/AI models by poisoning the data used for training, launching attacks during the learning and inference phases, and exploiting vulnerabilities in the model's APIs. Poisoning of the model by compromised clients or aggregators, may be easier in federated learning than in centralized learning due to the number of participating nodes, particularly if the security of nodes is not strongly assured and attested. System security, supply chain security, remote attestations. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 32 4 ARCHITECTURE AND TECHNOLOGY ENABLERS FOR CONFIDENTIAL ORCHESTRATION The section analyses architecture and technological building blocks for confidential orchestration. It describes an architecture for orchestration enabling us to provide structured view to confidential orchestration and analyse the available technology for 6G. The section also identifies and highlights gaps in the state-of-art and underlines potential solutions and areas needed further work. In Section 4.1, we will first present a high-level framework – the enabling building blocks and their relations – for confidential orchestration. Then we will look more closely and analyse the technologies for the lower and upper parts of the framework: respectively, the confidentiality supporting infrastructure in Section 4.2 and the orchestration control in Section 4.3. Then in Section 4.4 we discuss how orchestration is related to the data sharing architecture and can be supported by several enabling technologies. 4.1 ARCHITECTURE 6G software orchestration is a cooperation between the controlling entities on the operators’ side and computing assets on the infrastructure provider’s side. An architectural view to the main elements – the orchestration framework – is illustrated in Figure 2. The main domains include the operator domain i.e., the owner and provider of confidential 6G applications and data in the upper part of the figure, and the infrastructure domain, which is hosting the confidential 6G software artefacts, in the lower part of the figure. The trust model is highlighted with a trust boundary (dashed red line in the figure). In the upper side of the figure, we have a domain that is under operator’s control: the operators’ servers on the right and confidential containers on the left. In the lower side of the figure, we have infrastructure that is under the infrastructure provider’s full control in right, or that is under operators’ control through confidential computing technologies. The confidentiality of deployment can be based either on security hardware (here trusted containers and illustrated with light blue) or on cryptographic means (here encrypted containers and illustrated with dark blue). The separation of the basis of trust (encrypted vs. trusted containers) is illustrated with a blue dotted vertical line. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 33 Container orchestration system Trusted Execution Environment Confidential container Confidential container Hardwaresecured container Operator domain The trust model FHE sofware FHE sofware Encrypted container SW (crypto) based The implementation strategy HW (TEE) based FHE sofware FHE sofware (Unconfidential) container Monitoring Analytics & Decisions Reaction & Orchestration Platform attestor Confidential container instantiation 6G applications Code & data Control logic Infrastructure domain (cloud/edge/device) Node & container monitoring Security monitoring Orchestration policies (trust & sustainability driven) Container system Secure multi-domain coordination Orchestration intelligence / run-time decision making Security & quality requirements Secure deployment Secure configuration Operator / confidential application domain Infrastructure provider domain Security orchestration Figure 2. Confidential orchestration architecture The infrastructure domain is based on the following components and enablers: • Trusted containers – The hardware-secured containers will leverage available TEEs, which are integrated to container orchestration solutions, to isolated them from the host platform and other distrusted containers. A prominent trusted container approach is the Confidential Containers (CoCo) project. • Encrypted containers – Software-based applications can be secured using advanced cryptographic means such as fully homomorphic cryptography. • Infrastructure must also provide support for the containers (container system in the figure) by providing compatible CPU architecture, operating system, and container engine. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 34 The operator’s domain control components for orchestrating the services in the infrastructure. To emphasize the application-specific as well as dynamic nature of orchestration, we have divided the domain into four areas: 6G application as well as realms related to monitoring, analysis, and reactions. Monitoring phase i.e. collecting data on the behaviour of infrastructure and security landscape. When considering confidential orchestration, platform integrity information becomes a key metric alongside the traditional performance related metrics. Remote attestation is a process where the application provider (attestor) can verify trustworthiness of target host and containerization systems before critical containers are deployed. It is a method where a host authenticates its hardware and software configuration to a remote party. Remote attestation involves the following architectural entities: • Attester – the entity whose trustworthiness is being evaluated, i.e., the platform where to containers are deployed. • Relying party – the entity that needs to know the trustworthiness of the attester, i.e., the entity deploying containers. E.g. Attestation Service at CoCo Trustee • Verifier – entity evaluating evidence that the attester provides. E.g., Key Broker Service at CoCo Trustee. • Endorser – entity providing secure statements that vouch the integrity of the attestee’s hardware and software. Typically, the manufacturer of edge/cloud hardware to where containers are deployed. Analysis phase relates to making decisions on how the infrastructure deployment and configuration should be adjusted to support application and operator specific requirements and policies • Orchestration intelligence – Applications can be orchestrated in different cloud, edge, and device platforms. Decisions where and when applications are scheduled to operate are based on operators’ or application provider’s needs and on available information on the infrastructure, e.g., on the host platform’s capability or security health. Reactions phase triggers deployments, configurations, and run books to deploy or configure services. • Automated deployment, scaling, and management of 6G applications in confidential containers. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 35 • Deployment of applications secured with privacy-preserving means • Security orchestration – deploying or adjusting security applications 1 Secure control channel – Monitoring, deployment and configuration of containers requires secure services and secure communication between provider and host. Secure communication may also be required for different confidential containers. 4.2 INFRASTRUCTURE FOR CONFIDENTIALITY Confidential containers, which were introduced in Section 2.3.1, can be deployed on various platforms ranging from cloud environments, such as Microsoft Azure, Google Cloud, and Amazon Web Services, as well as on different hardware architectures, including Intel TDX and SDX, AMD SEV, IBM Secure Execution, ARM CCA, and Hygon CSV. However, due to performance and resource constrains confidential containers are not feasible for every platform. For the cases where CoCo are unavailable confidential applications must be orchestrated using alternative strategies. The CoCo consist of the following entities: - Infrastructure (AWS, Azure etc.) provides compute resource for workloads - Hardware TEES (Intel TDX and SGX, AMD SEV-SNP etc.) provide HW-based protection for data - Hypervisor (e.g. KVM, QEMU, cloud hypervisor) manage (confidential) virtual machines - Confidential computing services: o Attestation service (CoCo Trustee) validates hardware integrity o Key broker service (Coco Trustee) facilitates remote attestation o Key management service securely stores, manages, and backs up keys o Image build service (podman, buildah) build confidential containers and VS images o Image registry (Docker Hub, CSP registries etc.) store encrypted and signed container and VM images 1 Orchestration of 6G security applications – i.e. security orchestration – is a subset of 6G application orchestration. While this document mainly focuses to the security of orchestration also the security applications need to be orchestrated securely. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 36 4.2.1 CONFIDENTIAL CONTAINERS ON CLOUD PLATFORMS CoCo encapsulates pods within confidential virtual machines (CVMs), allowing cloud-native workloads to leverage confidential computing hardware with minimal modification. As CoCo's design philosophy prioritizes ease of use by abstracting away much of the hardware-dependent intricacies, it provides an accessible, platform-agnostic confidential computing solutions that can be easily adopted by mainstream cloud-native developers. Intel Trust Domain Extensions (TDX) represent a significant advancement in Intel's confidential computing portfolio, becoming widely available with the 4th Generation Xeon Scalable Processors (Sapphire Rapids). TDX is specifically engineered to enhance the security and privacy of virtualized workloads within cloud environments by creating a TEE that isolates entire virtual machines (VMs) from the underlying physical hardware, including the operating system, hypervisor, and other co-located VMs. AMD Secure Encrypted Virtualization-Secure Nested Paging (SEV-SNP) is a hardware-based confidential computing technology integrated into AMD EPYC processors from generation 3 and newer. As a virtualization-based confidential computing technology, SEV-SNP focuses on protecting a full confidential virtual machine (CVM) environment. AMD SEV-SNP supports running full operating systems within a CVM. Intel TDX and AMD SEV-SNP are classified as VM-based TEEs, meaning they encrypt memory across a traditional VM boundary and operate atop a Virtual Machine Monitor (VMM). They share the overarching objective of removing cloud and infrastructure providers from the guest application's Trusted Computing Base (TCB), integrating natively with Kubernetes, and enabling the deployment of unmodified workloads Intel Software Guard Extensions (SGX) is an earlier hardware-based technology that enables the creation of protected enclaves within memory, allowing only authorized application code to access sensitive data. Unlike full memory encryption technologies, SGX permits a specific application to establish its own protected enclave with a direct hardware interface, thereby limiting access and minimizing performance impact. SGX operates as a "process-based TEE," where a process requiring secure execution is logically divided into trusted and untrusted components. Figure 3 illustrates two architectural alternatives for CoCo. The left alternative is based on lightweight VM, such as Kata containers, on TEE and provides strong D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 37 isolation boundaries. The right alternative isolates specific processes and leverages technologies like Intel SGX and requires modifications to applications but can produce smaller trusted computing base. To the control plane, CoCo introduces container image registry and services for key management and brokering. Further, CoCo provides attestation service to verify the integrity of the TEE and the software running within it [7]. Virtual machine Kubernetes POD container container Pod container images Enclave software stack Image management Enclave agent Host HW TEE Kubernetes POD container container Pod container images Enclave software stack Image management Enclave agent Host Processbased TEE Control plane Container images registry Key broker service Attestation service Key management service Process-based TEEVM-based TEE Figure 3. Confidential Containers project defines two architecture alternatives (VM based in left and process based in right). The diagram adapted from the CoCo specification [8] Workflow for VM-based TEEs approach encapsulates pods within confidential virtual machines (CVMs): 1. Workload preparation includes building container images using standard tools 2. Kubernetes deployment and scheduling towards hosts that support running confidential containers 3. Execution flow in TEEs is the following. a. The confidential container runtime initiates VM TEE and launches a confidential virtual machine. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 38 b. Then an agent residing in newly launched enclave performs remote attestation to verify integrity of TEE. c. If the attestation is successful, the enclave agent obtains keys from the key broker service to verify and decrypt container images. d. The enclave’s images management components download container images, which are pulled and unpacked inside the confidential guest and then verified and decrypted. e. Enclave starts the container workload to run inside protected environment. 4.2.1.1 PRINCIPLE OF LEAST PRIVILEGE Principle of Least Privilege stands as a cornerstone of the security architecture, dictating that every entity, whether a human user, an automated process, or an individual machine—must be granted only the absolute minimum access rights required to perform its designated function, and no more. [68] Core Principle: Every entity human, automated process, or machine is granted only the minimum access rights necessary for its specific function. This strategy is crucial for reducing the attack surface and limiting potential damage from breaches. • Comprehensive Application: Principle of Least Privilege is applied across all layers of the system in the edge-cloud continuum, covering: o Human operators o Automated tasks o Individual computational nodes • Confidential Containers & AI Workloads: For confidential containers, especially those running AI workloads in environments like Kubernetes, Principle of Least Privilege is strictly enforced through: o Pod Security Standards (PSSs): These impose tight security requirements on pods and containers, such as: ▪ Running as non-root users. ▪ Preventing shared host network or IPC namespaces. ▪ Disallowing host filesystem mounts. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 39 ▪ Prohibiting the use of privileged or host ports. These restrictions significantly reduce the risk associated with containerized applications. o Service Accounts: These provide granular control over pod permissions by assigning roles and permissions precisely tailored to each pod's functions and its processes. • Zero Trust Networking: Security posture is further strengthened by integrating Zero Trust Networking principles: o Access is granted only after rigorous verification of both user and device credentials, regardless of origin. [69] o This approach eliminates ambient authority (e.g., direct root access) and ensures automated tooling does not implicitly extend its defined scope of authority. CoCo fundamentally incorporates the principle of least privilege to enhance security by restricting access to confidential workloads from OS, hypervisor and cloud/infrastructure providers. On the other hand, the privileges of each container and its processes can also be minimized, preventing them from accessing or affecting other containers or the host system, thereby maintaining operational performance. By enforcing these granular controls, CoCo limits container exposure and risk. 4.2.1.2 GAPS IN CONFIDENTIAL CONTAINER TECHNOLOGIES The current limitations on trusted container technology are: • Self-signed certificate authority certificates for the container registry are not supported, meaning the container registry must use public CA signed certificates. This means that only public CA can be used. • There is no support for encrypted container images. • The container image is downloaded and executed inside the CVM (TEE). Currently, this image is also downloaded on the worker node, meaning that there is a container image double pull. • Operating system support may be more limited. • Additional overhead. Running applications on confidential containers will have penalties on performance and energy consumption. • Confidential containers typically exhibit an increased pod startup time compared to standard pods. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 40 4.2.2 CONFIDENTIAL CONTAINERS ON EDGE PLATFORMS Confidential containers have specific requirements for the host platform. The hardware platform must provide sufficient memory and storage capacities, as well as support CPU architecture and operating system, which are supported by container technology. CoCo documentation [57] currently requires a minimum of 8GB RAM and 4 vCPU for the Kubernetes cluster node. Consequently, the CoCo project may not be directly applicable alternative for every situation and platform. In these cases, adaptations to the confidential application may be needed to enable a split architecture where only the most critical parts are executed in the secure world whereas the rest remain in the unsecure world. This subsection describes approaches for orchestrating confidential code into different resource restricted edge/IoT-environments. We explore applicability and practical challenges on different confidentiality strategies for orchestration on five alternative platforms. 4.2.2.1 ORCHESTRATION ON ARM Many ARM devices can support CoCo and thus in these cases the orchestration follows the same principles as in other platforms supporting CoCo. ARM provides two architecture generations supporting confidential computations: the newer Confidential Compute Architecture (CCA) on ARMv9-A onwards and the older TrustZone since ARMv6-A. The primary isolation approach for confidentiality in CoCo on ARM CCA is hardware-based virtualization. TrustZone provides a hardware-enforced separation between two worlds: the normal world and the secure world. CCA is designed to be applicable across various markets and devices that utilize ARM microprocessors, including edge devices. CCA introduces the concept of "Realms," which are hardware-protected execution environments, and which facilitates seamless portability of confidential workloads across ARM-based devices. Realms builds upon the foundations of older generation TrustZone architecture but offers a more dynamic approach to isolation. Realms can operate alongside ARM's TrustZone technology, minimizing the need for significant modifications to existing trusted applications. CoCo support for CCA in currently being developed [60][62]. In the recent release notes of CoCo (in version 0.8 [61]) CCA declared as untested or partially supported platform for CoCo with attestation. CoCo can work directly on TrustZone but this requires adaptation (e.g. running a very stripped-down container runtime within the Secure World of TrustZone or D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 47 The standard RA workflow, as architecturally outlined in RFC9334, involves an Attester (the TEE or the device hosting it) generating Evidence about its current state. This evidence typically includes cryptographic measurements of the hardware, firmware versions, and hash digests of the software stack running within the TEE. This evidence is then conveyed to a Verifier, which appraises it against predefined policies and a set of trusted reference values. Based on this appraisal, the Verifier generates Attestation Results. The Relying Party then utilizes these results to make an informed decision on whether to consider the Attester trustworthy and subsequently release confidential data or authorize sensitive operations. 4.3.3.2 TRUSTEE The CoCo Trustee project [77][78] provides a suite of tools and components designed for attesting confidential guests and securely delivering secrets to them. Although developed primarily for the CoCo project, its design allows for compatibility with a wide variety of applications and hardware platforms. Key components of the Trustee project include: - Key Broker Service (KBS) functions as the relying party in the RATS model, orchestrating the verification and release of sensitive information. - Attestation Service verifies the evidence provided by the TEE. In the RATS model, this component acts as the Verifier, assessing the authenticity and integrity of the attestation report. - Reference Value Provider Service manages the reference values that are used by the Attestation Service to verify TEE evidence. These reference values represent the expected, known-good state of the TEE and its software components. - KBS Client Tool is a utility designed for testing or configuring the KBS and AS, facilitating their setup and management. Within the confidential VM or enclave (the guest environment), two additional components play crucial roles: - Attestation Agent is responsible for obtaining the raw hardware evidence from within the guest TEE, which is then sent to KBS for verification. - Confidential Data Hub handles confidential operations within the guest, including initiating requests for secrets from KBS. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 48 4.3.3.3 SECURE DISTRIBUTED MULTI-DOMAIN ORCHESTRATION Software orchestration in 6G networks includes the distribution of control logic across different operator domains. To deliver seamless end-to-end services across different administrative and technological domains (belonging to different operators or service providers), the control logic needs to span these domains. This enables coordinated management of resources and services that involve multiple stakeholders. Secure cooperation between different domains can be based on different approaches. For instance, the zero-trust principle emphasizes the need to authenticate and authorize all cross-domain transactions. Techniques such as secure multi-party computation and distributed ledger technology can provide, respectively, confidentiality and integrity to the transactions. TEEs provide a complementary approach that can be used to protect the orchestration control logic when distributed across domains, ensuring confidentiality and integrity. Edge domain Confidential container Orchestrator Cloud domain Confidential container Orchestrator External administrative domain Confidential container E2E Orchestrator Administrative domain border Attested secure connection Attested secure connection Attested secure connection Figure 5. Attestation as an enabler for maintaining trust over multi-domain orchestration 4.4 SECURE DATA SHARING ACROSS MULTIPLE ENTITIES For enabling confidential data exchange in complex collaborative AI settings, the leveraging of Fully Homomorphic Encryption (FHE) and Secure Multi-Party Computation (SMPC) is paramount. These techniques allow joint computation over private inputs without ever disclosing the raw data, which is fundamental for applications like federated threat intelligence where multiple organizations collaborate on shared insights without revealing their proprietary data. [73] D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 49 Beyond the cryptographic protection of data in use, ensuring Data Provenance and Integrity is a critical security and trust component in distributed AI systems. Data provenance involves meticulously tracking the movements and transformations of each data point from its origin onward, creating a transparent, documented, and cryptographically bound record of its complete history. [74] This immutable trail is vital for preventing data tampering, designing inherently trustworthy systems, detecting anomalies or inconsistencies, and ensuring auditability and regulatory compliance. Provenance systems function by tagging newly digitized information with rich metadata, such as time, date, location, origination device type, and privacy rights, and cryptographically binding this information to the data. This metadata is then updated and rebounds at every subsequent transformation point throughout the data's lifecycle. Emerging technologies such as blockchains and other distributed ledgers [studied in CONFIDENTIAL6G Deliverable 4.2] are ideally suited for underpinning these tamper-proof provenance systems due to their inherent immutability and distributed consensus mechanisms, which create an unbroken chain of trust. Collaborative AI, particularly Federated Learning, inherently addresses the raw data sharing privacy problem but introduces a new set of complex challenges related to model integrity, communication efficiency, and data heterogeneity. This indicates that privacy-preserving collaboration in AI extends beyond simply not sharing raw data. It critically involves ensuring the integrity, authenticity, and trustworthiness of the model updates and the derived insights shared among participants. An effective orchestration framework for complex collaborative AI must therefore integrate advanced secure aggregation techniques (e.g., FHE, SMPC) to protect model updates during aggregation, optimized communication protocols (e.g., model compression) to mitigate bandwidth issues, and, crucially, robust data provenance mechanisms (potentially leveraging blockchain or distributed ledger technology [studied in CONFIDENTIAL6G Deliverable 4.2]) to cryptographically bind metadata to model updates and intermediate results. This allows participants to verify the origin, transformations, and integrity of the AI models and shared features, enabling detection and prevention of malicious training or data poisoning attacks. This moves the focus from merely "privacy" to a broader "trustworthiness of the entire collaborative AI pipeline". D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 50 Figure 6: Secure decentralized data sharing framework Secure decentralized data sharing framework [CONFIDENTIAL6G Deliverable 4.2] is as explained below: • Edge Layer: o This is the layer where the data is generated and consumed. It includes devices like IoT devices, user devices, etc. These devices interact with the system using secure protocols. • Confidential Networking Layer: o This layer is responsible for secure communication between devices and the system. It uses post-quantum cryptography and other enhanced cryptographic protocols to ensure the confidentiality and integrity of the data in transit. • Confidential Orchestration Layer: o This layer is responsible for managing the overall flow of data in the marketplace. • Computation Layer: D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 51 o This layer is where the computations on the data happen. It uses technologies like Secure Multi-Party Computation (SMPC), Trusted Execution Environments (TEEs), and Fully Homomorphic Encryption (FHE) to perform computations on encrypted data, ensuring the privacy of the data. • Application Layer • Data Ops: o This layer is responsible for managing the data lifecycle, including data acquisition, storage, and preprocessing. • DLT Layer D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 52 5 PROOF-OF-CONCEPTS AND EXPERIMENTS FOR CONFIDENTIAL ORCHESTRATION This section describes our experiences with the implemented concepts for confidential orchestration. 5.1 TESTING AND EVALUATION The process of testing is the evaluation of components taking into account specified functional platform requirements. The goal is to identify and rectify bugs, deficiencies, unexpected behaviour and to timely provide corrective measures. We should stress the fact that the testing procedures described in this deliverable will be performed on a technical level, by the development teams of the project themselves and not by real users. Testing in a pilot operation of the platform, falls within the scope of WP5 and will be elaborated there. However, we plan a small-scale testing operation in the context of WP4, which we describe in this deliverable. Testing is, usually, performed on two levels: i) Individual module ii) Integrated system testing. The former refers to the testing of the individual system components with respect to the requirements and the latter refers to the testing of the integrated system itself by assessing the interfaces among the modules and the involved data flow and the expected behaviour of the system “as a whole.” In order to effectively manage the complexity of the final system, testing will be initially performed on an individual module basis before the integration begins. After all modules are verified, then the integration will begin and when it is completed, a final set of tests will be performed mainly in the form of a small-scale pilot operation. 5.1.1 TESTING PROCEDURE DEVELOPMENT Our initial test plan is based on the following general considerations: • The adoption of a strategy to use. When testing the integrated modules and how the tests will be conducted (i.e., use cases). Our consortium has decided on the adopted methodology which is explained in this deliverable. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 53 • What will be tested - (e.g., software modules against set requirements). The modules that will be tested are described within the context of WP4. • What will be the time -frame of the testing (e.g., when it starts and ends). We have followed the task and WP sequencing as described in the DoA (WP4 and WP5). • Testing responsibilities among involved parties. These responsibilities have already been assigned and follow the corresponding module design and development responsibilities as described in the DoA, especially in WP4. • Pass and fail conditions for each performed test . The partners have proposed themselves the described tests for the modules they developed and thus, have identified accurately these conditions. • Potential risks involved when a test is performed. Actually, no privacy or other risk is foreseen for the described tests as they will be conducted in a restricted context and controlled environment by the involved partners. • Approval of the testing plan from all involved parties. All parties have agreed on the tests and the testing plan as described in this deliverable. 5.1.2 TESTING METHODOLOGY CONFIDENTIAL6G has decided to adopt an agile testing approach. Briefly, this approach considers testing and development as two intertwined phases and not sequential (i.e., testing following development). This decision was dictated by the tight delivery dates of the modules as well as the requirement for early integration before the use case testing of WP5 takes place. With respect to the practical, separate, and integrated module testing, the generic methodology proposed in will be followed. With respect to system testing, there are generally two main methodologies: (i) Incremental testing, (ii) Non-incremental testing Incremental testing includes several methods: • Bottom-Up approach – this method starts with testing the lower-level components, allowing early detection of issues. It needs drivers to simulate higher-level components that are not yet developed. • Top-Down approach – this method tests the highest-level components first, focusing on the system functionalities, data flows, and interfaces. It D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 54 minimizes the need for drivers but requires stubs to simulate unavailable lower-level modules. • Hybrid integration – this approach combines Top-Down and Bottom-Up testing. Lower-level modules are tested in parallel with their integration into the system, reducing the need for stubs and drivers but requiring more operators’ actions to ensure smooth operation. We will use the hybrid integration testing approach, in which lower-level modules are tested in parallel with their integration. This means that as modules are integrated, they are also tested simultaneously. Consequently, the need for stubs and drivers is minimized. However, this approach is a bit less systematic than the Top-Down and Bottom-Up approaches and may require more coordination effort. The testing procedures for the project components defined will be further described in a uniform way, by providing templates for each of the testing use cases. The se case templates are beneficial because they ensure consistency, clarity, and efficiency in documenting and executing test procedures. They guarantee that all use cases are documented in the same way, making them easier to read and understand. By using templates, relevant details are systematically gathered and organized, reducing the possibility of errors. The proposed template has two sections: Use Case Identification and Use Case Definition. The template was presented at CONFIDENTIAL6G Deliverable D4.2. 5.1.3 A TEST CASE FOR BASELINE CONFIDENTIAL ORCHESTRATION This test case describes the baseline scenario for orchestrating confidential workloads. Table 6. Secure deployment of application using confidential containers – use case identification Use Case ID: VTT: CoCo:01 Use Case Name: Use Case 1 – Deployment of security critical application using Confidential Container and Kubernetes Created By: Andrea Dalla Costa, Jani Suomalainen, Markku Kylänpää Last Updated By: Date Created: 03.06.2024. Date Last Updated: D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 55 Table 7. Secure deployment of application using confidential containers – use case definition Actors: Confidential application, Kubernetes operator (the end user making tests), infrastructure provider, infrastructure operator Description: This test should ensure that confidential application is correctly scheduled and executed on the target Kubernetes node with support for TEE. Deployment is done by a Kubernetes operator (the user) to the trusted infrastructure, which is provided by the infrastructure provider. The infrastructure operator cannot access the application. Optionally, the target platform integrity can be remote attested before deployment. Trigger: Confidential workload is set to be deployed. Kubernetes and confidential containers are used. Preconditions: A Kubernetes cluster has been setup with at least one node providing support for confidential containers. The Kubernetes template that describes the confidential workload is properly configured, with the correct execution runtimes selected. Postconditions: The workflow is successfully scheduled and deployed to a node with TEE support. The confidential application is running correctly. Normal Flow: 1. Workload Preparation - building of container images using standard tools (e.g. podman or Docker) 2. Kubernetes template preparation 3. Kubernetes deployment and scheduling towards hosts that support running confidential containers (applying templates describing the container and run time) 4. Execution flow in TEEs is the following. a. Confidential container runtime initiates VM TEE and launches a confidential virtual machine. b. Optional: an agent residing in newly launched enclave performs remote attestation to verify integrity of TEE. c. Optional: If the attestation is successful, the enclave agent obtains keys from the key broker service to verify and decrypt container images. d. The enclave’s images management components download container images, which are pulled and unpacked inside confidential guest and then verified and decrypted. e. Enclave starts the container workload to run inside protected environment. Alternative Flows: Exceptions: Attestation verification errors (should lead to deployment being aborted). Infrastructure errors (like nodes going offline) or resource exhaustion may prevent orchestration. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 56 Includes: Special Requirements: Legal Considerations: 5.2 CONFIDENTIAL CONTAINERS FOR FEDERATED LEARNING ON EDGE AND IOT PLATFORMS To understand and improve the feasibility of CoCo on embedded, resource restricted platforms, we built a test environment and profiled computing costs and energy consumption of a critical AI application. The measurement results alongside with an orchestration approach (described in Section 5.3), have been demonstrated at the Democompo’2025 event at Oulu, Finland on May 2025 [75], at EuCNC 2025 at Poznan, Poland on June 2025 [76], and at HAIC Secure Systems Demo Day 2025 at Espoo, Finland on June 2025 [79]. The demonstrations were accompanied by a poster Confidential Containerized Federated Learning for Distributed Security . 5.2.1 THE USE CASE As the background use case, we considered privacy threats against federated learning. FL is a prominent AI solution for private critical data but vulnerable for malicious aggregating nodes. In FL local nodes do not share their privacy critical data but gradients which are used by an aggregating node to create a global model. However, in the Improved Gradient Leakage attack (iDLG) [47] an adversary is able to reconstruct the private data from shared gradients. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 63 nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: security.level operator: In values: - high - weight: 20 preference: matchExpressions: - key: energyefficiency operator: Gt # higher value means more efficient values: - "7" containers: - name: C6G-container image: C6G-image Taints and Tolerations [37] control which pods can be scheduled on specific nodes. A taint is applied to a node and repels any Pod that does not have matching toleration. Tolerations are specified in the Pod specification and allow a Pod to be scheduled on a node with a matching taint. Kubernetes components for collecting metrics include kubelets , which run on each worker node and collect resources usage statistics, and the metrics server , which aggregates usage data in a cluster. Kube-state-metrics provides state information on Kubernetes objects form the API server. 5.3.2 SCHEDULER CUSTOMIZATION The default scheduler has certain limitations. It primarily relies on static configurations and resource requests, which may not accurately reflect the dynamic resource demands of applications at runtime. It lacks built-in capabilities to directly incorporate real-time runtime metrics, such as actual CPU utilization or network latency, into its decision-making process. Also, the default scheduler typically runs as a single instance, which can become a bottleneck in large clusters with a high volume of scheduling requests. The Kubernetes scheduling framework [39] enables development of customized scheduling logic. It is based on a pluggable architecture, which allows scheduling features to be implemented as plugins, which can be connected to several extension points. Registered plugins are then invoked during the scheduling D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 64 process to offer granular control over orchestration. Scheduling plugins access runtime metrics by integrating with external monitoring tools. 5.3.3 PRIORITIZATION BASED ON METRICS FOR CONFIDENTIAL COMPUTING To drive the orchestration based on the features and quality of the confidential computation approach, we defined the following metrics. These metrics can be used to label the infrastructure nodes with TEE. Table 10. Metrics for Confidential Container security and example levels Category Metric Levelling example Confidential container CoCo architecture can be process or VM based. With VM based the isolation may be stronger as HW-backed virtualization provides more robust security boundary. With VM-based the trusted computing base is also typically smaller. Process based isolation trade these security advances for performance. 0 – no CoCo support 1 – process based containerization 2 – VM based containerization Isolation Strength of isolation between TEE from REE. Measurement is typically based on SW and HW analysis by a trusted party. 0 – unknown, weak 1 – mediocre 2 – strong Memory isolation – How memory of trusted application is secured from other processes. The isolation strength depends on threat model and assumptions. 1 – HW based memory isolation (partitioning) 1 – Enclaves (memory isolation within user space) 1 – Memory encryption 1 – Memory integrity protection 2 – Combination (to prevent logical and physical memory attacks) CPU isolation – How TEE’s execution is protected from interference of other processes. The isolation strength depends on threat model and assumptions. 2 – Hardware partitioning 2 – Enclave 2 – Memory encryption Remote attestation Attestation completeness - what aspects of the TEE's state are included in the attestation report. 1 – operating system 2 – software components, configurations 3 – software configurations data Attestation soundness – strength of the cryptographic proof 1 – use of mediocre cryptographic algorithms 2 – use of strong cryptographic algorithms Attestation freshness - Ensuring the attestation report is recent and not a replay of a previous state 1 – before deployment, replay protection 2 – frequently 3 – before each (configuration) transaction D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 65 Sidechannel attack mitigation Mitigating application fingerprinting – Alternative mechanisms can be prevented to prevent side-channels and, e.g., other processes from observing system call patterns. Effectiveness of the method depends of the adversary and platform. 0 – no mitigation 1 – system call obfuscation (e.g. wrappers) 1 – system call timing randomization 1 – constant time execution 1 – Sandboxing allowed calls 2 – Combination of mitigations Vulnerability analysis Known vulnerabilities vs. maturity 0 – no analysis (unmature)/ known (unpatched) vulnerabilities 1– no known vulnerabilities in mature TEE product Security audited and penetration tested 0 – no testing 1 – passed tests by a trusted party Security certification Common criteria evaluation – where assurance level indicates the depth and rigor of evaluation 0 – no certification 1 to 7 according to EAL level Other certification 0 – testing 1 – trusted certification Kubernetes scheduling logic can then be built by reflecting above metrics against the operator’s requirements. As illustrated in Figure 10, the prioritization is done by selecting a) which features are mandatory and must be fulfilled with some minimal level, and b) which guide the prioritization and what are the weights for the prioritizing features. Prioritization value is the sum of weight multiplied level for required node label. In addition to considering the confidentiality parameters, real-world implementation must also include sustainability and other quality and performance relevant parameters into prioritization decision. For instance, the time overhead and energy consumption profile values, which were measured in Section 5.2, can be directly used as levels that are included to the priority value. Mandatory requirements Prioritize node Are there labels under minimum level? Priority=0 Yes Weighted preferences Security & sustainability labels Label, level Requirement Priority=⅀(level*weight) Figure 10. Prioritization logic D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 66 5.4 SIDE-CHANNEL ATTACKS ON CONFIDENTIAL CONTAINERS Figure 11. System event patterns for two containers hosting two distinct applications. Each arrow represents a system event (either a syscall or a capability), and arrows of the same colour reflect the same system event (e.g., the ‘write’ syscall or the ‘CAP_SYS_CHROOT’ capability). 5.4.1 CONTEXT In recent work at TID, we identified a novel side-channel attack targeting confidential containers running microservices. As shown in Figure 11, containers interact with the “outside world” in two ways: (i) over the network with other containers or cloud services, and (ii) through the host OS kernel. While the first interaction is typically encrypted and has been the focus of extensive traffic analysis research (e.g., [56]), the container–kernel interface remains largely unexamined and is the focus of our investigation. Our hypothesis was that system events (e.g., system calls and capabilities) sent via the container-kernel interface can constitute a unique fingerprint of the application. Crucially, these system events can be accurately collected from outside the confidential container, without introducing overhead or being detectable. The insights gained from this fingerprinting attack could ultimately enable adversaries to conduct more effective, efficient and stealthy attacks. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 67 5.4.2 THE ATTACK We formulate our fingerprinting attack as a multi-class classification problem, aiming to identify which Docker image is running inside a confidential container based on the system events it triggers. The attack proceeds in two distinct phases: an offline phase, executed prior to the victim container’s launch, and an online phase, carried out while the victim’s confidential container is actively running. In the offline phase, the attacker collects system events generated during the execution of various Docker images within confidential containers. The Docker images used in the offline phase may be attacker-generated, obtained from public repositories (e.g., Docker Hub), or compiled from publicly available source code. The resulting system event traces are used to train a machine learning (ML) model, where each Docker image corresponds to a distinct class. This trained classifier is then used in the online phase to perform real-time fingerprinting. In the online phase, the attacker collects the system events generated by the victim container (this assumes co-location between the victim and the adversary, a problem that falls outside the scope for this work) and feeds them into the pretrained model in order to infer the most likely Docker image running inside the confidential container. It is important to note that most of the computational effort is carried out during the offline phase, while the online phase serves primarily as an inference step in our evaluation. Our attack relies on two types of system events: system calls and capabilities. For each container trace, we extract two feature types: (i) binary features indicating the presence or absence of each event, and (ii) frequency features capturing the frequency of each event. To enhance model generalization and minimize noise, we performed feature importance analysis, allowing us to discard irrelevant or lowcontribution features. This process significantly reduces the dimensionality of the feature space, from 353 to 44 for system calls, and from 44 to 21 for capabilities. We then evaluated several ML models using 3-fold cross-validation combined with a grid search for hyperparameter tuning. While all models performed reasonably well, the Random Forest classifier achieved the best performance. In the following, we present detailed results, focusing on those obtained with the Random Forest classifier. For our experiments, we constructed a dataset consisting of 150 official Docker images sourced from Docker Hub. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 68 5.4.3 RESULTS Table 11 reports the accuracy, precision and F1-score of our trained Random Forest ML model for fingerprinting applications running within the TEE. The results are presented for Docker images executed both with and without Docker parameters, and for two feature types: binary or frequency-based vectors. The evaluation is structured in two parts: the first two rows show results using system call data, while the last two rows correspond to results based on capability data. Table 11. Fingerprinting accuracy, precision, and F1-Score As shown in the first two rows of Table 11, the trained ML model achieves significantly higher accuracy, precision, and F1-score when using frequency-based feature vectors, for both system calls and capabilities. The results also suggest that fingerprinting is slightly more effective when applications are executed with their corresponding Docker parameters. Overall, these findings demonstrate that system events invoked by applications inside confidential containers can leak information, enabling adversaries to accurately infer which application is being executed. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 69 6 CONCLUSIONS AND FUTURE WORK Orchestration plays a vital role in the management of 6G networks. Its integration with the confidential computing paradigm provides cost-efficiency with security, enables new use cases, and facilitates cooperation between operators, end-users, and infrastructure providers. We have worked on the different aspects of orchestration and containerized confidentiality by exploring the major security threats, assessing feasibility and adaptability of orchestration on heterogeneous infrastructure of confidential continuum, as well as considering impacts of emerging AI technologies. This report analysed the threats and the requirements coming from the 6G use cases, discussed architectural blueprints, and documented our experiments and demonstrations for confidential orchestration. Main observations, key takeaways, and future research recommendations: • CoCo is a maturing project which brings confidential workflow automation easily available for various TEE-capable platforms. Our experiments highlight CoCo’s feasibility also on ARM based edge devices with reasonable performance penalties. However, for many platforms CoCo is not yet available, and it remains to be seen how well the industry and open-source community are able to develop optimized solutions to support trustworthy containers in lightweight manner. • The proposed K8s scheduling approach highlighted the need to also consider the parameters of confidential computing when orchestrating and optimizing the performance of 6G services. To support prioritization in the scheduling process we defined metrics for confidentiality. AI and data-driven orchestration will be an important part of the future networks; however, the optimization problem is not easy task. Future work is needed to develop and learn models that can capture operators’ requirements for security and performance as well as run-time parameters from platforms and to transform them to automated resource allocations, prioritizations, and security configurations. • The novel side-channel attack presented in this work demonstrates that the interactions of confidential containers with the “outside world” (which are not protected by default) can inadvertently reveal application-level information that should remain confidential. Further research is required to assess the implications of such fingerprinting attacks across a range of 6G use cases and to develop effective mitigation strategies. Particular attention should be D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 70 given to identifying which countermeasures are practical, considering the trade-offs between performance, security & privacy and deployment complexity. • Confidential container solutions can be the backbone for secure orchestration of federated AI, integrating TEE-based model execution and DP noise injection to prevent leakage. Future work should cover designing of confidential AI pipelines that optimize trade-offs between privacy, security, and efficiency in multi-party federated learning. This task studied the enablers for confidential orchestration. Future work is needed on studying their application for different technologies to be orchestrated and for different use cases. Within CONFIDENTIAL6G, the work continues in Task 4.4, which studies orchestration of confidential AI workloads, as well as in the use case 1 of WP5, which studies confidential orchestration in the context of an airline application scenario. D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 71 REFERENCES [1] Jani Suomalainen, Kimmo Ahola, Mirko Sailio, Gabor Kiss, Gabor Megyaszai, Rizwan Asif, Petri Jehkonen, Jonathan Rivalan. Tactical Orchestration - Network, Security, and Drone Intelligence for Mission-Critical Operations. Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), pp. 1061-1066, Antwerp, Belgium, 3-6 June 2024. https://doi.org/10.1109/EuCNC/6GSummit60053.2024.10597035 [2] Pérez Palma, N., Skarmeta Gómez, A. ja Bisson, P. (editors) (2025) Innovative Approaches for 6G Security: Challenges, Solutions, and Impact. 6G-SNS-IA Security Working Group. White paper. doi: 10.5281/zenodo.14619619. [3] Ahmad, A., Schultz, A., Lee, B., & Fonseca, P. (2023). An extensible orchestration and protection framework for confidential cloud computing. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) (pp. 173-191). [4] Confidential Containers. Project site. Accessed March 13th, 2025. https://confidentialcontainers.org/ [5] Confidential Containers. Github site. Accessed March 13th, 2025. https://github.com/confidential-containers [6] David Hadas, Pradipta Banerjee, Jens Freimann, Ariel Adam. Deployment considerations for Red Hat OpenShift Confidential Containers solution. Red Hab Blog, 2024. Accessed March 13th 2025. https://www.redhat.com/en/blog/redhat-openshift-confidential-containers-deployment-solution [7] Ariel Adam, Pradipta Banerjee. Introducing Confidential Containers Trustee: Attestation Services Solution Overview and Use Cases. Red Hat Blog, 2024. Accessed March 13th 2025. https://www.redhat.com/en/blog/introducingconfidential-containers-trustee-attestation-services-solution-overview-anduse-cases [8] CoCo Project. Confidential Computing Overview. Accessed March 13th, 2025. https://github.com/confidential-containers/confidentialcontainers/blob/main/architecture.md [9] Kubernetes. Cluster Architecture. 2025. Accessed March 13th, 2025. https://kubernetes.io/docs/concepts/architecture/ [10] Popa, R. A. (2024). Confidential Computing or Cryptographic Computing? Tradeoffs between cryptography and hardware enclaves. Queue , 22 (2), 108132. [11] Johnson, M. A., Volos, S., Gordon, K., Allen, S. T., Wintersteiger, C. M., Clebsch, S., ... & Costa, M. (2024). Confidential Container Groups: Implementing confidential computing on Azure container instances. Queue , 22 (2), 57-86. [12] Johnson, M. A., Volos, S., Gordon, K., Allen, S. T., Wintersteiger, C. M., Clebsch, S., ... & Costa, M. (2023). Parma: Confidential Containers via Attested Execution Policies. arXiv preprint arXiv:2302.03976 . D4.3 Confidential Orchestration CONFIDENTIAL6G Horizon-JU-SNS-2022 - G.A. 101096435 72 [13] Edgeless Systems. Blog entry, 2022. The differences between Homomorphic Encryption (HE) and Confidential Computing (CC), accessed April 10, 2025, https://www.edgeless.systems/blog/the-differences-between-homomorphicencryption-he-the-differences-between-homomorphic-encryption-he-andconfidential-computing-cc [14] Mohammed, M. A., Garcia-Zapirain, B., Nedoma, J., Martinek, R., Tiwari, P., & Kumar, N. (2022). Fully homomorphic enabled secure task offloading and scheduling system for transport applications. IEEE Transactions on Vehicular Technology , 71 (11), 12140-12153. [15] Kubernetes. Security. Web documentation, accessed April 11th 2025. https://kubernetes.io/docs/concepts/security/ [16] Kubernetes. Network Policies. Web documentation, accessed April 11th 2025. https://kubernetes.io/docs/concepts/services-networking/network-policies/ [17] SentinelOne. Kubernetes Security Risks: 10 Vulnerabilities You Need to Know. Web article, 2024. Accessed April 11th 2025. https://www.sentinelone.com/cybersecurity-101/cloud-security/kubernetessecurity-risks/ [18] Kaliappan, V. K., Yu, S., Soundararajan, R., Jeon, S., Min, D., & Choi, E. (2022). High-secured data communication for cloud enabled secure docker image sharing technique using blockchain-based homomorphic encryption. Energies , 15 (15), 5544. [19] Ariel Adam, Tanay Baswa, Pradipta Banerjee, Suraj Deshmukh, Jens Freimann, Magnus Kulke, Prashanth Harshangi. Protecting your intellectual property and AI models using Confidential Containers. Red Hat Blog. Accessed April 11th 2025. 2023. https://www.redhat.com/en/blog/protecting-yourintellectual-property-and-ai-models-using-confidential-containers [20] Red Hat OpenShift. Web page. Visited 14th April 2025. Available: https://www.redhat.com/en/technologies/cloud-computing/openshift [21] HashiCorp Nomad. Web page. Visited 14th April 2025. Available: https://www.nomadproject.io/ [22] Docker. Swarm mode. Web page. Visited 14th April 2025. Available: https://docs.docker.com/engine/swarm/ [23] Amazon Elastic Container Service. Web page. Visited 14th April 2025. Available: https://aws.amazon.com/ecs/ [24] Azure Container Instances. Web page. Visited 14th April 2025. Available: https://azure.microsoft.com/en-us/products/container-instances [25] Google. Cloud Run. Web page. Visited 14th April 2025. Available: https://cloud.google.com/run [26] Apache Mesos. Web page. Visited 14th April 2025. Available: https://mesos.apache.org/ [27] Rancher. Web page. Visited 14th April 2025. Available: https://www.rancher.com/