i-CORA - A Large-Scale Experimentation Platform for End-to-End 5G Services

1 i-CORA - a Large-Scale Experimentation Platform for End-to-End 5G Services Min Xie∗, Muhammad Faheem Awan ∗, Jane Frances Pajo ∗, Abdelhakim Cherifi ∗, Eric Lajoie †, John Gill ‡ ∗Telenor Research and Innovation, Telenor, Norway; †EMEA at RedHat, Germany; ‡EMEA at Emblasoft, France Abstract—The development of 5G and Beyond 5G (B5G) technologies relies on the availability of experimentation facilities that can evaluate and validate the performance of these technologies. It is of great interest and challenge to design, deploy and operate large-scale experimentation platforms to meet the high requirements of various vertical use cases for the 5G services. This paper describes an iCORA platform that we build with multiple partners in Norway to support several EU-funded projects (5GMediaHUB, IMAGINE-B5G, FIDAL and COMMECT) and vertical use cases. The platform is cloud-native and consists of four parts: a multi-vendor end-to-end 5G network with three RAN sites serving general use cases, two mobile private networks (MPNs) and three Networks on Wheels (NOWs) serving dedicated verticals, and an open source platform composed of open source solutions. i-CORA offers both advanced standalone 5G services and value-added services (e.g., security and testing) to verticals in Public Protection and Disaster Relief (PPDR), media, eHealth, Industry 4.0, etc. In this paper, we address the challenges and lessons learned during the implementation and operation of the iCORA platform. I. INTRODUCTION As the mobile network development evolves with advanced virtualized, distributed, intelligent, and cloudnative technologies, it becomes necessary to build experimental platforms capable of assessing and validating these advanced technologies [1]. On one hand, the platforms should be designed and operate to offer the fundamental 5G network services, e.g., network slicing services. On the other hand, they should be open to new technologies (e.g., new radio, Open-RAN), allowing continuous evolution from 5G to Beyond 5G (B5G) and 6G in a sustainable way. Moreover, the experimental platforms are expected to accommodate multiple vertical use cases at scale [2], which significantly complicates the network operations and service provisioning. The resulted high complexity requires automation to deal with the lifecycle management (LCM) of various services, from the infrastructure and network services to vertical applications. To meet these requirements, we design and build a large-scale experimental i-CORA (innovative, Cloudnative, Open, Robust, and Automated) platform to support the research and innovation demands from EU projects, vertical use cases, and mobile operators’ business areas. Evolved from the 5G-VINNI project [3], i-CORA is a multi-vendor platform built in a cloud-native environment. It consists of both (near-)commercial and open source solutions supplied by more than a dozen of partners. It not only offers end-to-end (E2E) 5G services across multiple geographical locations in Norway, but also interconnects with other European experimental platforms (e.g., in Greece and Spain) to support cross-facility use cases (UCs). More importantly, it is equipped with a fullstack management and orchestration (MANO) system and able to automate the lifecycle management of network services and vertical applications. Particularly, considering the seemingly contradictory requirements from different vertical UCs, the platform set up two stages with separate infrastructures to serve general UCs and specific UCs individually. General UCs aim to validate the vertical applications and/or KPIs, and thus require a reliable testbed with fundamental 5G services. Specific UCs target at validating advanced and innovative technologies and solutions, which requires flexible and open testing environments. The iCORA platform serves multiple EU projects, through which we co-create and experiment vertical UCs in PPDR, Healthcare, Industry 4.0, Media, Agriculture, Aquaculture and Forestry. The platform is expected to validate the KPIs required by these UCs and the new cutting-edge technologies and solutions. To meet these requirements, i-CORA is evolving and expanded in a continuous and sustainable way to improve the service performance and enhance the platform capabilities. Consequently, the platform becomes increasingly complex, which challenges both the operations and management of the platform, and the service provisioning and assurance for vertical UCs. Throughout this process, we gained experiences and learned lessons. In this paper, we first describe how the i-CORA platform is designed in Section II. The details on the infrastructure and deployment of the key i-CORA components are presented in Section III, followed by the operations and offered services in Section IV. The challenges and lessons learned are summarized in the concluding Section V. II. I-CORA ARCHITECTURE The i-CORA platform serves EU projects 5GSOLUTIONS [4], FUDGE-5G [5], 5G-HEART [6], 5GMediaHUB1, FIDAL2, IMAGINE-B5G3, 1https://www.5gmediahub.eu/ 2https://fidal-he.eu/ 3https://imagineb5g.eu/ 2 COMMECT4, and 5G-EMERGE5. It consists of four parts: a main multi-vendor platform (MVP) offering an E2E public 5G network; a set of Mobile Private Networks (MPNs); a set of Networks on Wheels (NOWs); and an Open Source Platform (OSP) (as shown in Fig. 1). The Multi-Vendor Platform (MVP) is built by multiple vendors and acting as a public 5G network to offer E2E 5G services to all general UCs. It includes the radio access networks (RAN), the transport network (TN), and the 5G standalone (SA) core network (CN). In addition, the MVP also provides full-stack management and orchestration as well as professional security and testing services. Currently, there are three operational RAN sites, supplied by Huawei and Ericsson, and located in Fornebu (outdoor), Trondheim (indoor), and Svalbard(outdoor) in Norway. Another four sites are under deployment to improve the coverage and/or extend the service into the University of Oslo (UiO), Fornebu (indoor), and Trondheim (outdoor). The CN is deployed in the Telenor headquarter in Fornebu. It explores the service-based architecture (SBA) and is jointly supplied by three vendors: CASA systems, ENEA, and Oracle. This unique multi-vendor CN not only avoids the vendor lock-in but also takes advantages of the SBA, allowing the addition of new cloud-native network function (CNFs) from new vendors to enhance the existing CN. The interconnection between the RAN sites and CN is realized in three ways, depending on the location of the RAN sites: i) via direct cabling for the Fornebu site which is close to the CN infrastructure; ii) via a commercial TN service (Nordic Connect) provided by Telenor for the Trondheim and Svalbard sites; iii) via the Uninett/SIKT for the UiO site to support the pan-European G´ EANT network for research and scientific excellence. The Fornebu infrastructure is also interconnected with external parties via Uninett/SIKT, including public clouds (e.g., Oracle Cloud Infrastructure (OCI) and Amazon Web Services (AWS)) and other EU experimental platforms in Greece and Spain. Open Source Platform (OSP) is co-located with the near-commerical MVP in the Fornebu infrastructure. It is built to investigate the open source solutions and their performance. The OSP is cloud-native, based on the open source Kubernetes Microk8s for the infrastructure orchestration and open source MANO (OSM) for orchestrating the network services (NSs) and the UC-specific vertical applications. The vertical applications in the OSP can consume the 5G services from two sources: i) the E2E 5G services provided by the MVP; or ii) the open source 5G services provided by an open source Open Air Interface (OAI) solution, including the SA 5GC and emulated gNB. Note that there are many open source solutions available. The main reason to deploy an OAI 5GC is to enable a smooth evolution towards O-RAN as OAI also provides an open-source O-RAN solution, which can be conveniently integrated with its 5GC. 4https://www.horizoneurope-commect.eu/ 5https://www.5g-emerge.com/ Both the OSP and the MVP can interoperate with each other via Open APIs, and with external components, e.g., the experimentation tools in 5GMediaHub, or customer portal in IMAGINE-B5G and FIDAL. Network on Wheels (NOW) is a standalone and fully autonomous solution to provide on-demand 5G network coverage for dedicated UCs. It integrates the RAN, CN, and vertical applications as an one-inall mobile solution that can be transported easily. It can be used in remote areas either without coverage or with damaged connectivity infrastructure caused by natural disasters. The NOW is quick to deploy, simple to operate, secure and ruggedized. The i-CORA platform has two transportable NOWs on trucks (NOW1and NOW2) and one portable NOW1.5 that can be carried to fit in any vehicle. The three NOWs offers advanced services like coverage on demand with guaranteed Quality of Service (QoS) and edge computing to UCs in PPDR, forestry and media. Mobile Private Networks (MPNs) have indoor coverage and are deployed at different sites in Oslo, Norway, to support specific vertical UCs, such as eHealth in the Rikshospitalet and Industry 4.0 in the ABB premise. These MPNs are equipped with the 5G New Radio (NR) while the CN and vertical applications are deployed on the edge server(s) to provide specialized vertical services. For example, in the eHealth UC, a secure communication service is created for doctors, nurses, and paramedic staff to transmit the sensing data collected by sensors that attach to patients and monitor the patients’ medical conditions. The sensing data needs extreme security to protect the patients’ privacy while being transmitted over the NPN 5G to be presented on a dashboard. The overview of all RAN sites, the corresponding CNs and UCs is summarized in Table I. Note that NOWs and MNPs are currently isolated from the MVP and OSP and their use is manually scheduled. But later NOWs and MNPs will interoperate with MVPs and/or OSPs and connect with a unified customer portal such that UCs can order services of the four parts via a single interface. Then their use may be managed via a common operation system support/business system support (OSS/BSS). III. I-CORA INFRASTRUCTURE Since NOWs and NPNs have isolated and dedicated infrastructure, this section focuses on the common infrastructure, shared by MVP and OSP (Fig. 2). The main infrastructure spreads across multiple locations in Fornebu, Trondheim, and Svalbard. It comprises hardware from world-class suppliers, including around 60 servers from Intel, HPE, and Nokia; and a dozen of networking equipment from HPE, Nokia, and Dell. Since i-CORA supports multiple EU projects simultaneously, a bullet-proof networking plan is designed to securely separate these projects from each other both physically (hardware isolation) and logically (different VLANs and security zones for UCs within the same project). Furthermore, some components can be deployed in public clouds such as AWS and OCI and interconnected with i-CORA seamlessly via Site-to-Site virtual private network (VPN), which creates more flexible deployment options. 3 Figure 1: iCORA Architecture Site Freq (GHz) BW(MHz) TX and RX mode HW SA Core Use Cases Fornebu 3.3-3.4 100 64T64R Huawei Multi-vendor All Svalbard 3.7-3.8 100 64T64R Ericsson Multi-vendor PPDR, Forestry Trondheim 3.61-3.7 90 64T64R Ericsson Multi-vendor Robots; Smart City; Media NOW1 3.3-3.4 100 64T64R Huawei Athonet PPDR; Media; Forestry NOW2 3.3-3.4 100 64T64R Nokia Azure Core PPDR; Media; Forestry NOW1.5 3.3-3.4 100 4T4R Huawei Cumucore PPDR; Media; Forestry ABB MPN 3.3-3.4 100 4T4R Nokia Cumucore Industry 4.0 Riks MPN 3.3-3.4 100 4T4R Nokia Fraunhofer Fokus eHealth Table I: i-CORA overview: RAN sites (all in the TDD duplex mode and in the C-band), core type, and UCs Edge-Prep(4)Edge-Prep(4) Firewall Switches Exp Service Cluster(6) Management Cluster 1 (9) Secure Service Cluster (8) Application Cluster (7) Management Cluster 2 (13) VMs (3) SIKT Nordic Connect Open Source Platform (8) OpenShift (4.12) Public Cloud Other EU Platforms Svalbard Trondheim Oslo OpenShift (4.10) OpenShift (4.12) OpenShift (4.10) OpenShift (4.12) • URLLC slice • Customized slice • Security agents • URLLC slice • Customized slice • Security agents • eMBB slice • Security agents • eMBB slice • Security agents • ACM • AAP • NFVO • Security controller • ACM • AAP • NFVO • Security controller • E2E-SO • Customer portal • E2E-SO • Customer portal • Vertical Apps • Testing tools (container) • Vertical Apps • Testing tools (container) • Svalbard SNO • Trondheim SNO • xxx SNO • Svalbard SNO • Trondheim SNO • xxx SNO • OSM • OAI 5GC • Vertical Apps • OSM • OAI 5GC • Vertical Apps • VM-based MANO • VM-based testing tools • VM-based MANO • VM-based testing tools Openshift Microk8sSNO KVM Figure 2: i-CORA Fornebu Infrastructure for the MVP and OSP: the number of servers is indicated in the brackets of each cluster. 4 Security is essential in i-CORA. We use Palo Alto NextGeneration Firewalls to secure the infrastructure and Palo Alto Prisma to secure the cloud-native environment, which provides container protection and visibility. Ultimately, we create a Zero-Trust environment which protects the work of UCs and partners from external threats. The physical infrastructure is virtualized into cloud native clusters, supported by RedHat OpenShift. Given the diverse requirements of platform management and operation, multiple OpenShift Container Platform (OCP) clusters have been created to serve different purposes. Two separated service clusters are created to serve different types of UCs. The Secure Service Cluster contains a reliable and stable enhanced mobile broadband (eMBB) slice to serve general UCs requiring fundamental 5G services. The eMBB slice is connected with the real gNodeB (gNB) and consumed by multiple UCs concurrently. The main KPIs for this cluster are typical quality of service (QoS) metrics (e.g., throughput, latency, and packet loss rate) as well as reliability and robustness, i.e., the slice should be available for as long as possible. On the other hand, the Experimental Service Cluster aims to provide multiple experimental slices (e.g., an ultrareliable low latency communications (URLLC) slice for mission-critical communications and a customized slice for verticals with specific applications) to explore new features, functions, and services for specific UCs. Therefore, it is not mandatory to guarantee reliability and robustness. Instead, the experimental slices may be created and terminated frequently to test different new components. By default, these slices are connected with emulated gNB for function tests. In case the new technologies are proven to be mature,reliable, and useful, the slice may be migrated into the Secure Service Cluster and connected with real RANs to serve general UCs. The Management Cluster is to host the MANO components, such as RedHat Advanced Container Management (ACM) and Ansible Automation Platform (AAP), Nokia Cloud Operations Manager (NCOM) and Nokia Orchestration Centre (NOrC), and Palo Alto Prisma controller. Since these MANO components support different OCP versions, two Management clusters are created, with OCP4.10 and OCP4.12, respectively. The Application Cluster serves the vertical applications and other value-added services, e.g., the testing tools. These applications can consume the services provided by the two Service Clusters. The Edge-Prep is a temporary infrastructure for validating the edge clusters before they are shipped to the premise. This infrastructure emulates the on-prem physical environment (e.g., devices and networking connectivity) to prepare for the infrastructure deployment and automation. It also validates the functionality of the designed edge (e.g., local breakout or autonomous edge) such that the edge devices can be deployed to operate immediately upon arrival. Since most edge clusters are based on one single and powerful server, Single-Node OpenShift (SNO) is deployed on the EdgePrep infrastructures. The OSP is based on open source Microk8s and used to deploy open source components like OSM, OAI 5GC and OAI emulated gNB. This cluster has limited access to the two Service Clusters and is selectively accessible by external entities. Although i-CORA is cloud-native, we maintain a VM infrastructure since some tools or components are still VM-based, e.g., testing tools from Emblasoft and the cross-domain service orchestrator from 5G-SOLUTIONS. Depending on the applications, these VMs are controlled by different hypervisors and connected to specific components in the Service clusters or Management clusters. IV. ICORA SERVICES AND USE CASES By combining the services produced by one or more clusters, we offer a series of services to vertical UCs. A. 5G Network Slicing Services Network slicing is a fundamental service offered by iCORA in the MVP. The MVP can create and manage multiple network slices in the Experimental Service Cluster. As shown in Fig. 3, the network slice is designed in a way that the control plane CNFs are shared by more than one slices whereas the user plane function (UPF) is dedicated to individual slices. The network slice selection function (NSSF) will select the appropriate network slice based on the UE requirement. More slice design options will be investigated in the future, e.g., reserve dedicated unified data management (UDM) and unified data repository (UDR) for a slice to protect the data integrity. The advantage of Fig.3 is the use of SBA that allows new SBA-compliant CNFs to be added into and extend the CN easily and flexibly. At present there is ongoing work on introducing security edge protection proxy (SEPP) to enable cross-operator roaming services. Later network exposure function (NEF) and network data analytics function (NWDAF) will be introduced to explore the potential of advanced AI from 3rd party applications. B. Automation i-CORA is specialized in its comprehensive MANO, as shown in Fig. 4. The i-CORA MANO has a multi-layer hierarchy, including the RedHat container infrastructure orchestration ACM and AAP, and Nokia NCOM as NFVO (NFV Orchestrator) and NOrC as an E2E service orchestrator (E2E-SO). The AAP is responsible for managing the containers and automating the deployment of CNFs or tasks via Ansible playbooks. The Nokia orchestrators NCOM and NOrC are compliant with the ETSI MANO architecture [7]. NCOM is responsible for the LCM of network services (NS), composed of one or more NFs, including CNFs and VNFs; whereas NOrC takes services produced by network domains of RAN, TN, and CN to create E2E services and offer them to vertical customers. The orchestration system enables the deployment automation of infrastructure, network services, and applications. Specifically, RedHat AAP onboards services prepared with Ansible playbooks whereas Nokia NCOM onboards services described in helm charts. Considering 5 UPF-1 NRF NSSF UDR UDMAUSF PCF UPF-2 AMF SMF Control Plane Data Plane N4 Firewall Slice 1 Slice 2 N1 N3 N3 N2 N6 N4 Solver VM Solver VM N3 VM N6 VM Figure 3: Multiple 5G Standalone Network Slices in i-CORA the need for a smooth transition from the VM-based to the container-based services, i-CORA supports a hybrid deployment approach that allows i) container-based services packaged with the helm charts to be onboarded via NCOM; ii) VM-based services or container-based services without the helm charts to be onboarded via AAP. Then, following the hierarchy in Fig. 4, the NCOM oversees all services and triggers their deployments either via the helm charts or AAP, by calling the corresponding APIs. As NCOM can manage multiple clusters, this hybrid approach gives flexibility i) for operators to select and integrate NSs from multiple vendors regardless of whether these services are containerized or helm-chart ready; ii) for vertical customers to create their own communications services by attaching their vertical applications to the NSs, while the vertical applications are deployed on-prem or in the edge for the sake of security. The i-CORA orchestration system has standardized or open APIs (e.g., ETSI SOL005, TM Forum Open APIs, or REST APIs) and thus makes it easy to integrate with other ecosystem (as shown in Fig. 4). For example, it can interact with Operating Support System (OSS) and other entities via TM Forum Open APIs to be consumed by verticals. One ongoing activity is the integration with an Open Source OSS Openslice, which acts as a customer portal towards verticals and allows them order and manage the services provided by the i-CORA platform. This portal will serve IMAGINE-B5G and FIDAL projects. C. Testing and Validation Testing service is highly demanded by different stakeholders to i) test integration (operators); ii) validate new technologies and services (solution and service providers); and iii) validate service KPIs (vertical customers). For operators, integration and function test is essential, especially in the multi-vendor environment. In i-CORA, we use the testing tools from Emblasoft to test the integration of the SA 5GC, composed of CNFs from the three vendors; and then integration of the RAN and CN for the E2E service. These tools provide emulated components to ease the integration test and produce packet captures for debugging and troubleshooting. For solution and service providers, a similar procedure is followed, e.g, by using the emulated components for the interoperability tests when a new CNF is introduced. For instance, we are working on the SEPP-based roaming service and using the emulated UE for the function test since the commercial UEs do not support this type of roaming yet. For vertical customers, the performance test checks if their required KPIs are met. A series of active testing tools can be deployed in the i-CORA infrastructure to validate the performance of various components and services, such as Emblasoft Evolver and Keysight LoadCore for the 5G core performance and conformance testing, Keysight ThreatSim for network vulnerability scanning, and Hawkeye for application performance testing. D. Use Cases i-CORA can serve a variety of verticals (see the last column of Table I) and pave the way towards solving the main technological challenges in the respective industries through near-commercial pilots, with advanced network features and new deployment architectures. PPDR can benefit from the 5G and B5G services provided by i-CORA in two ways: through public/commercial 5G networks of the MVP or nomadic 5G private networks of the NOW, which gives localized control and rapid temporary deployments, ideal for temporary emergency 6 E2E Service & Slice Orchestration Nokia Orchestration Center Network Service & Function LCM Nokia Cloud Operations Manager Container Infrastructure Orchestration Advanced Cluster Management Ansible Automation Platform Security Container Infrastructure Orchestration Advanced Cluster Management Ansible Automation Platform Security ContainersVMs AMF SMF UPF AMF SMF UPF NSSF PCF NRF NSSF PCF NRF UDR UDM AUSF UDR UDM AUSF PrismaPrisma NS X – NTNU App NS Y – YARA App NS Z – Testing NS – Emblasoft testing Network Services (NS) Network Slices eMBB Slice URLLC Slice Customized Slice Customer Portal Customer Portal REST APIK8s API ETSI SOL005 TMF Open APIs Figure 4: i-CORA Orchestration Architecture situations or events, e.g., natural catastrophe like flooding or avalanche. Figure 5 shows a pilot on Search and Rescue Operations demonstrated in FUDGE-5G, where a simulated natural catastrophe hit a small village situated in the Norwegian mountain ranges and destroyed multiple buildings with several people missing. In the rescue mission, given that the public telecommunication infrastructure (both fixed and mobile) is severely damaged, one NOW was instantly deployed in the emergency area as mobile command and control hub to provide connectivity and support critical communications between the first responder teams. First, the 5G SA powered drones were flown to the accident site and broadcasted the live video feed to the rescue personnel along with the police control room and a media master control room. Then the rescue team, equipped with 5G powered situational awareness rescue kits, was dispatched to search the spotted person in need of help. The 5G technology enables the team send live video feeds from the accident site to the emergency service respondents in real time such that critical decisions can be made in time. Aquaculture is one of the emerging verticals foreseeing the 5G potentials. In recent years, 5G trials (e.g., [8]) have been conducted to assess the capabilities of today’s commercial 5G networks in terms of supporting aquaculture scenarios. Uplink throughput is currently seen as the most critical KPI for applications which require feeding decision support with concurrent live streaming from multiple high definition cameras. This calls for further experimentation with RAN slicing technologies and uplink favored configurations that can potentially be explored in iCORA once near-commercial solutions become available. On top of that, replacing the high-maintenance underwater Figure 5: Network on Wheels used in Search and Rescue operation. fiber optic cabling with on-site wireless connectivity is the ambitious target, which may be realized by i-CORA’s MPNs or Mobile Edges for keeping the traffic locally. Media expects performance improvement (e.g., throughput and latency, among others) enabled by B5G technologies, due to the ever-increasing consumer demands for Ultra-High Definition (UHD) streaming, gaming, and AR/VR/XR technologies [9]. As part of 5GMediaHUB, IMAGINE-B5G and FIDAL, various media-related UCs will be validated and trialed on i-CORA. For instance, the Networked Music Performance (NMP) UC from 5GMediaHUB involves a Pan-European setup, as shown in Fig. 6, where musicians will be located in two Norwegian cities (Oslo and Trondheim) supported by i-CORA, and the audience will be present 7 Figure 6: NMP Pan-European setup for 5GMediaHUB. in Barcelona, Spain supported by CTTC. Media-specific network applications are developed and provided by different partners in the project to enable next-generation media production and delivery. V. CHALLENGES AND LESSONS LEARNED In the development and operation of the i-CORA platform, we experienced many challenges, from which we learned how to improve and enhance the platform. Integration is one of the biggest challenges in all largescale multi-vendor platforms. Standardized or open APIs are prerequisite to enable the smooth integration of multiple components. However, the standardized interfaces of management and orchestration components are not always supported. For example, ETSI SOL005 is usually required as the standardized interface of NFVO but many marketavailable orchestrators do not support ETSI SOL005. Version control is not trivial for integration. The asynchronization between versions may fail the integration, particularly when a subset of components upgrade while the others do not follow. The i-CORA platform is based on OpenShift, whose version is upgraded every 3 months (on average), following the Kubernetes lifecycle. However, the upgrade cycle of 5G-SA CNFs is usually much longer than 3 months, which results in the asynchronization between the OCP version and supported OCP version of 5G-SA CNFs. The challenges motivated us to propose the staged mechanism with separated Secure Service and Experimental Service cluster, the latter of which is the playground for the integration test. Automation is mandatory for i-CORA to support multiple EU projects and UCs concurrently. However, the realization of automation not only requires the solution availability (e.g., MANO components AAP and NCOM for automating the LCM of containers and NSs) but also relies on the support of involved components. For instance, to automate the LCM of NS with NCOM, all CNFs should be prepared with helm charts, which, however, are not available for some CNF vendors yet. Therefore, we develop the hybrid deployment approach and plan a pragmatic automation roadmap, starting with the deployment automation and moving forward to more challenging assurance automation after all partners will be ready. Supporting multiple UCs concurrently challenges us to tradeoff the resource sharing with guaranteed UC requirements, e.g., QoS/KPIs or new features. One solution is the staged mechanism that gives us more flexibility to experiment new slice design and/or configurations in the Experimental Service Cluster, without interrupting the test cases in the Secure Service Cluster. Besides, we are investigating and experimenting different isolation techniques to optimally isolate UCs physically, virtually, or hybrid. Isolation is useful to address a new challenge Exposure Management, raised by network exposure that attracts lots of attentions recently. Verticals desire a certain level of exposure of network data and management capabilities to enrich their applications. However, it remains an open question how different exposure levels can be controlled for multiple UCs, which will be studied in future. ACKNOWLEDGMENT This work is supported by the European Commission through the Horizon-2020 5GMediaHUB project (grant No. 101016714), the HORIZON-JU-SNS2022 IMAGINE-B5G project (grant No. 101096452) and FIDAL project (grant No. 101096146), and the HORIZON-CL6-2021-COMMUNITIES-01 COMMECT project (grant No. 101060881). REFERENCES [1] 5GPPP, “The European 5G Annual Journal 2021,” 2021. [2] M. Gupta et al., “The 5G EVE End-to-End 5G Facility for Extensive Trials,” in IEEE Internaltional Conference on Communications Workshops, 2019. [3] M. Xie, A. Gonzalez, P. Grønsund, H. Lønsethagen, P. Waldemar, C. Tranoris, S. Denazis, and A. Elmokashfi, “Practically Deploying Multiple Vertical Services into 5G Networks with Network Slicing,” IEEE Network Magazine, Jan/Feb 2022. [4] 5G-SOLUTIONS, “Deliverable D2.1: Setup and Operation of 5G Infrastructure,” 2022. [5] D. Gomez-Barquero et al., “FUDGE-5G: Fully Disintegrated Private Networks for 5G Verticals,” 2021. [6] A. Gonzalez, P. Grønsund, M. Xie, and P.-H. Lehne, “Achieving High Throughput and Low Latency with 5G: A Real Implementation Experience,” IEEE Communications Magazine, Jun 2021. [7] ETSI, “ETSI GS NFV-MAN 001: Network Functions Virtualisation; Management and Orchestration,” Dec 2014. [8] J. F. Pajo, et al., “Digitalization in the aquaculture industry: Validation trials over a commercial 5g network,” in 2023 Joint European Conference on Networks and Communications 6G Summit (EuCNC/6G Summit), 2023, pp. 520–525. [9] H. Khalili, et al., “Implementation of 5g experimentation environment for accelerated development of mobile media services and network applications,” in 26th Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), 2023, pp. 153–160.