Formal Models for the Energy-Aware Cloud-Edge Computing Continuum: Analysis and Challenges

Formal Models for the Energy-Aware Cloud-Edge Computing Continuum: Analysis and Challenges Yashwant Singh Patel Dept. Computing Science Ume˚ a University, Sweden [email protected] Paul Townend Dept. Computing Science Ume˚ a University, Sweden paul.to[email protected] Per-Olov ¨ Ostberg Biti Innovations & Dept. Computing Science Ume˚ a University, Sweden [email protected] Abstract—Cloud infrastructures are rapidly evolving from centralised systems to geographically distributed federations of edge devices, fog nodes, and clouds. These federations (often referred to as the Cloud-Edge Continuum) are the foundation upon which most modern digital systems depend, and consume enormous amounts of energy. This consumption is becoming a critical issue as society’s energy challenges grow, and is a great concern for power grids which must balance the needs of clouds against other users. The Continuum is highly dynamic, mobile, and complex; new methods to improve energy efficiency must be based on formal scientific models that identify and take into account a huge range of heterogeneous components, interactions, stochastic properties, and (potentially contradictory) service-level agreements and stakeholder objectives. Currently, few formal models of federated Cloud-Edge systems exist - and none adequately represent and integrate energy considerations (e.g. multiple providers, renewable energy sources, pricing, and the need to balance consumption over large-areas with other non-Cloud consumers, etc.). This paper conducts a systematic analysis of current approaches to modelling Cloud, Cloud-Edge, and federated Continuum systems with an emphasis on the integration of energy considerations. We identify key omissions in the literature, and propose an initial high-level architecture and approach to begin addressing these - with the ultimate goal to develop a set of integrated models that include data centres, edge devices, fog nodes, energy providers, software workloads, end users, and stakeholder requirements and objectives. We conclude by highlighting the key research challenges that must be addressed to enable meaningful energy-aware Cloud-Edge Continuum modelling and simulation. Index Terms—Continuum, modelling, green energy, brown energy, cloud computing, edge computing, fog computing. I. INTRODUCTION Driven by the bandwidth, processing and ultra-low latency demands of modern applications and devices, cloud infrastructures are rapidly evolving from centralized systems to geographically distributed federations of resources. Large volumes of data move back and forth between the network edge, intermediate fog nodes, and distant cloud data centers, whilst low latency devices connect to local edge resources. These federations, often referred to as the Cloud-Edge Continuum, are the critical fabric on which modern digital systems depend; indeed, in Europe they are seen as a key strategic technology for the region’s digital transformation [1]. The size of Continuum systems is growing at an enormous rate; some authors predict that over 50 billion IoT devices will be deployed in the Continuum by 2025, with orders of magnitude more endpoints brought about by 5G/6G systems [3]. This results in Continuum systems consuming huge amounts of energy; whilst a single hyperscale Cloud data center may consume over 100MW of power (equivalent to over 80,000 European homes) [4], data centers as a whole are predicted to consume as much as 8% of global electrical supply by 2030 [5] with a similar percentage consumed by network transmission. This places great strain on local and even national power grids [7], and has the potential for significant environmental impact - especially as nearly 80% of world’s energy is still generated by brown energy (non-renewable) sources such as fossil fuels, which leave a very high carbon footprint [6]. This is becoming a critical issue as society’s energy challenges grow, and is a great concern for power grids which must balance the needs of cloud-edge systems against other users [7]. It is therefore imperative to develop methods to mitigate, optimise and where possible reduce energy consumption in Cloud-Edge Continuum systems. A promising approach is intelligent task placement; through careful monitoring of large federated systems, software tasks can be allocated to the most energy-efficient resource available, taking into account service requirements, pricing, etc. As an example, tasks can be deliberately allocated to nodes that are currently using high amounts of green (renewable) energy (e.g. wind power, solar power etc.) Conversely, tasks can be migrated away from nodes in geographical areas experiencing high energy demand to increase local power grid availability for other users and businesses (thus balancing load on regional and national power grids.) These scheduling decisions may take into account energy pricing alongside user and software service-level objectives (SLOs)-e.g. tasks that require ultra-low latency may be allocated to a local data center even if it uses energy from a non-renewable source, while less latency critical task may be scheduled to more sustainable locations. However, the Continuum is highly dynamic, mobile, and complex: new algorithms, mechanisms and methods to improve energy efficiency must be based on formal scientific models that identify and support a huge range of heterogeneous components, interactions, stochastic properties, and (potentially contradictory) service-level agreements and stakeholder objectives. Use of formal models not only encourages researchers to take into account all necessary components in a highly complex system, but also facilitates validation through mathematical proofs and simulation. In the literature, conceptual models have been presented utilising techniques such as stochastic process algebra, discrete event simulation, queueing theory, approximation theory, game theory, graph theory, trace-driven simulation, and stochastic petri nets etc. However, few formal models of federated CloudEdge systems exist - and none adequately represent and integrate energy considerations (e.g. multiple providers, renewable energy sources, pricing, and the need to balance consumption over large areas with other non-Cloud consumers, etc.). This lack of models is a particular concern when developing autonomous management systems; manual approaches are no longer feasible [8], but existing management mechanisms do not consider energy constraints, policies, and optima across large federations. This paper conducts a systematic analysis of current models for energy-aware Cloud, Cloud-Edge, and federated Continuum systems, and most notably identifies the key gaps in the literature that must be addressed to allow a comprehensive energy-aware model for federated Continuum systems to be created. We then propose an initial high-level architecture and research approach to begin to address these gaps, with the ultimate goal to develop a set of integrated models that include data centers, edge devices, fog nodes, energy providers, software workloads, user and stakeholder objectives, service levels, and energy considerations. We conclude by highlighting key challenges and future research opportunities in the context of energy-aware Cloud-Edge Continuum modelling and simulation. II. BACKGROUND: MODELLING CLOUD SYSTEMS The architecture behind Cloud and Edge systems has evolved rapidly over time, moving from “traditional” nonfederated Cloud systems to Cloud-Edge architectures and then federated Continuum systems. It is useful to first discuss the conceptual nature of these approaches before investigating the modelling behind them. A. Traditional cloud systems In a non-federated “traditional” cloud system, a single cloud service provider typically manages one or more geographically dispersed data center sites. A typical geo-distributed cloud data center environment [9] integrating a single cloud service provider, multiple end-users, and several energy sources is shown in Figure 1. Here the cloud service provider manages mgeographically distributed data center sites (DCs), i.e., DC1,DC2,DC3,...,DCm. To offer services and resources to cloud consumers, each DC is linked to a backbone network and makes use of a variety of energy sources such as the commercial grid and green energy sources, networking & power equipment, and other devices. DCs can also control how much energy they use; reducing this lowers their energy costs and carbon footprint. For instance, a data center may use either Fig. 1. Non-federated cloud systems traditional resources, such as the electricity grid, or combination of green energy sources, such as solar panels and wind turbines. Additionally, data centers may also have installed diesel generators to address power outages and anomalies. Another important component within this environment is the cloud user, who submits service requests in the form of several parameters such as instance type, storage, holding time, starttime, end-time, etc. B. (Non-federated) Cloud-Edge systems The traditional cloud system architecture has numerous drawbacks, including latency issues arising from a data center’s distance from end users, and the need for a single data center to handle potentially massive numbers of users and network connections. Certain applications with strict communication latency restrictions, such as Ultra-Reliable Low Latency Communications (URLLC) and Enhanced Mobile Broadband (eMBB) services, which have a unit millisecond delay requirement, are not suited for the traditional cloud approach. To deliver comparable services with lower latency, edge, and fog computing models play a crucial role [10], [11]. A basic form of a non-federated cloud-edge system is shown in Figure 2, where processing of client tasks is performed at the data source rather than on a centralized server or in the cloud layer. In the edge layer, computing resources such as processors, storage, and networking capabilities are located at the edge of the network to move the burden of processing and storing service and device requests closer to the proximity of the original data source. C. (Federated) Continuum systems Non-federated Cloud-Edge architectures are an effective method to manage device latency; however, this approach Fig. 2. Non-federated cloud-edge systems is still relatively inefficient if resources are “siloed” - for example, if a fog node is saturated with tasks, there is no obvious mechanism for offloading to other local fog nodes etc. Continuum systems aim to address this issue, creating a federated and loosely-coupled architecture whereby tasks can be scheduled, monitored, and offloaded as necessary, potentially across different providers [12]. An example of this is shown in Figure 3. Continuum systems differ from earlier federated approaches (such as [13]), due to a much heavier focus on spatial location and heterogeneous physical resources. In this hierarchy of federated cloud edge infrastructure, edge computing delivers Fig. 3. Federated cloud-edge systems processing and storage at the very edge of the network i.e., near to where data is produced, whilst Fog nodes function as a bridge between the edge and the cloud [14]. The edgefog-cloud hierarchy allows the processing of data and move both vertically from the edge to the cloud and back again, and horizontally, across the edge or fog to dynamically adapt to the needs of the end-devices and end-users’ applications while considering the resource constraints (e.g., capacity, processing cost, latency, location, and security) [12]. III. CURRENT RESEARCH STATUS OF CLOUD-EDGE CONTINUUM MODELLING In recent years, several new approaches have been introduced to model resource distribution across the Cloud-Edge Continuum. This section discusses an overview of the most relevant works available in the literature, and investigates models (non-federated cloud-edge models, federated cloudedge models, and energy-aware cloud-edge models) from both technological and architectural perspectives. A. Non-federated cloud-edge models For non-federated cloud-edge systems, several models are presented for offloading applications and managing resources between the constrained edge and distant cloud data center. Rahmanian et al. [18] attempt to develop a tool named as ‘MicroSplit’ for efficient splitting of microservices. Initially, this tool analyses the possible dependencies between the microservices, and applies the Louvain method to split the microservices between the two layers of edge-cloud. The authors test its performance in multiple cloud-edge settings and improve latency with a reduction in mean response time. To address real-time performance and security issues of tasks, Singh et al. [33] design a scheduling algorithm ‘RT-SANE’. Through extensive experiments, they show that the algorithm attains a higher “success ratio” in comparison with existing approaches. To manage the dynamic allocation of resources and services in the Cloud-to-Edge Continuum, Tusa et al. [15] provide a unified resource management approach comprising both cloud data center and network resources. Their goal is to reduce the silo-effect, and provide end-to-end slices for the orchestration of all types of resources (i.e. compute, network) and services. To maintain the trade-off between (quality of service) QoS level and required computational resources of microservices, Fu et al. [34] design a run-time system called ‘Nautilus’. The system is composed of a communicationaware microservice mapper, a load-aware scheduler, and a resource manager. Through experimental results, it is shown that in comparison with traditional cloud systems, Nautilus minimizes computational resource and network bandwidth usage significantly while ensuring the necessary 99 percentile latency. To deploy latency-critical services in a private edgecloud environment, Ascigil et al. [27] develop uncoordinated resource allocation schemes. Specifically, the authors propose a centralized algorithm to model the QoS requirements of latency-critical services considering user response deadlines. Pop et al. [35] present a fog computing platform-enabled reference framework for Industrial IoT applications, offering both service and resource management. This is based on deterministic networking and virtualization to promise interoperability along with security. Etemadi et al. [36] design a centralized approach to resource orchestration in a simulated environment which enables deep learning to perform resource auto-scaling at run-time. Ullah et al. [37] design a mechanism named ‘MiCADO’ for the orchestration of applications in cloud-edge environments. They implement a real solution with case studies in the areas of video processing and healthcare. B. Federated cloud-edge models In the direction of federated cloud-edge models, Kar et al. [47] present a survey of offloading techniques in federated (Continuum) systems. Their study also provides an analysis of recent research into applying traditional optimization and machine learning approaches to federated cloud-edge systems. Soumplis et al. [12] identify critical resource allocation challenges in the integration of edge, fog, and cloud systems, presenting a heuristic and ILP-based technique for workload placement in the Continuum. Through simulation, they postulate that the resulting mechanisms effectively meet administrator-set objectives, utilising the processing power of the resources at various resource layers (edge, fog, and cloud), and reducing latency at the expense of higher cost. Silva et al. [40] review the applicability of incorporating context awareness to enhance IoT data sharing across Edge and Cloud. The article provides a general overview of the needs of various IoT contexts and updates solutions that take context-awareness indicators into account to deliver operational gains, such as reducing latency and energy usage. To establish directions for future study, the authors demonstrate that although context awareness is important in IoT contexts, its integration to enable more dynamic IoT environments is still limited. With an emphasis on container-based orchestration and fogenabled architectures, Svorobej et al. [42] evaluate different orchestration methods throughout the cloud-to-thing Continuum. Kampars et al. [43] investigate application layer protocols that can be applied for communication between the cloud, edge, and IoT levels. To create and manage the mobile-edgecloud computing Continuum, Baresi et al. [20] suggest the A3-E prototype architecture, which supplements functionality offered by FaaS platforms. Their results indicate that A3E is capable of deploying microservices and significantly reducing latency and battery consumption. Son et al. [21] suggest dynamic resource provisioning strategies for latencyaware Virtual Network Function placement in distributed edgeclouds. Their work assigns latency-sensitive services between cloud and edge to ensure desired QoS levels. A number of federated frameworks have also been developed in the industry, such as Zadara, BEACON, and Kubefed etc. Zadara’s federated program [44] enables service providers to manage edge computing and administer distributed clouds and supply computing resources close to users with minimal propagation latency. BEACON [45] manages the automatic deployment of applications and services across federated cloud infrastructures. Through a centralized API, Kubefed [46] enables the management of multiple Kubernetes clusters. The objective is to make multi-geo application deployment easier. An extremely popular open-source framework for managing, deploying, and scaling containers (Kubernetes) can also be used to build clouds, edges, and fog. C. Energy-aware models for cloud-edge Continuum To produce an energy-efficient data forwarding scheme for cloud-edge Continuum, Saraswat et al. [38] design a deadline-driven ubiquitous system. At each layer, they estimate fractions of the task to be computed for minimizing energy consumption. Overall performance is analysed using variable factors such as data size, deadline, delay, accuracy, network topologies, and energy consumption etc. To enable sustainable edge computing with distributed renewable energy resources, Li et al. [23] design a prototype model which supports coordination between edge and energy supply systems. It integrates a microgrid (e.g. a solar-wind hybrid energy system) and edge devices to ensure full utilization of renewable energy while maintaining QoS levels for time-sensitive IoT applications. Jeong et al. [29] develop an energy-efficient scheduling technique for federated edge clouds. The scheduling approach allocates services with actual traffic requirements to satisfy QoS levels, with the aim that it can maximize co-location of services placed on one server whilst reducing the total energy consumption of services. To address the problem of multi-task offloading, Sharma et al. [16] suggest a hybrid approach integrating first-order meta-learning and deep Q-learning strategies. The authors use simulation to measure improvements in applications’ energy consumption and training time under different settings of cloud-edge environments. For green mobile edge cloud environments, Chen et al. [24] develop a multi-user, multi-task computation offloading problem and apply the Lyaponuv optimization technique to decide on an energy harvesting policy. Hasan et al. [25] introduce the Aura architecture design, a highly mobile and localized ad-hoc cloud model to utilise IoT devices for work offloading techniques and upgrading apps. Through performance studies of Aura-powered IoT devices, they show the model’s efficacy in terms of job completion times, memory usage, predicted CPU clock cycle requirements, energy consumption, and cost. Gou et al. [26] suggest an architecture for collaborative computation offloading over FiWi (Fibre Wireless) networks. To reduce the total energy consumption of all the mobile devices while meeting the computation execution time limit, they address the issue of cloud-edge collaborative computation offloading. For scaling and offloading optimization, Yahya et al. [30] present a two-tier architecture comprising of an access network and a core network. To optimize capacity, they introduce a two-phase optimization approach by adjusting capacity and offloading ratios repeatedly. To address privacy disclosure, Xu TABLE I ANALYSIS OF CURRENT APPROACHES TO CLOUD-EDGE COMPUTING CONTINUUM MODELLING Source Type of Model Continuum Coverage      Optimization Objectives      Technique Energy Model Application Evaluation Device Edge Fog Cloud Energy Cost Latency Other 2023 [15] Graph-based × × ✓✓× × ✓ ✓ End-toend slice ×IoT, AI, digital twins Test-bed 2023 [16] Stochastic ✓ ✓ ×✓×✓×✓DMQTO ×IoT Simulation 2022 [17] System model ×××✓ ✓ ✓ ✓ ✓ Surrogate modeling E, T, CAI, industrial Simulation & Testbed 2022 [18] Graph-based ×✓×✓× × ✓ ✓ Louvain method ×Social network Test-bed 2022 [12] Integer linear programming ×✓✓✓×✓ ✓ ✓ Heuristic ×Real-time applications Simulation 2021 [19] System model ×××✓ ✓ × × × VM consolidation, scaling, brownout B, R, CWeb-based Test-bed 2021 [29] Discrete event simulation ×✓×✓ ✓ × × ✓ESFEC × Face recognition, Online text translation Simulation 2021 [30] Queueing theory ✓ ✓ ××✓×✓×LA-TPIO × eMBB, URLLC, mMTC Simulation 2021 [31] Stochastic ✓ ✓ ××✓×✓×DRL-E2D ×Smart home, Video, AI Simulation 2021 [32] Queueing theory ✓ ✓ ××✓×✓×DCOS × Healthcare, AR, Face recognition Simulation 2021 [33] Trace-driven simulation ✓×✓✓× × ✓ ✓ RT-SANE ×Real-time applications Simulation 2021 [34] Graph-based & Stochastic process ×✓×✓× × ✓ ✓ Nautilus ×Real-system applications Test-bed 2021 [35] Architecture Analysis Design Language ✓×✓✓× × ✓ ✓ Architecture ×Industrial IoT Test-bed 2021 [36] Trace-driven simulation ✓×✓✓×✓ ✓ ✓ Deep learning ×IoT applications Simulation 2021 [37] System model ×✓✓✓× × × ✓MiCADOEdge × Video processing, healthcare Test-bed 2020 [38] Queueing theory ✓✓✓✓✓ ×✓ ✓ Newton method ×Ubiquitous computing Test-bed 2020 [39] Queueing theory ×✓✓×✓×✓×Ant colony ×Smart city Simulation 2019 [20] System model ✓ ✓ ×✓× × ✓ ✓ A3-E ×AR application Test-bed 2019 [21] Trace-driven simulation ×✓×✓× × ✓ ✓ Dynamic algorithm × Timesensitive application Simulation 2018 [23] System model ✓ ✓ ×✓ ✓ × × × Framework R IoT applications Test-bed 2018 [24] Queueing theory ✓ ✓ ××✓× × ✓ Lyaponuv Optimization RMulti-user multi-task Simulation 2018 [25] System model ✓×✓×✓× × ✓Aura system ×Prototype Test-bed 2018 [26] Approximation theory & Game theory ✓ ✓ ×✓ ✓ ×✓ ✓ Game theory ×FiWi networks Simulation 2017 [27] Mixed integer linear program ×✓×✓× × ✓ ✓ Centralized algorithm ×Real-time application Simulation 2017 [9] System model ×××✓ ✓ × × ✓Binpacking R, B, O HPC applications Simulation 2016 [28] Mixed integer linear program ×✓✓×✓×✓×EECO × 5G heterogeneous networks Simulation et al. [39] present an intelligent offloading technique for smart cities, preserving privacy, enhancing offloading efficiency, and promoting edge utility. To achieve trade-offs between service response time, energy, and maintaining load balance while ensuring privacy during service offloading, the authors adopt an ant colony optimization approach. For mobile edge computing in 5G heterogeneous networks, an energy-efficient computation offloading technique is suggested in [28]. The authors address an offloading system’s energy minimization problem, taking into account the expenses associated with both task computing and file transport. Li et al. [31] present a task offloading policy that considers task deadline times. To determine the optimum offloading strategy and address the scalability issue of the deep Q-network action space, they develop an edge-to-device deep reinforcement learning approach. To improve the deep Q-network algorithm, Zhang et al. [32] present a heuristic offloading technique that minimizes both latency and energy consumption. The prime idea behind the use of a heuristic algorithm is to reduce the convergence time in hybrid edge computing networks. D. Key omissions in Continuum modelling Table I provides a comparative analysis of recent approaches to Cloud-Edge Computing Continuum modelling. The analysis lists multiple aspects of each work, including model type, Continuum coverage, optimization objectives, type of applied technique, the energy model used, the prospective application area, and the evaluation method employed. When describing energy models, we consider brown energy (B), renewable energy (R), cooling energy (C), compute devices and cooling components (E), off-site utility grid (O), and thermal energy (T). After careful investigation of the current research on Continuum modelling, we observe that in the literature, several formal models for traditional cloud systems have been proposed, e.g. [51] but these do not capture the dynamic nature of cloud-edge systems or integrate stochastic properties, energy providers, pricing, and renewable energy sources. Most work assumes a single data center, precluding intrinsic challenges faced with the management of federated systems, such as how to monitor and schedule multiple complex resources across multiple networks in a scalable and decentralised manner with SLO awareness [52], and how to balance accuracy with decision making latency (many recent approaches, such as [53], [54], use machine-learning methods that are too slow to provide the ultra-low latency scheduling required by edge applications). A basic model that integrates nodes with energy providers is presented in [19] but does not consider federated edge systems or cross-site monitoring issues, while [23] only considers micro-grid integration with edge nodes, with no centralised cloud integration. Of work that does consider energy-aware federated systems, little has been achieved; [55] propose an integration between smart grids and cloud-edge systems but the proposed architectural model is extremely high-level and does not consider monitoring overhead, task properties, or decentralised control of the system. The overall analysis of the state-of-the-art on cloud-edge Continuum highlights the lack of unified systems, formal models, and methods to seamlessly integrate various energy factors including temporal pricing, renewable energy sources, energy provider requirements, resource restrictions, and balance consumption over large-areas with other non-Cloud consumers. Research in this field typically results in either reference architectures or simulated system environments, with computing, networking, and storage resource management serving as the primary focus. These observations demonstrate the absence of a unified resource orchestration technique capable of integrating the pricing models, types of workloads, multi-objective optimization, monitoring, and controlling strategies, QoS and SLO requirements of end-users, heterogeneous systems and networking technologies, energy policies, energy providers, energy sources, and administration of compute and network resources in the energy-aware federated cloud-edge Continuum. There is a clear need to bridge this gap and exploit the modelling of Continuum key components, their relevant stochastic properties and interactions, and their integration with key energy factors. IV. RESEARCH CHALLENGES Based on the omissions described in the previous section, we identify seven key research questions that must be addressed to adequately integrate energy considerations into a formal model for the Cloud-Edge Continuum. A. How to model the system? In the literature, there is a lack of formal models for federated cloud-edge systems in general; no existing model incorporates energy providers, pricing, and sustainability. The creation of formal energy-aware models for federated cloudedge systems is a challenging task due to a lack of empirical data to calculate stochastic properties, a lack of analysis to model geographical energy distribution factors such as supply and demand of green and brown energy sources, a limited understanding of temporal energy pricing, and limited modelling of energy provider policies & restrictions. To address this, empirical data must be assessed across a range of disciplines, and appropriate model types identified for each sub-system. B. How to combine multiple models? Once models for each sub-system in an energy-aware Cloud-Edge infrastructure have been created, there are still significant challenges with regard to integrating these models. These challenges include how to best integrate different model types (e.g. a graph-based model integrating with a model based on queueing theory), how to determine appropriate granularities when simulating the models, how to mathematically reason across the combined model, etc. These challenges are not unique to Cloud-Edge systems, but various solutions in the literature need to be properly assessed to determine which is appropriate for the scale and number of interactions required. C. How to model/consider different regions or sub-sets of the system ? Optimization at local level e.g., for a specific sub-system (single data center, application, device, etc.) is relatively straightforward to achieve. However, optimising or balancing resources across geographically federated regions and providers is an extremely challenging task due to the heterogeneity of the respective control systems, different API models, multiple ownerships, conflicting priority levels, user fairness constraints, monitoring and scheduling complexities of multiple resources across multiple networks with SLO awareness. D. How to develop a self-stabilizing model? In the Continuum, failure of a node (from server to data center level) will impact performance and result in task interruption. An application’s sub-tasks may run on various edge nodes; all sub-tasks executing on a specific resource will be interrupted if it fails, and any sub-tasks that depend on those interrupted sub-tasks will likewise be interrupted (a partial manifestation of the “long tail” problem seen in e.g. [22]). There is therefore a challenge to create a failureresilient scheduling model that can recognize dependencies between tasks and reschedule sub-tasks impacted by failure events to limit interruptions. A further challenge is to develop a self-stabilizing architecture which can recover from transient faults automatically without any manual intervention, as it is predicted that the failure probability of edge servers will be far higher than that of cloud servers [50]. E. How to maintain energy-performance trade-offs? There are several studies that have investigated to enhance the performance of individual cloud or edge systems. Most of the existing studies are focused on resource management in a non-federated cloud-edge system but do not consider federations of resources (e.g. cloud-edge). Additionally, there are no best practices or guidelines to optimize or monitor the overall performance of the federated-Cloud Edge Continuum. In a federated cloud-edge system, nodes and regions have different SLOs and pricing-as do energy providers. It is a critical task to optimize between individual and regional SLOs while ensuring performance. Therefore, we need to balance local and global optima at different levels within the stack (e.g. edge, fog, cloud, regional etc.) How to arbitrate and optimize conflicting service levels and energy requirements in a holistic manner across these levels is not yet fully understood. F. How to model green energy-driven cloud-edge systems? Many new challenges arise when considering the impact of Cloud-Edge resources on power grids, especially when other users and demands on those power grids are taken into account. Different power grids may have different capacities and sources of renewable energy at any moment in time; for example, a power grid in region Amay at a specific point in time incorporate 20% of its available power from renewable sources and have 30% free capacity. Later in the same day or week, those numbers may change to 10% and 15% respectively. It may therefore be extremely valuable to schedule tasks in a Continuum between different grids to improve utilisation of renewable sources and available capacities (and hence lower costs) whilst maintaining service levels for users and applications. Modelling these factors and ultimately integrating these models into energy-aware resource management systems is a significant and vitally important challenge that needs to be addressed. As observed in [23], approximately 80% of today’s energy is still produced from brown energy sources; mechanisms to increase the use of green energy sources in the Continuum will go a great way towards reducing its carbon footprint (and hence impact on the environment). G. How to develop validation models for energy-aware cloudedge systems? The model-based simulation of any cloud-edge system can utilise some existing simulators (such as EdgeCloudSim [59], ENIGMA simulator [60]). However, to iteratively test different aspects of an entire federated cloud-edge system such as decentralized monitoring, arbitration, and optimization is a challenging task due to limited scalability scenarios, Continuum mobility behaviours, topology configurations, network behavior at different levels of granularity, and energy considerations. In addition, designing a software-defined networkingbased testbed to monitor and track the energy consumption of an entire federated cloud-edge infrastructure adds another level of complexity. V. HIGH-LEVEL MODEL TO RESOLVE KEY OMISSIONS Although there are some preliminary studies on federated cloud-edge systems but they are still in their early stages. Thus, it opens several opportunities for future research in energyaware cloud-edge Continuum architectures. To resolve some key research challenges; (i) Identification of key components, their characteristics, and interactions; (ii) Integration of energy considerations such as energy providers, energy sources, energy pricing, and energy policies and restrictions etc., we propose a high-level design of a perspective model as shown in Figure 4. A. Unified architecture To reason over federated cloud-edge systems, key components, and their relevant features and interactions need to be identified and modeled; no cloud-edge model has yet been created that integrates multiple components such as energy providers, renewable energy sources, energy pricing, energy provider policies, and restrictions. The major challenges are to identify key hardware, network, and energy components within a cloud-edge system and categorize these into a layered stack. Interactions between components and layers are required to be analysed and formally modeled. To model such a system, the formal model can incorporate three aspects: (i) Creation of a formal layered model: For the Fig. 4. A high-level model for energy-aware Continuum systems development of such models, we need to identify and categorize the different energy, network, hardware, and software components prevalent in cloud-edge systems into a series of interacting layers. Interactions between and across these layers are needed to be explored and defined. (ii) Identify and build models of typical cloud-edge workloads: This task is concerned with identifying common types of workload submitted to cloud-edge infrastructures and quantifying their resource consumption, duration, network, and energy characteristics. (iii) A predictive energy consumption model for data centers and workloads: Utilizing the outcomes of (i) and (ii) as the basis for developing a method to quickly estimate predicted energy consumption within cloud-edge nodes. This could be used as part of the decision mechanism when balancing and optimizing software placement. B. A high-level perspective model To address the core research challenges and establish a comprehensive framework, we aim to develop integrated models that encompass various components such as data centers, edge devices, fog nodes, energy providers, software workloads, and the requirements and objectives of users and stakeholders. We propose a perspective model for energyaware cloud-edge computing Continuum as shown in Figure 4that identifies the cloud-edge Continuum infrastructure, energy provider policies & metrics, controller, end-users, workload manager, and networking components for an energy-aware design, and interconnection between them. The operational aspects of these components are elaborated as follows: (i) End users: End users submit their service requests to the cloud-edge system through the end-users layer. Within this layer, users have the ability to specify certain QoS restrictions for their requests. These may include parameters such as maximum tolerable delay, available bandwidth for data transfer, deadline, budget, as well as specific security and privacy requirements. By providing these QoS restrictions, users can communicate their desired service levels and constraints to the cloud-edge system, allowing it to prioritize and allocate resources accordingly. (ii) Workload manager: There are several common types of workloads that are typically submitted to the cloudedge infrastructure. These include IoT data processing, collaborative applications, web and application hosting, video streaming and content delivery, data storage and retrieval, big data analytics, offloading workloads, and real-time applications etc. To process these different types of incoming workloads, the workload manager utilizes various hosting frameworks. Commonly used hosting engines include containerization platforms like Docker and Kubernetes, serverless computing platforms such as AWS Lambda and Google Cloud Functions, and content delivery networks (CDNs). The selection of a hosting engine depends on factors such as workload requirements, resource constraints, latency considerations, and scalability needs. (iii) Cloud-Edge Continuum infrastructure: In a similar vein to the E2Clab platform [56], the infrastructure environment in the Cloud-Edge Continuum consists of several managers: (i) Layer Manager: This manager is responsible for maintaining the geographical distribution of Continuum resources. It analyzes heterogeneous scenario deployments, including single-layer cloud systems or multi-layer systems that incorporate cloud, fog, and edge resources; (ii) Service Manager: The service manager encompasses various components such as data producers, gateways, ingestion systems, and processing frameworks. It facilitates the management of services within the Continuum; (iii) Network Manager: The network manager is responsible for establishing communication rules between different layers and services. It ensures efficient data transfer and connectivity across the Continuum; and (iv) Workflow Manager: The workflow manager handles the specification of dependencies, execution logic, performance metrics, and life-cycle management of running services. It enables the coordination and orchestration of complex workflows within the Continuum. (iv) Energy provider policies and metrics: The energy provider policies and metrics module is introduced to integrate energy considerations with the cloud-edge computing Continuum design. It encompasses various components that facilitate details about energy sources i.e., brown energy and green energy, primary and secondary energy storage devices utilized for storing surplus energy, information about rented infrastructures, grid control policies, power regulations (such as energy gentrification [7] perspectives to prioritize user requests for grid owners), and energy price metrics for both off-site and on-site utility grid providers. The functioning of these components is discussed as follows: (1) Energy sources: The primary objective of the available energy sources is to maximize renewable energy usage for reliable and efficient Cloud-Edge Continuum systems. However, simply minimizing operational energy usage or wastage is insufficient from a sustainability perspective. It is equally crucial to minimize the power supply to the infrastructure. In order to achieve this objective, the implementation of a demand response program [61] becomes essential. This program ensures that the power demand of the Continuum systems can be effectively met by coordinating with the available power supply from local or external energy sources. It employs direct or indirect load control strategies to optimize power usage and maintain a balance between the demand and supply of electricity [23]. (2) Grid control policies & regulations: Grid control policies and regulations may exhibit variations across countries, regions, and utility companies, as they aim to balance several objectives. These include ensuring grid reliability, promoting the adoption of renewable energy, optimizing energy markets, and safeguarding consumer interests in the energy sector [7]. The energy grid control policies and regulations component stores information about the rules and guidelines established by governing authorities to govern the operation, management, and control of the energy grid. It encompasses various aspects of grid operations, including policies dictating the types of power generation sources allowed, such as renewable energy sources (solar, wind, hydro) or traditional fossil fuelbased power plants; requirements for grid interconnection and power quality standards; load management measures aim to maintain grid stability and prevent overload conditions; grid resilience policies focus on enhancing the resilience of the grid to withstand disruptions, energy market, and pricing regulations; policies addressing the integration of distributed energy resources (DERs), such as rooftop solar panels or small wind turbines, into the grid; environmental policies to promote cleaner energy production and reduce greenhouse gas emissions.; policies focusing on energy consumers’ rights etc. These grid control policies and regulations provide a framework for governing energy grid operations and ensuring the reliable and sustainable functioning of the energy-driven Cloud-Edge Computing Continuum. (3) Energy storage devices: The objective of this module is to store renewable energy when the Continuum’s energy demand is low (i.e. off-peak hours of a day). By doing so, the stored energy can be utilized during peak hours, reducing the need to consume brown energy. This approach helps to optimize energy costs as well. (4) Energy price metrics: The energy price metrics component plays a crucial role in optimizing energy consumption by considering price variations. The energy price varies depending on the type of energy source used, such as off-site utility grid, on-site green energy, and on-site brown energy. Each type of energy may have different price characteristics. For instance, off-site brown energy may have varying carbon intensities and carbon taxes across different locations. Moreover, energy prices can fluctuate during different periods of the day, including on-peak and off-peak hours. By considering these factors, the controller can make informed decisions to effectively manage energy consumption and costs. (v) Controller: A controller, whether centralized or distributed, based on the MAPE-K (Monitoring, Analysis, Planning, Execution, and Knowledge) model [19], [57], [58], is essential to support resource provision, monitoring, and allocation in the Cloud-Edge Continuum. To develop interactions with the system, sensors, which are hardware-attached devices responsible for collecting data from various levels, and effectors, which are actuator devices used to enable or disable services through API calls, are applied. The monitoring module receives information regarding energy usage and resource utilization through these sensors. The analysis module characterizes workloads based on multiple factors, including time sensitivity, resource intensity (such as compute, memory, data, network), location, and performance requirements. It utilizes cost models to calculate energy cost, carbon cost, energy wastage, and considers the impact on climate. The planning module utilizes allocation policies to make scheduling decisions and analyzes the potential consequences of implementing changes in the system. The execution module utilizes effectors to perform resource scheduling on the Continuum infrastructure. It employs proactive scheduling policies for multi-objective optimization. The optimization objectives