An Efficient Storage Solution for Cloud/Edge Computing Infrastructures

An Efficient Storage Solution for Cloud/Edge Computing Infrastructures (Invited Paper) Antonios Makris1, Ioannis Korontanis1,2, Evangelos Psomakelis1,2, Konstantinos Tserpes1,2 1School of Electrical and Computer Engineering, National Technical University of Athens, Greece 2Department of Informatics and Telematics, Harokopio University of Athens, Greece Abstract—Edge computing emerges as a promising paradigm for managing and processing the immense data volumes generated by Internet of Things (IoT) devices. By moving data and computation closer to the client/devices, edge computing facilitates latencyand bandwidth-sensitive applications that would not be feasible through cloud and remote processing alone. Despite recent advancements, concerns persist regarding the requirements of cloud/edge-based applications. A distributed edge storage solution becomes indispensable to ensure data proximity, mitigate network congestion, and accommodate changing demands. Nevertheless, implementing an efficient edgeenabled storage system poses numerous challenges due to the distributed and heterogeneous nature of the edge, as well as its constrained resource capabilities. To this end, this paper presents an efficient cloud/edge storage framework with consideration on performance (QoS), latency reduction, and energy efficiency. The evaluations demonstrate significant improvements by reducing data operation execution time, enhancing performance, alleviating network infrastructure strain, and optimizing energy efficiency. Index Terms—edge computing, edge storage, container-based virtualization, cloud computing, internet of things, local registry, peer-to-peer architectures I. INTRODUCTION The rapid proliferation of the Internet of Things (IoT) has introduced a profound and transformative shift in the field of service utilization. It has also introduced a novel computing paradigm that necessitates the processing of data at the edge of the network. Traditional cloud infrastructures have struggled to cope with the vast volumes of data generated by IoT applications, as end devices are usually distant from the cloud servers. This geographical separation introduces additional processing and network overhead, leading to elevated latency, limited bandwidth, and overall performance degradation. This confluence marks the birth and realization of Edge computing, bringing together the power of cloud services with decentralized processing at the network edge. Edge computing has garnered significant attention in recent years [1]–[5] and is considered a key enabler for meeting the increasingly strict requirements of next-generation applications [6]. Unlike traditional centralized cloud computing, edge computing places computational resources closer to end-users in the so-called edge [7]. Amongst others, this has the benefit of reducing the latency times. It provides more responsive services and offers higher scalability and availability. Edge computing significantly reduces the amount of data in transit to remote clouds, enabling data processing near the data sources. This expands the possibilities for high-bandwidth and delaysensitive applications that are not feasible with cloud and remote processing alone [8]. Typically, edge nodes are characterized by limited computation, storage, network, and power resources. They commonly feature varied hardware and software setups and are distributed across multiple locations. Efficient data sharing between edge nodes poses one of the main challenges in developing edge applications, and it can be achieved either within application frameworks or by leveraging an external storage service. Despite significant advancements in providing efficient edge storage solutions, there are still some issues to be addressed related to the functional and non-functional requirements of cloud/edge-based applications [9]. Edge storage can significantly improve data access, enabling latencysensitive applications [10]. However, the distributed, dynamic, and heterogeneous nature of the edge computing environment, combined with diverse application requirements, introduces several challenges, including hardware and software incompatibilities, limited resources of edge devices, security and privacy concerns, scalability issues, latency and bandwidth constraints, interoperability, energy efficiency optimization, data management, fault tolerance, reliability, and resource allocation. This paper presents an efficient cloud/edge storage framework that focuses on performance (QoS), latency minimization, and energy efficiency, emphasizing on the resolution of the problem of data distribution and offloading based on application requirements. Additionally, a localized Docker registry is implemented and integrated into the framework, bringing application images closer to the edge, reducing network traffic and delays during deployment, migration, and scaling operations. Employing a local registry further contributes to the optimization of energy consumption. The evaluations demonstrate significant improvements by reducing data operation execution time, enhancing performance, alleviating network infrastructure strain, optimizing energy efficiency and establishing a foundation for automation aimed at optimizing the allocation and pre-allocation of data in edge clusters. The rest of this paper is organized as follows: Section II explores related work concerning storage solutions for edge computing infrastructures from the perspectives of data management, performance, latency minimization through the utilization of edge-driven registries, and energy efficiency. Section III provides an overview of the framework’s system architecture. Section IV presents the experimental evaluation of the proposed approach. Finally, Section V summarizes the merits of our work and highlights some perspectives that require further attention in the future. II. RELATED WORK A. Edge Storage Solutions: Navigating Data Management in Edge Computing Environments With the rapid expansion of edge computing, IoT devices generate vast quantities of data at the edge. In response to this, edge computing has emerged as a prominent paradigm to alleviate network strain and enable low-latency and highthroughput data processing by provisioning computing and storage resources at the network’s edge. Consequently, the storage and efficient management of the massive volumes of data generated at the edge become paramount [11]. Psaras et al. [12] conducted an initial investigation of the advantages and obstacles associated with edge data storage, in order to alleviate the strain caused by the substantial volume of data generated by IoT devices on the core network, which is tasked with transmitting this data to cloud storage. Confais et al. [13] advocate for specific properties in each edge storage system, including low access time, network containment between miniclouds, and availability through partitioning and mobility. Traditional file systems like distributed HDFS or Lustre appear unsuitable for edge environments due to their centralized nature. On the other hand, object storage software systems which leverage Peer-To-Peer (P2P) mechanisms, can cope better with the proposed edge storage requirements. Clarke et al. [14] propose an epidemic approach for decentralized storage systems, offering functionalities such as data publication, replication, and retrieval. Sonbol et al. [15] introduced EdgeKV, a decentralized storage system comprising a local layer with multiple groups composed of geographically proximate nodes, and a global layer wherein diverse groups are linked via gateway nodes. These layers are interconnected through a ring overlay, facilitating a rapid and dependable system that employs data replication and ensures strong consistency. In a similar vein, Nicolaescu et al. [16] proposed SEND, an in-network storage and management framework situated at the network edge. SEND aims to enhance network edge performance and Quality of Service (QoS). Efficient data placement research plays a pivotal role in the development of reliable edge storage solutions, especially when heterogeneous storage systems across edge and cloud nodes require data exchange. Additionally, concerning resource management, various challenges related to adapting to dynamic environments and optimizing collaboration among multiple edge servers on a large scale must be tackled. The efficiency of near real-time decision-making can be significantly enhanced by bringing analytics “closer” to the data. Consequently, edge architectures can mitigate the volume of data traversing the network, thereby reducing latency and overall costs. Lujic et al. [17] propose a three-layer architecture model for managing data storage on the edge. The edge storage component is responsible for storing and sending data to an adaptive algorithm which dynamically determines the balance between data quality and quantity. Additionally, the edge storage component receives data from the cloud layer and stores the results generated by the adaptive algorithm. Since edge nodes lack the storage capacity of specialized devices, it is imperative to decide which data segments should be stored to enhance real-time performance, while less frequently accessed data can be uploaded to the cloud. To tackle the challenges posed by limited storage space in edge computing and mitigate data loss resulting from unstable networks, Xing et al. [18] proposed a distributed multi-level storage system model. This model is based on a multiple-factors Least Frequently Used (mLFU) replacement algorithm. While direct algorithms such as LFU prove effective on the edge where computational capabilities are limited, they solely account for the frequency of data access. On the other hand, mLFU takes into consideration not only the access frequency but also the importance of data, defined as the correlation degree between the file and the program. B. Edge-Driven Registries: Streamlining Deployment and Minimizing Latency Given the prevalent adoption of containerization as the microservice paradigm in both cloud and edge computing domains [19], there is a burgeoning interest in examining the benefits of employing local registries for storing application images. The primary objective is to expedite container deployment by reducing both the time required for image downloads and latency. Ismail et al. [20] suggested that storing images in an edge local registry can pre-cache the images, resulting in reduced network usage and decreased service deployment time. While remote registries can lead to significant waiting and downloading times for platforms, local registries positioned on the edge can operate on hosts with limited resources while managing numerous image pulling and pushing requests. To address the prolonged times associated with downloading application images from remote registries, some platforms adopt a hybrid approach incorporating both remote and local registries. Gupta et al. [21] introduced a solution leveraging containerization techniques for deploying and managing deep learning models. This approach offers benefits such as low latency, data privacy, and minimal space requirements. During their study, an edge server retrieved images from a centralized registry and stored them in a local registry. This enabled the edge server to fulfill future requests for the same model from the local registry, thereby reducing download times. To tackle the challenge of registry scalability and alleviate pull delays at the edge, Gazzetti et al. [22] introduced a streamed deployment approach. This method involves employing a singular device, referred to as the Gateway, which interacts with the cloud and retrieves images on behalf of nearby devices. Subsequently, these images are disseminated among these devices in a peer-to-peer fashion, leading to reduced delays, minimized network usage, and enhanced scalability. Similarly, Kangjin et al. [23] proposed Faster Image Distribution (FID), a largescale image distribution system that leverages the combined bandwidth of both the Docker Registry and other nodes within the cluster to enable Docker image distribution. Platforms and tools can leverage local registries at the edge to mitigate image download speed, but strategic placement is imperative when dealing with many nodes. Knob et al. [24] introduced a novel deployment solution for distributing container registries within an edge topology using a community-based placement algorithm that optimizes registry distribution within a relational graph. The proposed solution utilizes a twophase algorithm to generate communities and designate a central node as the host for the new registry. This approach demonstrates improved performance and effectively minimizes the occurrence of non-started containers. Becker et al. [25] introduced EdgePier, a fully decentralized container registry designed for edge sites, utilizing peer-to-peer connections to reduce deployment times. EdgePier enables the exchange of image layers without dependence on centralized orchestration entities. Zheng et al. [26] introduced Wharf, a middleware designed to distribute Docker images across a distributed file system with the aim of reducing storage usage, network load, and job completion times within a cluster. They optimized global state access synchronization and utilized Wharf to partition Docker’s runtime state, resulting in image retrievals up to 12 times faster compared to using Docker on local storage. As previously noted, integrating local registries into the Edge infrastructure can effectively reduce download and waiting times. However, Edge devices possess limited storage capacity, necessitating optimization of registries to maximize storage efficiency [27], [28]. Zhao et al. introduced Slimmer [29], a Docker registry incorporating support for file deduplication. Diverging from traditional Docker registries, Slimmer adopts a unique approach to managing shared image layers. During image uploads, Slimmer efficiently manages concurrent client requests, enabling asynchronous processing. Upon receipt of a layer, Slimmer refrains from immediate unpacking, instead preserving it in a persistent staging area as a compressed tarball. C. Insights into Energy Efficiency in Edge Computing Infrastructures Edge infrastructures present a promising solution to the latency issues inherent in Cloud infrastructures. However, to ensure its viability, the research community has increasingly focused on understanding and optimizing the energy consumption of Edge systems. Jiang et al. [30] proves that Edge environments are trying to become energy aware. Their review presents energy aware edge hardware designs, edge computing architectures, edge operating systems, edge services and applications and computing offloading. Ahvar et al. [31] present a taxonomy of various Cloud-related architectures, categorized by their energy usage, including that of network devices and cooling systems. According to their experiments, a completely distributed architecture consumes 14 to 25 percent less energy compared to fully centralized and partly distributed architectures, respectively, due to the lack of intra-data center networks and large-scale cooling systems. Even the network infrastructure has the potential to consume a considerable amount of energy. Habibulah et al. [32] introduce a method that utilizes a clustering approach based on a spectral algorithm to reduce the energy consumption of network infrastructure. This approach involves deactivating parts of the network during periods of low traffic to conserve energy. Even distributed architectures can exhibit high energy consumption if they perform heavy operations. Mocnej et al. [33] examine how IoT in Edge infrastructures can impact the energy consumption of devices. Their findings suggest that Edge infrastructures can be effective for constrained devices with limited battery life when used properly. They calculate energy consumption during overhead by performing operations like duty cycling and employing a value prediction model. The experiment reveals that the additional computation required by the value prediction model causes minimal overhead, which can be easily overridden even with a low prediction success rate. To accurately gauge energy consumption, infrastructure owners should be equipped with measurement tools. A simulation tool called Sphere is presented in [34]. Sphere is capable of evaluating the performance of Edge infrastructures by setting up scenarios such as infrastructure topology, orchestration models, incoming workload patterns, resource management models, and scheduling policies. Cruz [35] provides information on various tools that can be installed on operating systems to monitor the energy consumption of hardware. One such tool, PowerJoular, presented in [36], is particularly valuable for this purpose. This command-line software facilitates realtime monitoring of power consumption from both software and hardware components. Aslanpour et al. [37] presents a more advanced tool called WattEdge. WattEdge can measure various factors contributing to energy consumption, including CPU usage, memory usage, storage activity, network connectivity, bandwidth usage, communication protocols, and energy sources such as batteries. III. SYSTEM ARCHITECTURE The proposed framework is based on Kubernetes1, MinIO2 and Prometheus3technologies. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. An open-source framework developed by IBM, known as MinIO, serves as the storage solution. MinIO is inherently decentralized and highly scalable, functioning as a peer-to-peer solution designed to be cloud-native and capable of running as lightweight containers managed by external orchestration services like Kubernetes. It supports a hierarchical structure to create federations of clusters. MinIO utilizes object storage over block storage, so 1https://kubernetes.io/ 2https://min.io/ 3https://prometheus.io/ TCP/ 5000 Lookup         DNS:ldr.reg Private key Docker Registry Secret Service Account Create Secret cert-manager ldr.registry docker push docker pull Dataset CRD kind: Dataset metadata: name: minio-data Dataset Lifecycle management via Operator Integration with multi-cloud/edge remote datasets Workload Annotation apiVersion: v1 kind: Pod metadata: name: minio-client-sample spec: volumes: - name: minio-storage persistentVolumeClaim: claimName: minio-data Workload pods Configure and pass COS credentials via K8s secrets Dataset Lifecycle Framework Dataset #1 Dataset #2 Cluster Deploy Application Submit the application to the K8s Cluster Dynamic Lifecycle Framework - Data sources DLF transparently links pods with target data sources by exploiting Dataset abstraction Cloud Edge Fig. 1: Conceptual overview of the proposed framework it is in fact a combination of the two systems, preserving the lightweight distributed nature of block storage while providing the plethora of metadata and easy usage of the object storage. Unlike other object storage solutions primarily intended for archival purposes, MinIO is designed to deliver the high-performance object storage suited for modern big data applications. The Kubernetes Dataset Lifecycle Framework (DLF) provided by IBM’s Datashim4is employed on top of MinIO, allowing the edge storage component to be used as a mountable virtual disk drive. A detailed description of the DLF is provided in Section III-A. Prometheus, a widely used opensource monitoring and alerting tool, is responsible for collecting real-time monitoring data about node performance and overall component behavior. This information enables analysis of various applications and facilitates optimization of cluster architecture, configurations, and data distribution strategies. Finally, a localized Docker registry (LDR) is established to bring application images closer to the edge, thereby reducing network traffic and image download times. LDR hosts the Docker images and utilizes Kubernetes containerization to deliver its services. Additionally, LDR generates a set of secrets to enable secure communication between the registry and its clients using the HTTPS protocol and a basic authentication scheme. A detailed overview of LDR is available in Section III-B. Figure 1 provides a conceptual view of the proposed framework, highlighting the components utilized in an illustrative scenario involving a K8s cluster with two nodes. The proposed framework is open-source and available for access at5. Within the context of the proposed solution, a collection 4https://datashim.io/ 5https://github.com/Efficient-Computing-Lab/EdgePersist of bash and YAML scripts has been developed to manage all configuration, installation, and deployment procedures required both before and after the deployment of MinIO workers. These configurations encompass firewall rules, DNS settings, package installations, and security checks tailored to the setup environment, architectural specifications, physical machine resources, and associated software. These tasks facilitate the semi-automated deployment of the proposed framework, establishing complex pipelines that would typically be executed manually by a system administrator in most scenarios. This approach ensures seamless scalability across each cluster, irrespective of the underlying physical machines functioning as nodes. A. Kubernetes Dataset Lifecycle Framework The hybrid cloud/edge environment is swiftly emerging as the preferred approach for organizations aiming to strike the right balance between scalability, performance, and security. Consequently, it has become customary for organizations to employ a blend of on-premises data centers (private cloud) and cloud/edge solutions from various providers to store and manage their data. However, numerous challenges arise when applications need to access the data. Developers must possess knowledge of the exact location of data and manage the appropriate credentials to access the designated data sources housing their data. Moreover, access to cloud/edge storage often remains opaque from a cloud management perspective, making it challenging for infrastructure administrators to monitor which containers have access to specific cloud storage solutions. Despite the widespread endorsement of containerized components and microservices as the optimal solution for efficient storage deployment and management in hybrid edge/cloud infrastructure, containerization complicates the access of workloads to shared file systems. Consequently, as more applications run on Kubernetes for batch processing, end-users are burdened with the task of configuring and optimizing data access [38]. To tackle the aforementioned challenges, the Dataset Lifecycle Framework (DLF), an open-source project, is employed, providing containerized applications with seamless and automated access to data sources. DLF enables users to access remote data sources by incorporating a mount point into their containerized workloads. Its primary aim is to enhance usability, security, and performance, offering users a higher level of abstraction for dynamic storage provisioning within their applications. By integrating DLF into Kubernetes pipelines, it can mount object stores as Persistent Volume Claims (PVCs), which serve as segments of storage within the cluster, presenting them to pipelines as a POSIX-like file system. Furthermore, DLF utilizes Kubernetes access control and secret management, eliminating the necessity for pipelines to operate with elevated privileges or manage sensitive secret keys, thereby bolstering platform security. DLF is designed to be cloud-agnostic and due to Container Storage Interface (CSI)6, it is highly extensible to support various data-sources. DLF introduces the Dataset as a Custom Resource Definition (CRD) 7, serving as a reference to existing S3 or NFS data sources. Acting as a declarative structure, the Dataset abstracts access details and serves as a unified point of reference for data within Kubernetes. Creating a CRD represents the initial step in adding custom logic to the Kubernetes cluster. Subsequently, a component embedding domain-specific application logic for the CRD must be created. DLF leverages the Operator-SDK, an open-source component of the Operator Framework8, to provide the necessary tooling and automation for developing these components in an efficient, automated, and scalable manner. Operator-SDK is employed to create the Dataset Operator in DLF, primarily tasked with responding to the creation or deletion of a new Dataset and materializing the specific object. B. Localized Docker Registry The localized Docker registry (LDR) establishes a Docker image registry within each cluster, bringing Docker and VM images closer to the edge devices. A cluster comprises interconnected resources available at the edge capable of hosting instances of application components. This functionality acts as a proactive caching mechanism, optimizing download delays and network traffic. Upon successful deployment of the LDR, an image synchronization daemon known as the Registry Sync Daemon (RSD) is initiated in another container, awaiting triggering messages to populate the LDR instance with new images. Subsequently, each node within the same cluster gains access to these new images via the LDR, facilitating rapid deployment of the associated services. LDR utilizes the Docker 6https://kubernetes-csi.github.io/docs/ 7https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/ custom-resource-definitions/ 8https://operatorframework.io/ registry technology for storing and distributing container images. It integrates the official Docker registry image9with Kubernetes orchestration, a MinIO object storage backend, and a set of automated deployment and configuration scripts. This integration allows LDR to automatically deploy and scale the Docker registry as required, while centrally controlling configuration options such as communication protocols, SSL certificates, credentials, connection ports, and others. This configuration also facilitates fine-tuning of the backend storage, ensuring optimal physical placement of images tailored to the requirements of the application use cases. VM images are treated as objects, stored in a MinIO bucket, and accessed through various means including an S3 API, a web interface, or the DLF functionality added by LDR on top of MinIO. This setup makes the buckets accessible as mountable virtual disks. RSD functions as a synchronization mechanism capable of making real-time decisions regarding image pre-loading. Specifically, it actively monitors a Kafka message broker topic, awaiting triggering messages signaling new or updated images. Moreover, it exposes an API endpoint for manually initiating pre-loading tasks. Upon initiation of a pre-loading task, RSD endeavors to access the target image, utilizing any pre-registered credentials if necessary. Upon successful access, it commences replicating the image to the host cluster via the deployed LDR instance. In addition, RSD offers an extensive array of API endpoints, empowering users to trigger various tasks and retrieve information regarding available images, provided functionalities, and the status of ongoing tasks. IV. EVALUATION A. Performance Analysis: Resource Utilization and Quality of Service The proposed framework is responsible for providing optimized edge storage services including data storage, retrieval and migration tasks, offloading, Quality of Service (QoS) violation prevention and mitigation. More specifically, it provides a reliable, fast, stable and secure shared storage engine, accessible by all devices and users in an edge-cloud. Additionally, it aims at improving the Quality of Experience (QoE) of the end-users by migrating data “close” to them, thus reducing data transfers delays and network utilization. To assess the framework’s effectiveness, various resource utilization and QoS (performance) metrics are collected utilizing the Prometheus system. Data collection occurs at the edge, facilitated by Prometheus agents operating on the edge nodes responsible for data storage. Specifically, data is collected at regular intervals of 5minutes over the operational span of the component, i.e. for the whole duration that the edge storage component is active and waiting for serving data requests. The evaluation metrics employed are divided into two categories: i) Resource consumption - CPU available ( total, used), RAM available (total, used), HDD available (total, used), Network available (total, used) and ii) Performance - Throughput, Data request response time and Network time. The evaluation is 9https://hub.docker.com/ /registry conducted using two deployments of the framework, one in a local and one in a remote cloud. Framework’s performance is scrutinized through a dataset comprising small to medium binary files, ranging from 15KB to 10M B. These files are forming the evaluation dataset that is stored in various MinIO buckets, created and managed by the storage framework in the local and remote deployments. Subsequently, these buckets are mounted onto new pods using the DLF. These pods serve as clients, generating data requests directed at the storage framework, and capturing performance metrics associated with these requests. Figure 2 illustrates the percentage change of various resource utilization metrics -CPU Usage, Memory Usage, Available Memory, Disk Write Latency, Disk IO timeduring intense data transactions and during normal functionality of the node. As the results suggest, the storage framework CPU Usage MEM Usage Avail. Mem Disk Write L DiskIO Time Metric 0 10 20 30 40 50 60 Mean Fig. 2: Percentage change of various resource utilization metrics demonstrates minimal RAM usage on the node, with slight increments observed in CPU utilization and disk operations. This underscores framework’s lightweight nature, suitable for deployment across a range of edge devices. Specifically, RAMrelated metrics show negligible alterations, CPU usage sees a slight uptick, and disk metrics reflect a more significant increase, indicative of intensive I/O operations. Client-side metrics collected to evaluate the impact of the storage framework on QoE, provide a clearer understanding of how it enhances the response times of diverse data requests. Figure 3 and Figure 4 depict the comparison between read, write, and delete operations for local and remote deployments, respectively. Given the object store nature of MinIO, it is evident that write operations require more time compared to read and delete operations. Conversely, read and write operations exhibit minimal disparity, with the primary difference being the network delay for the final file transfer. Notably, these evaluations involve file transfers of multiple small to mediumsized files. The comparison of operation response times reveals similar trends but on different scales. For the local deployment, response times range from 3 to 17 milliseconds, whereas for the remote deployment, response times vary between 84 to 450 milliseconds. This distinction becomes more pronounced when directly comparing response times, as depicted in Figure 5. Specifi0 5 10 15 20 7284 7613 10344 10453 13492 16098 17682 20407 20923 22298 24041 24381 24750 25365 26122 27608 29083 30834 31925 32403 32814 32815 32831 33636 35606 36293 36425 36925 37134 37213 37622 38710 39503 40272 40608 41367 41773 43313 45947 47416 47535 48013 52195 52254 54562 56208 57915 58677 61635 62535 66362 66599 68447 71501 73397 75293 78085 84118 95528 98018 108075 118111 126940 132768 179772 193385 211651 216029 254684 625285 local_delete local_write local_read Fig. 3: Read, Write and Delete operation response times in milliseconds for the local deployment 0 100 200 300 400 500 10344 17682 31925 36293 7613 10453 13492 20923 22298 24381 24750 26122 29083 32814 35606 39503 40608 48013 52195 54562 68447 7284 16098 20407 24041 27608 30834 32403 32815 32831 33636 37134 41367 41773 43313 45947 52254 56208 58677 73397 25365 36425 36925 37213 37622 38710 40272 47416 47535 62535 66362 71501 61635 78085 108075 98018 118111 57915 84118 126940 66599 216029 95528 132768 75293 193385 211651 179772 254684 625285 remote_delete remote_write remote_read Fig. 4: Read, Write and Delete operation response times in milliseconds for the remote deployment cally, request response times for the local deployment consistently remain under 20 milliseconds across all file operations, significantly outperforming the remote one. loc. Delete loc. Read loc. Write rem. Delete rem. Read rem. Write Operation 0 25 50 75 100 125 150 175 200 Duration Fig. 5: Comparison of response times for various operations for the remote and local deployments B. Latency Analysis: Key Considerations in Edge Application Delivery The seamless delivery of applications on resourceconstrained edge devices presents unique challenges due to limited network bandwidth, latency constraints, and intermittent connectivity. Moreover, the substantial size of application images can strain the limited network bandwidth and introduce significant latency when downloaded from remote repositories. The localized Docker registry (LDR) introduced by the proposed framework serves as a local cache on container images, enabling edge nodes/devices to retrieve these images from nearby storage instead of fetching them over the network. The proposed framework has been deployed in the context of two European funded projects; CHARITY10 and ACCORDION11. These projects offer a wide set of real life use cases that provide ample opportunity for testing and evaluating the proposed solution. The feasibility and efficiency of the LDR is evaluated through the examination of two specific use case scenarios: Collaborative VR medical training and Multiplayer Mobile Gaming. These use cases were selected for their unique characteristics and specific demands. Specifically, the collaborative VR medical training use case necessitates deploying a large VM image, approximately 25GB in size, close to the end users. On the other hand, the multiplayer mobile gaming use case needs multiple small files, sized less than 3MB, to be exchanged in real-time between multiple participating nodes. During the pilot evaluations of the collaborative VR medical training application, retrieving the 25GB VM image from a remote repository caused significant network congestion, leading to delays in both image download and other concurrent network operations. By pre-positioning the VM image within the LDR on the same edge node prior to initiating a new VR session request, the transfer delay was minimized, as the deployment of the new VM occurred from a local repository instead of a remote one. The RSD was configured to pre-load the application images into the LDR. In initial tests, without LDR pre-loading, the application deployment time exceeded 10 minutes, occasionally reaching up to 20 minutes. With LDR pre-loading, this deployment time was reduced to 1-2 minutes. This means that LDR achieved a deployment time up to 10 times faster than raw Kubernetes deployment. In the multiplayer mobile gaming use case, multiple small files needs to be exchanged in real-time between multiple participating nodes. During the pilot phase evaluation, significant improvements were observed, particularly in operations like write, read, and delete, with a six-fold reduction in workload. In addition, the LDR solution significantly aids in reducing both network load and deployment time for new game servers by strategically situating the game server Docker images near the edge nodes that will host them, before they are actually needed. By deploying a localized registry on the edge nodes, the burden of downloading these large application images from remote repositories can be alleviated. The inclusion of LDR streamlines the process of storing and distributing container images, providing improved control, scalability, and optimized deployment capabilities at the edge. The evaluation results reveal a significant reduction in application deployment time, indicating the positive impact of the proposed solution. C. Energy Efficiency Analysis: Considerations for Energy Reduction As edge computing gains traction as a solution to latency challenges inherent in traditional cloud architectures, researchers are increasingly focusing on optimizing the energy 10https://www.charity-project.eu/ 11https://www.accordion-project.eu/ efficiency of edge systems. A growing body of research explores various aspects of energy-aware designs, computing architectures, operating systems, services, and applications. In addition, tools and methodologies for measuring and monitoring energy consumption in edge environments play a crucial role in guiding optimization efforts and ensuring efficient resource utilization. In response to the growing focus on optimizing energy efficiency in edge computing, our research endeavors to assess the energy consumption facilitated through the use of the proposed LDR component. Through a detailed examination of energy consumption patterns across different facets of the framework, we seek to provide valuable insights for enhancing energy efficiency in edge computing deployments. For the purpose of evaluation, the primary experiment centered on comparing the energy consumption associated with pulling Docker images of different sizes from Dockerhub and the LDR. The installation of the framework occurred on one Raspberry Pi, while another Raspberry Pi was employed to retrieve (pull) images from either Dockerhub or the LDR. Table I details the hardware resources of the two Raspberry Pi devices. CPU RAM Model 4-core 4 GB Raspberry Pi Model 4B TABLE I: Hardware resources of the Raspberry Pi devices The first challenge in conducting the experiment was finding a tool capable of monitor energy consumption. Upon reviewing the documentation of Prometheus12, it was discovered that its agent, Node Exporter, could monitor hosts. Consequently, EdgeCloud Mon [39] was employed to establish a monitoring stack comprising both Prometheus and Node Exporter as Kubernetes pods. Remote Site PowerJoular Registry Local Cluster Monitors Energy Consumption Raspberry Pi Raspberry Pi Pulls Docker Images Dockerhub Pulls Docker Images Fig. 6: Energy consumption experimental structure EdgeCloud Mon13 is an open-source lightweight monitoring tool designed to oversee Kubernetes clusters. However, Node 12https://prometheus.io/docs/guides/node-exporter/ 13https://github.com/Efficient-Computing-Lab/EdgeCloud-Mon Exporter failed to provide the specified metric for ARM architectures. As a result, PowerJoular [36] emerged as an alternative solution for monitoring CPU energy consumption (W atts). PowerJoular was deployed on the Raspberry Pi responsible for image retrieval to ascertain its energy consumption during the pulling process. The structure of the experiment is depicted in Figure 6. As previously mentioned, the objective is to retrieve images of various sizes from both a remote registry and the LDR. Details of the images utilized, along with their sizes, are provided in Table II. Both Ubuntu14 and CIMG15 images are public available. The Ubuntu Docker image is the smallest in size, whereas the CIMG for Android is the largest, comprising multiple API SDKs, command line tools, build tools, Ant, Gradle, and Google Cloud SDK. The Forecasting Docker image is a custom private image developed by us, incorporating TensorFlow and Keras, thus falling into the medium size category. Docker Image Size Ubuntu 56.4 MB Forecasting 2.4 GB CIMG 6.4 GB TABLE II: Docker Images The experimental results indicated a slight reduction in energy consumption when retrieving a Docker image from the LDR. Specifically, for the Ubuntu Docker image, the average energy consumption was 3.41 W atts when obtained from LDR, compared to 3.52 W atts when pulled from a remote registry ((Figure 7). However, it’s worth noting that this case may not be entirely representative due to the lower frequency of CPU power samples per unit of time during image retrieval from the LDR. Conversely, for larger images requiring more Fig. 7: Energy Consumption Comparison - remote versus LDR (Ubuntu Docker Image): The orange line represents CPU energy consumption (W atts) when pulling from a remote registry, while the blue line represents CPU energy consumption when pulling from LDR. 14https://hub.docker.com/ /ubuntu 15https://hub.docker.com/r/cimg/android retrieval time, the experiment demonstrated a decrease in CPU power when utilizing the LDR. In the case of the Forecasting Docker image, the average energy consumption was 3.83 W atts from LDR, slightly lower than the 3.86 W atts from a remote registry (Figure 8). Similarly, for the CIMG Docker Fig. 8: Energy Consumption Comparison - remote versus LDR (Forecasting Docker Image): The orange line represents CPU energy consumption (W atts) when pulling from a remote registry, while the blue line represents CPU energy consumption when pulling from LDR. image, the average energy consumption was 3.83 W atts from LDR, compared to 4.06 W atts from a remote registry (Figure 9). However, fetching images from a remote registry results in prolonged retrieval times, leading to heightened latency compared to the LDR. Consequently, this extended process leads to sustained high energy consumption as the edge device persists in pulling the image. Fig. 9: Energy Consumption Comparison - remote versus LDR (CIMG Docker Image) Pull: The orange line represents CPU energy consumption (W atts) when pulling from a remote registry, while the blue line represents CPU energy consumption when pulling from LDR. V. CONCLUSION The challenges of implementing an efficient edge-enabled storage system are compounded by the heterogeneous and diverse nature of the edge, as well as its limited resources. This paper introduced an efficient edge storage solution, offering a comprehensive framework that focuses on performance (QoS), latency minimization, and energy efficiency. Our evaluations reveal significant improvements achieved through reductions in data operation execution time, performance enhancements, alleviation of network infrastructure strain, and optimization of energy efficiency. The benefits yielded by the proposed solution can be further augmented by integrating artificial intelligence mechanisms such as pro-active caching of data and automated scaling. To incorporate these functionalities, a new sub-component needs to be implemented that will be trained on various use cases creating profiling processes and predictive models that will empower an artificial intelligence module to make real-time or predictive decisions, automatically triggering actions within the clusters and the storage enabler. ACKNOWLEDGMENT This project has received funding from the Horizon Europe Framework Programme under Grant agreement No 101135775 (PANDORA). Additionally, it received funding from the European Union’s Horizon Europe Research and Innovation Programme under grant agreement No. 101120990 (SOPRANO). This paper reflects only the authors’ view and the Commission is not responsible for any use that may be made of the information it contains. REFERENCES [1] M. Chiang and T. Zhang, “Fog and iot: An overview of research opportunities,” IEEE Internet of things journal, vol. 3, no. 6, pp. 854– 864, 2016. [2] Z. Hao, E. Novak, S. Yi, and Q. Li, “Challenges and software architecture for fog computing,” IEEE Internet Computing, vol. 21, no. 2, pp. 44–53, 2017. [3] Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young, “Mobile edge computing—a key technology towards 5g,” ETSI white paper, vol. 11, no. 11, pp. 1–16, 2015. [4] M. Patel, B. Naughton, C. Chan, N. Sprecher, S. Abeta, A. Neal et al., “Mobile-edge computing introductory technical white paper,” White paper, mobile-edge computing (MEC) industry initiative, vol. 29, pp. 854–864, 2014. [5] M. Satyanarayanan, “The emergence of edge computing,” Computer, vol. 50, no. 1, pp. 30–39, 2017. [6] D. Sabella, A. Alleman, E. Liao, M. Filippou, Z. Ding, L. G. Baltar, S. Srikanteswara, K. Bhuyan, O. Oyman, G. Schatzberg et al., “Edge computing: from standard to actual infrastructure deployment and software development,” ETSI White paper, pp. 1–41, 2019. [7] L. Ferrucci, M. Mordacchini, M. Coppola, E. Carlini, H. Kavalionak, and P. Dazzi, “Latency preserving self-optimizing placement at the edge,” in Proceedings of the 1st Workshop on Flexible Resource and Application Management on the Edge, 2020, pp. 3–8. [8] A. Makris, A. Boudi, M. Coppola, L. Cordeiro, M. Corsini, P. Dazzi, F. D. Andilla, Y. G. Rozas, M. Kamarianakis, M. Pateraki et al., “Cloud for holography and augmented reality,” in 2021 IEEE 10th International Conference on Cloud Networking (CloudNet). IEEE, 2021, pp. 118– 126. [9] A. Makris, I. Kontopoulos, E. Psomakelis, S. N. Xyalis, T. Theodoropoulos, and K. Tserpes, “Performance analysis of storage systems in edge computing infrastructures,” Applied Sciences, vol. 12, no. 17, p. 8923, 2022. [10] T. Theodoropoulos, A. Makris, A. Boudi, T. Taleb, U. Herzog, L. Rosa, L. Cordeiro, K. Tserpes, E. Spatafora, A. Romussi et al., “Cloud-based xr services: A survey on relevant challenges and enabling technologies,” Journal of Networking and Network Applications, vol. 2, 2022. [11] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” IEEE internet of things journal, vol. 3, no. 5, pp. 637–646, 2016. [12] I. Psaras, O. Ascigil, S. Rene, G. Pavlou, A. Afanasyev, and L. Zhang, “Mobile data repositories at the edge,” in USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), 2018. [13] B. Confais, A. Lebre, and B. Parrein, “Performance analysis of object store systems in a fog and edge computing infrastructure,” in Transactions on Large-Scale Data-and Knowledge-Centered Systems XXXIII. Springer, 2017, pp. 40–79. [14] I. Clarke, O. Sandberg, B. Wiley, and T. W. Hong, “Freenet: A distributed anonymous information storage and retrieval system,” in Designing privacy enhancing technologies. Springer, 2001, pp. 46– 66. [15] K. Sonbol, ¨ O. ¨ Ozkasap, I. Al-Oqily, and M. Aloqaily, “Edgekv: Decentralized, scalable, and consistent storage for the edge,” Journal of Parallel and Distributed Computing, vol. 144, pp. 28–40, 2020. [16] A.-C. Nicolaescu, S. Mastorakis, and I. Psaras, “Store edge networked data (send): A data and performance driven edge storage framework,” in IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 2021, pp. 1–10. [17] I. Lujic, V. De Maio, and I. Brandic, “Efficient edge storage management based on near real-time forecasts,” in 2017 IEEE 1st International Conference on Fog and Edge Computing (ICFEC). IEEE, 2017, pp. 21–30. [18] J. Xing, H. Dai, and Z. Yu, “A distributed multi-level model with dynamic replacement for the storage of smart edge computing,” Journal of Systems Architecture, vol. 83, pp. 1–11, 2018. [19] A. Makris, K. Tserpes, and T. Varvarigou, “Transition from monolithic to microservice-based applications. challenges from the developer perspective,” Open Research Europe, vol. 2, p. 24, 2022. [20] B. I. Ismail, E. Mostajeran Goortani, M. B. Ab Karim, W. Ming Tat, S. Setapa, J. Y. Luke, and O. Hong Hoe, “Evaluation of docker as edge computing platform,” in 2015 IEEE Conference on Open Systems (ICOS), 2015, pp. 130–135. [21] N. Gupta, K. Anantharaj, and K. Subramani, “Containerized architecture for edge computing in smart home : A consistent architecture for model deployment,” in 2020 International Conference on Computer Communication and Informatics (ICCCI), 2020, pp. 1–8. [22] M. Gazzetti, A. Reale, K. Katrinis, and A. Corradi, “Scalable linux container provisioning in fog and edge computing platforms,” in EuroPar 2017: Parallel Processing Workshops: Euro-Par 2017 International Workshops, Santiago de Compostela, Spain, August 28-29, 2017, Revised Selected Papers 23. Springer, 2018, pp. 304–315. [23] W. Kangjin, Y. Yong, L. Ying, L. Hanmei, and M. Lin, “Fid: A faster image distribution system for docker platform,” in 2017 IEEE 2nd International Workshops on Foundations and Applications of Self* Systems (FAS*W), 2017, pp. 191–198. [24] L. A. Dias Knob, F. Faticanti, T. Ferreto, and D. Siracusa, “Communitybased placement of registries to speed up application deployment on edge computing,” in 2021 IEEE International Conference on Cloud Engineering (IC2E), 2021, pp. 147–153. [25] S. Becker, F. Schmidt, and O. Kao, “Edgepier: P2p-based container image distribution in edge computing environments,” in 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC), 2021, pp. 1–8. [26] C. Zheng, L. Rupprecht, V. Tarasov, D. Thain, M. Mohamed, D. Skourtis, A. S. Warke, and D. Hildebrand, “Wharf: Sharing docker images in a distributed file system,” in Proceedings of the ACM Symposium on Cloud Computing, ser. SoCC ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 174–185. [27] A. Makris, E. Psomakelis, E. Carlini, M. Mordacchini, T. Theodoropoulos, P. Dazzi, and K. Tserpes, “Pro-active component image placement in edge computing environments,” Future Generation Computer Systems, vol. 157, pp. 344–359, 2024. [28] T. Theodoropoulos, A. Makris, E. Psomakelis, E. Carlini, M. Mordacchini, P. Dazzi, and K. Tserpes, “Gnosis: Proactive image placement using graph neural networks & deep reinforcement learning,” in 2023 IEEE 16th International Conference on Cloud Computing (CLOUD). IEEE, 2023, pp. 120–128.