FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems

FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems Herbert Woisetschläger [email protected] Technical University of Munich Germany Alexander Erben [email protected] Technical University of Munich Germany Ruben Mayer [email protected] University of Bayreuth Germany Shiqiang Wang [email protected] IBM Research United States Hans-Arno Jacobsen [email protected] University of Toronto Canada Abstract Federated Learning (FL) has become a viable technique for realizing privacy-enhancing distributed deep learning on the network edge. Heterogeneous hardware, unreliable client devices, and energy constraints often characterize edge computing systems. In this paper, we propose FLEdge, which complements existing FL benchmarks by enabling a systematic evaluation of client capabilities. We focus on computational and communication bottlenecks, client behavior, and data security implications. Our experiments with models varying from 14K to 80M trainable parameters are carried out on dedicated hardware with emulated network characteristics and client behavior. We find that state-of-the-art embedded hardware has significant memory bottlenecks, leading to 4 × longer processing times than on modern data center GPUs. CCS Concepts: •Computing methodologies → Distributed artificial intelligence;•General and reference → Performance. Keywords: Federated Learning, Performance Benchmark ACM Reference Format: Herbert Woisetschläger, Alexander Erben, Ruben Mayer, Shiqiang Wang, and Hans-Arno Jacobsen. 2024. FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems. In 24th International Middleware Conference (MIDDLEWARE ’24), December 2–6, 2024, Hong Kong, Hong Kong. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/3652892.3700751 MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong ©2024 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0623-3/24/12 https://doi.org/10.1145/3652892.3700751 1 Introduction Federated Learning (FL) has become an established middleware abstraction to facilitate distributed and privacypreserving deep learning (DL) [ 6 , 23 , 60 ]. With increasing access to data by eliminating the need to transfer data to a central location, FL reduces network communication and fosters privacy. Privacy in FL applications is typically realized by introducing differential privacy [ 2 , 42 ]. There is ample research on benchmarking FL workloads, which focuses on different fields of application, data heterogeneity [ 6 , 23 ], FL aggregation strategies [ 23 , 35 ], DL model personalization [ 9 ], and federated hyperparameter optimization [ 66 ]. However, these benchmarks typically target server-grade hardware and are simulation-based [ 6 , 23 ]. As such, an often neglected factor is the type of system FL workloads are deployed to: edge computing systems. Edge computing is a technology for managing a fleet of devices distributed across geographies [ 57 ], serving latencycritical or data-sensitive tasks [ 24 , 50 ]. Systems at the edge are characterized by diverse hardware and low client reliability [ 13 ], varying network quality [ 22 , 36 ], and energy constraints [ 30 ]. Additionally, FL workloads are prone to a high degree of data heterogeneity [ 18 , 43 , 54 ]. This creates a set of challenges when placing FL workloads at the edge since servers and clients are strongly intertwined. While clients facilitate the training on their local data, servers use FL strategies (e.g., FedAvg) to aggregate the trained client models into a global model that generalizes over all clients. As clients in edge computing systems are often embedded devices, their computational capabilities and points of most energy-efficient operations vary. Additionally, devices can be more than 5 years old in current systems that require high functional safety [ 14 , 53 ]. Older devices usually do not have DL accelerating hardware, while more recently introduced platforms are powerful system-on-a-chip (SoC) devices carrying a GPU [ 46 , 48 ], leading to a major gap in computational capabilities. In practice, this requires careful deployment planning for FL workloads, especially when large language models (LLMs) are involved [15]. 88 This work is licensed under a Creative Commons Attribution‐NonCommercial‐ ShareAlike International 4.0 License. MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Woisetschläger et al. FLEdge extends existing FL benchmarking works by introducing systematic studies on client behavior, communication and energy efficiency, and hardware diversity. As these dimensions are critical to FL system design, we help researchers and practitioners develop new FL efficiency methods and drive practical adoption of FL, especially in edge computing systems. Client behavior. State-of-the-art benchmarks for FL applications provide a variety of DL models, datasets, and FL strategies [ 6 , 23 , 69 ]. Yet, no work systematically studies the effects of client dropouts on the overall training performance in real systems. Client dropouts will likely happen in edge computing systems where clients and network connectivity can be unstable [ 58 ], which makes it an integral part to analyze, especially with regard to the interference effects with differential privacy (DP) that depend on the number of clients actually submitting model updates during a training round. Communication efficiency. Equally important to the client behavior are network conditions, as it is an integral component for an efficient edge deployment. However, many contributions discuss communication efficiency in FL systems [ 32 , 41 , 51 , 68 ] without providing metrics that help estimate the viability of a workload to be federated into an edge computing system with wireless communication, like 4G [1]. Energy efficiency. Existing benchmark works predominantly focus on computational speed [ 35 , 69 ]. However, two key design variables in edge computing systems are computational and energy efficiency, which both contribute to the overall energy efficiency of an FL system. Energy is a scarce resource in edge computing systems and ultimately determines how fast we can process a workload depending on client hardware. Hardware diversity. While simulated embedded devices can be used to evaluate the scalability of FL algorithms in depth, they neglect the underlying processor architecture and memory bandwidth. This results in significant performance differences when deploying on different types of hardware and limits the applicability to edge computing systems [5, 23, 57]. Overall, FLEdge aims to improve the understanding of FL application deployment in edge computing systems and specifically answers the following research question: How well do state-of-the-art FL workloads respond to deployment in edge computing systems? Our contributions are as follows: (1) We introduce FLEdge – a hardware-centric benchmarking suite for FL workloads in system-heterogeneous environments. FLEdge extends the widely used FL framework Flower [ 5 ] with a module to control client behavior, an adaptive user-level differential privacy adapter that accounts for client dropouts, a network emulator, and extensive monitoring capabilities to evaluate the computational efficiency of embedded devices. Our code base is publicly available.1 (2) FLEdge provides a holistic evaluation pipeline for FL workloads w.r.t. client behavior and communication efficiency. The client behavior module allows flexible modeling of client reliability based on environmental conditions and the intended deployment target. This enables practitioners to quickly evaluate the robustness of their FL workloads. We also add a freely controllable network adapter that allows for the emulation of realistic connectivity and the exploration of the viability of deploying FL workloads in edge computing systems. (3) We conduct extensive experiments on computational capabilities and energy efficiency of widely used embedded devices when running FL workloads. Our experimental results show that the stateof-the-art embedded AI accelerator is challenged with backpropagation, which calls for alternative solutions that better use the hardware architecture. Our experiments on client behavior in conjunction with differentially private FL workloads show a high sensitivity of model quality w.r.t. client reliability. Our work is structured as follows. In Section 2, we outline the requirements analysis for our benchmark. In Section 3, we introduce our methodology. In Section 4, we outline our experimental design, including datasets, DL models, FL strategies, and practical considerations. Section 5 discusses experimental results of our benchmark. Section 6 contains related work. Section 7 discusses lessons learned and in Section 8, we conclude our work. 2 Requirement Analysis As edge computing applications are gaining popularity, especially in realistic environments without perfect control over system parameters, challenges in designing efficient systems arise. On a client, we care about energy efficiency and training reliability, as well as privacy, such that the data is processed in a timely and secure manner. At the same time, there are often (mobile) clients with heterogeneous computing resources involved, which may negatively affect the training speed. That said, we need a hardware-centric benchmarking framework to assess the feasibility of FL applications on the network edge. Furthermore, as FL aims to be a communication-efficient ML paradigm, cost and efficiency are key decision variables. The objective of our work is to evaluate the end-to-end FL pipeline, investigating system components that have come short in existing research: 1https://github.com/laminair/FLEdge. 89 FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Client behavior. Generally, client reliability is key for a distributed application. While for most edge computing applications, tasks run individually on clients, e.g., in the context of CV applications with MobileNet models [ 26 ], for FL, there are significant dependencies between clients and the server. FL typically involves an iterative training procedure where a set of 𝑁 clients trains a shared model over multiple rounds 𝑡∈𝑇 . In each round, a subset 𝑀∈𝑁 clients is selected for training. Training an FL model involves aggregating multiple client updates into a single model: 𝑤𝑡+1=1 |𝑀|Í𝑚∈𝑀𝑤𝑡 𝑚 . For this, we use FL strategies like FedAvg [ 41 ]. In edge computing, clients are unreliable compared to data center services and may show a high failure rate. Thus, 𝑀 can vary within a training round, and consequently, fewer client updates will be aggregated. As such, it is a requirement to test the robustness of state-of-the-art FL strategies with varying realistic client behavior typically found in edge computing systems. Privacy. Another key concern for FL systems is data privacy, as clients are unwilling to share their data. FL already increases the level of data privacy to a certain extent [ 41 ] but does not provide a guarantee for privacy to clients. Formal methods like (𝜖, 𝛿) - Differential Privacy (DP) can be applied to obtain such a guarantee. DP can be applied on sampleor user-level, where the latter is of special interest for FL [ 2 , 42 ]. As clients in FL systems only share their model updates, it is sufficient to apply DP to model updates and achieve the same guarantees as with sample-level DP [ 67 ]. The objective of user-level DP is to deny membership inference or gradient inversion attacks. This only works if the noise level is calculated appropriately based on the number of clients that actually submit a model update in an FL training round. As such, bringing together robustness and data privacy requires user-level DP to adapt to suddenly failing clients is a requirement. Communication efficiency. Along with varying client behavior and privacy levels, communication is an integral part of ensuring a high service quality of FL workloads in edge computing systems. In many use cases, tasks are running independently on clients [ 58 ]. However, in FL systems, there is a strong coupling between clients and the server as the global model state is maintained on the server. Therefore, we are interested in the processing latency and every aspect of network communication for FL on the edge. Typically, there are three scenarios for deploying edge networks. The first and most reliable is a wired connection, often found in factories [ 7 ]. The second is a high-bandwidth wireless connection, such as 4G LTE [ 1 ], and the third is a low-bandwidth and high latency connection, often found in remote areas [ 29 , 58 ]. The anticipation of network connectivity is key as to whether it makes sense to federate the training process for a given DL model. Interestingly, existing works measure network conditions in different ways. Some focus on available bandwidth [ 45 , 70 ], and others focus on the amount of data transmitted [ 61 , 64 ]. As such, there is a necessity for a comprehensive and unifying metric that quickly answers the question of whether it is viable to deploy an FL workload into an edge computing system with heterogeneous network conditions based on the amount of time we use on computation vs. communication. Communication cost. FL applications in edge computing systems are often operated over wide-area networks [ 63 ], involving a multitude of network hops. While it is theoretically possible to measure and report the energy cost per hop, it is challenging to achieve in a real-world system since each networking component is often owned by a different entity, such as the client owner, the internet service provider, and the cloud operator. Thus, we require a solution to reliably estimate the total communication cost of an FL workload. Energy efficiency. To evaluate energy efficiency, we need a reliable method to evaluate the characteristics of existing devices for FL workloads. In edge deployments, clients have previously been employed as a data collection platform only and are now required to run computationally intensive FL workloads. This substantially increases energy consumption, which is often a challenge due to limited power availability [ 11 ]. To mitigate these effects, hardware acceleration on embedded devices has been introduced [ 46 ]. For instance, GPUs generally provide a significant performance benefit over CPUs while being more energy efficient and cost-saving at the same time [ 55 ]. Therefore, it is a requirement to investigate the energy efficiency of different hardware w.r.t. their projected workloads. While DL workloads in the cloud are hard to measure for their energy efficiency due to virtualized hardware [ 19 ], the benefit of embedded devices is their SoC design that allows for measurement of the real-time energy consumption of the entire device and all individual device components (e.g., CPU, GPU). Hardware diversity. Edge computing is designed around the pattern of offloading computational tasks to edge devices, usually embedded devices in the proximity of a data source. This provides better control over the trade-off between computation and communication [ 39 ]. Yet, a major challenge in these systems is hardware diversity, either due to long product life cycles in industrial systems or due to a lack of infrastructure control. For instance, widely used embedded devices for industrial edge require extensive reliability testing and therefore include hardware as old as five years [ 53 ]. As such, it is necessary to benchmark not only the most recently released devices but also older generations still being used in a wide variety of systems [ 65 ]. Also, embedded devices are becoming increasingly capable as modern platforms, such as the NVIDIA Jetson AGX Orin, often entail integrated GPUs. This makes benchmarking hardware more complex and is also likely to change the energy demand of devices for workloads. A hardware-centric benchmark must, therefore, account for a variable number of components on an SoC-based embedded device. 90 MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Woisetschläger et al. NVIDIA Jetson Nano & AGX Orin Embedded SoCs incl. GPUs for DL acceleration with limited availability Training Embedded Devices on the Edge Raspberry Pi 4B 4 GB CPU-only embedded devices with broad availability in existing use cases Heterogeneous Client Behavior & Privacy Model Aggregation & Validation Deep Learning Workload Coordination Initial Model Client Server FedAvg FedAdam FedAdaGrad FedYogi FedProx qFedAvg User-level DP with varying noise levels & varying client dropout likelihoods CV NILM NLP Trained Models Client - Server Communication … 1 / 2 4 Key Embedded devices for FL training Communicating untrained model parameters Communicating trained/updated model parameters 1 2 Hardware heterogeneity: Eval. of computational limits of stateof-the-art embedded devices Energy efficiency Detection of computational bottlenecks when scaling FL workloads on clients 3 4 Communication efficiency: Viability evaluation of FL workloads in heterog. networks Client behavior: Effects of client dropouts on differentially private FL workloads 3Benchmark Objectives Figure 1. Federated Learning protocol for one training round from a system perspective with 1 - 4 indicating the focus areas for our work and the benchmark subjects for FLEdge. Our system uses Flower as the underlying FL framework and implements each component in a modular and extensible manner. 3 FLEdge: Benchmarking Framework Client behavior. As robustness is important to all FL workloads, and edge computing systems suffer from heterogeneous client behavior, we configure our testbed to simulate realistic client failures. For FL workloads, it does not matter whether the failure origin is hardware or communication. We model the client dropout as an independent binomial distribution since we assume clients will reenter training for future rounds and clients do not interfere with one another, 𝑝𝑑 𝑚=𝑃(𝑚)=Bin(𝑝).(1) We vary the failure likelihood 𝑝 from 0% to 50% in our experiments, with 0% being very reliable clients as they are often found in factories and 50% representing unreliable clients, e.g., mobile clients with wireless connetion. For example, the FL strategy FedAvg is extended with 𝑝𝑑 𝑚as follows, 𝑤𝑡+1= 1 |𝑀|∑︁ 𝑚∈𝑀 (𝑤𝑡 𝑚·𝑝𝑑 𝑚).(2) The behavioral pattern can be freely exchanged for other patterns that suit particular use cases. Communication efficiency. To transfer the model updates from the clients to the server, we need to consider communication efficiency and find a method to express the viability of adding an embedded computing platform under given network conditions. To evaluate communication efficiency, we use granularity [ 27 ], which is an established metric for evaluating the training efficiency in a distributed system by comparing the computation time against the communication time, 𝐺= 𝑇computation 𝑇communication .(3) 𝐺≫ 1is considered favorable for distributing workloads, while 𝐺≃ 1or 𝐺< 1indicates little utility when distributing a workload to a given system or device. Our benchmark allows the emulation of different realistic connectivity profiles per client to explore the effect of 𝐺. Communication cost. It is important to consider the communication costs of scaling an FL system since we operate in a data-parallel regime. While theoretically, we can scale the number of clients such that we fully saturate the server’s network bandwidth, the effects of a large number of clients are limited [ 8 ]. However, to establish the scalabilitycost trade-off, we employ the per-bit communication cost model that allows us to assert a client with constant communication costs per model update transmission [ 28 ]. The model allows us to calculate the energy we consume at every hop in a network, 𝑃t=𝐸𝑡· B =(𝑛as ·𝐸as +𝑛lc ·𝐸lc +𝑛lb ·𝐸lb +𝐸bng +𝑛𝑒·𝐸𝑒+𝑛𝑐·𝐸𝑐+𝑛𝑑·𝐸𝑑) · B.(4) 𝐸as , 𝐸lc , 𝐸lb , 𝐸bng , 𝐸𝑒 , 𝐸𝑐 , 𝐸𝑑 are the per-bit energy consumption of 𝑛as edge ethernet switches, zero or one 𝑛lc LTE client modem, zero or one 𝑛lb LTE base station, the broadband network gateway (BNG), one or more edge routers 𝑛𝑒 , one or more core routers 𝑛𝑐 , and one or more data center Ethernet switches 𝑛𝑑 , respectively. B denotes the size of a model update in bits. Computational efficiency. Aside from communication in edge computing systems, energy is a vital component of system efficiency as it may be a scarce resource in remote areas. Therefore, our testbed contains real-time measurement capabilities and control mechanisms to limit the power draw of each embedded device. This enables us to explore the deployment characteristics of FL workloads in a wide variety of edge computing systems. We measure the energy efficiency as the throughput in samples per second (denoted 91 FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Train Val Test Split 0 2 4 6 8 Samples (in 100) (a) BLOND Train Val Test Split 0 5 10 15 20 Samples (in 1K) (b) FEMNIST Train Val Test Split 0 50 100 150 Samples (in 1K) (c) Shakesp. Train Val Test Split 0 1 2 3 4 Samples (in 1K) (d) Samsum Figure 2. The non-IID subsets for our clients are sampled from a Dirichlet distribution (𝛼=1). as 𝑄 ) divided by the average power draw (denoted as 𝑊 ) over the experiment time: 𝜂𝑒= Q 𝑊.(5) Hardware diversity. To develop a detailed understanding of where potential inefficiencies could come from, we employ a micro-benchmark to study the effect of hardware diversity on the training performance of FL workloads. It focuses on the timing of the DL step times, namely the batch loading, forward, loss calculation, backward, and optimizer steps. With this, we get a detailed understanding of computational inefficiencies on embedded devices and uncover differences to data center hardware that potentially become bottlenecks for state-of-the-art FL workloads. Additionally, we use the PyTorch profiler with Kineto support to investigate performance bottlenecks in our FL clients. This provides us with the runtime of individual kernels, highlighting potential bottlenecks. 3.1 Protocol Our setup follows the widely used client-server architecture for FL workloads [ 6 , 23 , 69 ] and is depicted in Figure 1. The four focus areas and their practical utility are discussed in the following. Training. Training is facilitated entirely by the clients. They receive the hyperparameter configuration along with the model parameters from the server and train for exactly 1 local epoch before communicating the updates to the server. We do this for our client dropout and privacy experiments to isolate client failure effects on model quality and avoid any interference with data drift [31, 44]. Communication. We integrate a controllable network interface on each client as edge computing systems that involve embedded devices can be found in various environments (e.g., production lines or remote weather sensing stations). In addition to the steady 1 Gbit/s network link, which resembles industrial settings, our testbed allows us to emulate wireless communication (e.g., LTE, 3G) per client for remote setups. Privacy. A core promise of FL is to preserve privacy. For user-level DP, the model aggregation method is extended by adding Gaussian noise to the model weights (cf. Equation (6)). Table 1. The datasets in our benchmark vary in modality, size, and data heterogeneity to resemble a large variety of real-world use cases. Pipeline Dataset Name Total Samples Samples per Device Format Size NILM BLOND 13,164 234±132 HDF5 5.46 GB CV FEMNIST 814,255 13,958±2,106 PNG 3.34 GB NLP Shakespeare 4,226,054 75,131±18,281 TXT 0.38 GB NLP Samsum 11,780 491±723 TXT 0.012 GB With the additional noise 𝜉∼ N ( 0 , 𝐼𝜎2 Δ) , we introduce a natural trade-off between model accuracy and the level of privacy. The variance 𝜎2 Δ depends on the number of clients sampled in a training round and on their L2 update norm, as with more heterogeneous client updates, more noise has to be added to ensure (𝜖,𝛿)-DP, 𝑤𝑡+1= 1 |𝑀|∑︁ 𝑚∈𝑀 ((𝑤𝑡 𝑚+𝑧·𝜉) · 𝑝𝑑 𝑚)(6) The strength of the privacy guarantee is measured by the privacy budget 𝜖 where lower values are better and provide higher levels of privacy [ 16 ]. 𝜖 depends on 𝛿 , the likelihood of unintentional input data leakage. Usually, 𝛿 is set to a value equal to the inverse of the anticipated total number of samples in a dataset (e.g., for the BLOND dataset, it would amount to 𝛿=1 13,164 , see Section 4.3). As such, our system entails server-side user-level DP. In this way, we can adjust the DP noise levels on the server based on the number of updates received rather than having to request a partial recalculation of DP model updates from each client. 3.2 Practical Assumptions & Configuration As we are interested in the performance of FL workloads, we configure our testbed to serve under realistic conditions for industrial edge computing systems. We describe the configuration along Figure 1. Training. Clients are dedicated to the FL workload only and assign all available resources to the task with the respective optimal hyperparameter configuration as depicted in Table 3. We choose a client participation rate of 20% for all experiments, but it can be freely configured for other scenarios. Communication. The network is configured to either emulate a factory environment with a synchronous 1 Gbit/s link [ 53 ] or to run a client-specific 4G LTE wireless [ 1 ] network with higher latency and asynchronous 15 MBit/s upload and 40 MBit/s download bandwidths that is often found when using mobile or remotely embedded devices. The LTE bandwidths represent the global average connectivity [ 56 ]. For the per-bit energy consumption model, we adopt the measurements and network topology from Jalali et al . [28] , Vishwanath et al . [59] and consider two distinct scenarios for our communication cost analysis 𝑃𝑡 : (I) a wired 1 Gbit/s interconnect as it can be found in, e.g., factories, and (II) a 92 MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Woisetschläger et al. Table 2. FL strategy hyperparameters. Key: 𝜂 , 𝜂𝑙 = serverside learning rates, 𝛽1 , 𝛽2 = Weight decay rates, 𝜏 = Serverside adaptivity level, 𝑞 = Fairness parameter for qFedAvg, 𝜇 = proximal parameter for FedProx. FL Strategy Dataset 𝜂 𝜂𝑙𝛽1𝛽2𝜏 𝑞 𝜇 Further Details FedAvg All - - - - - - - FedAvg does not have any server hyperparameters FedAdam All −1.5−1 0.9 0.99 1𝑒−2- - FedAdaGrad All 0 0 - - 1𝑒−3- - FedYogi All −1.5−1.5 0.9 0.99 1𝑒−5- - FedProx Shakespeare - - - - - - 1𝑒−2 Others - - - - - - 1 qFedAvg Shakespeare - - - - - 1𝑒−2Others - - - - - 1wireless setup where clients connect via an LTE gateway. For (I), we use a network topology with 𝑛as = 2, 𝑛lc = 0, 𝑛lb = 0, 𝑛bng = 1, 𝑛e= 3, 𝑛c= 4, 𝑛d= 2. For (II), we set 𝑛as =0,𝑛lc =1,𝑛lb =1,𝑛bng =1,𝑛e=4,𝑛c=4,𝑛d=2. Privacy. The server is set up to handle both workloads that require very light privacy guarantees, i.e., for workloads that deal with non-sensitive data and for workloads ( 𝜖> 5) that require very tight guarantees (𝜖<1) [2, 42]. Client behavior. The client dropout module is designed to account for a wide variety of use cases that create different wear and tear on embedded devices and reduce their reliability. Typically, in industrial environments, service quality is essential. Thus, the availability times often range well above 95% of the operation time [ 3 ]. With our study, we not only account for these services but also look into the effects of clients with very low reliability as they are found in loose collaborative learning tasks [ 25 ]. To do so, we vary the client dropout likelihood from 0% to 50%. Model aggregation & validation. The server runs stateof-the-art FL strategies and is configured according to related work. The detailed parameterization is available in Table 2. DL workload coordination. The server is set up to coordinate state-of-the-art FL workloads that you would find in systems with both high and low reliability to provide a holistic picture of use cases. 4 Experimental Setup Our hardware-centric benchmark is deployed to a dedicated edge computing testbed. We use it to explore FL workloads in resource-constrained environments and explore the effects of power and network limitations on FL performance. For this benchmark, we deploy widely used state-of-the-art datasets from the NILM, CV, and NLP domains for FL applications, including an LLM workload. 4.1 Testbed We aim to explore FL applications in real systems by introducing two data center and three different embedded device types. Table 3. Our hyperparameters are chosen based on related work and hyperparameter sweeps for the best possible performance. Dataset Model Optim. LR W. Decay Dropout Hidden Dim Params Minibatch BLOND CNN SGD 0.055 0.0 14,000 128 LSTM SGD 0.045 0.001 0 15 40,000 128 ResNet SGD 0.052 0.001 100,000 128 DenseNet SGD 0.075 0.001 252,000 128 FEMNIST CNN Adam 0.001 0.0 33,000 32 Shakespeare LSTM SGD 0.8 0.0 0 256 819,000 32 Samsum FLAN-T5-Small AdamW 0.0001 0.0 0 N/A 80,000,000 1 - 128 Data center GPU (GPU). We use a GPU-accelerated data center node with 64 CPU cores, 256 GB of memory, and an NVIDIA A6000 (GPU) as our baseline device. The NVIDIA A6000 has a memory bandwidth of 768 GB/s. The VM has 3 TB NVMe storage. The VM is running Ubuntu 20.04 LTS with Python 3.9 and PyTorch 1.10. We use CUDA 11.6 and cuDNN 8.6. x86-CPU-base Clients (VM). As a proxy for x86-based embedded devices without DL acceleration, we use virtualized systems with 4 CPU cores and 4 GB of memory (VM). The VM has a memory bandwidth of 25.6 GB/s and a network interconnect of 1 GBit/s. We use Ubuntu 20.04 LTS with Python 3.9 and PyTorch 1.10. We use the estimates from SelfWatts [ 19 ] as the power utilization profile for each VM. Fieni et al. calculate approx. 50 Watts for a 4 CPU core VM on Intel Xeon E5 processors. NVIDIA Jetson AGX Orin 64GB (Orin). Our Orins are running Jetpack 5.1. This includes Ubuntu 20.04 LTS with Python 3.8, PyTorch 1.13, and CUDA 11.8. As the libraries are compiled specifically for the platform, we cannot adjust the stack to older PyTorch versions. The Orins have a memory bandwidth of 204 GB/s, a disk size of 64 GB, and a 10 Gbit/s network connection. They come with 2048 CUDA cores and 64 Tensor Cores. We measure power via the internal hardware-based monitoring functionality. NVIDIA Jetson Nano 2GB (Nano). The devices run Ubuntu 18.04 LTS with Python 3.6 and PyTorch 1.10. 2 The Nano has a memory bandwidth of 25.6 GB/s. The latest supported CUDA version is 10.2 with cuDNN 7.2. The Nanos carry 128 CUDA cores but no Tensor cores. The maximum power draw of a Nano is 15 Watts. Raspberry Pi 4B 4GB (RPi). They run with Ubuntu 20.04 LTS, Python 3.9, and PyTorch 1.10. There is no hardware acceleration available. The RPi has a memory bandwidth of 25.6 GB/s. The RPis have a class 10 32 GB SD card each and a 1GBit/s network interface. The RPis have a peak power draw of 10 Watts. 4.2 Software Stack FLEdge is implemented on top of widely used FL libraries. We use Flower [ 5 ] to run the FL routine and PyTorch Lightning 2 Jetson Nano with Maxwell architecture [ 47 ] only supports CUDA 10.2 and PyTorch 1.10 with Python 3.6. 93 FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Table 4. Compounding effects of client dropouts and differential privacy for FedAvg across varying 𝑧levels and client dropout rates. BLOND FEMNIST Shakespeare CNN DenseNet LSTM ResNet CNN LSTM 𝑧 𝑝 𝜖 Acc. 𝜖Acc. 𝜖Acc. 𝜖Acc. 𝜖Acc. 𝜖Acc. Loc. baseline N/A 0.96 N/A 0.89 N/A 0.95 N/A 0.91 N/A 0.75 N/A 0.59 0 0% ∞0.75 ∞0.74 ∞0.77 ∞0.73 ∞0.70 ∞0.53 10% ∞0.70 ∞0.73 ∞0.69 ∞0.73 ∞0.69 ∞0.52 20% ∞0.32 ∞0.69 ∞0.74 ∞0.72 ∞0.68 ∞0.51 50% ∞0.46 ∞0.67 ∞0.66 ∞0.70 ∞0.66 ∞0.49 0.3 0% 8.0 0.73 8.0 0.70 8.0 0.74 8.0 0.63 6.1 0.67 6.6 0.53 10% 8.1 0.73 8.1 0.70 8.1 0.70 8.1 0.63 6.6 0.66 6.6 0.52 20% 8.1 0.40 8.1 0.69 8.1 0.70 8.1 0.62 6.6 0.65 6.6 0.49 50% 8.1 0.44 8.1 0.67 8.1 0.64 8.1 0.60 6.7 0.61 6.6 0.44 0.5 0% 2.3 0.54 2.3 0.64 2.3 0.74 2.3 0.61 2.1 0.62 2.0 0.52 10% 2.4 0.54 2.4 0.63 2.4 0.74 2.4 0.61 2.1 0.61 2.0 0.49 20% 2.4 0.38 2.4 0.60 2.4 0.72 2.4 0.60 2.1 0.61 2.0 0.45 50% 2.4 0.36 2.4 0.57 2.4 0.70 2.4 0.60 2.1 0.59 2.0 0.44 1 0% 0.4 0.51 0.4 0.56 0.4 0.64 0.4 0.60 0.4 0.44 0.4 0.52 10% 0.4 0.51 0.4 0.56 0.4 0.61 0.4 0.60 0.4 0.43 0.4 0.44 20% 0.4 0.36 0.4 0.56 0.4 0.60 0.4 0.59 0.4 0.44 0.4 0.44 50% 0.4 0.34 0.4 0.54 0.4 0.57 0.4 0.57 0.4 0.39 0.5 0.39 1.3 0% 0.2 0.48 0.2 0.56 0.2 0.69 0.2 0.42 0.3 0.34 0.3 0.51 10% 0.2 0.45 0.2 0.56 0.2 0.64 0.2 0.42 0.3 0.34 0.3 0.44 20% 0.2 0.38 0.2 0.56 0.2 0.63 0.2 0.42 0.3 0.32 0.3 0.44 50% 0.2 0.34 0.2 0.51 0.2 0.61 0.2 0.39 0.3 0.28 0.3 0.39 1.5 0% 0.2 0.34 0.2 0.30 0.2 0.37 0.2 0.16 0.3 0.28 0.3 0.48 10% 0.2 0.34 0.2 0.30 0.2 0.37 0.2 0.15 0.3 0.28 0.3 0.44 20% 0.2 0.37 0.2 0.29 0.2 0.35 0.2 0.15 0.3 0.20 0.3 0.43 50% 0.2 0.19 0.2 0.28 0.2 0.31 0.2 0.15 0.3 0.17 0.3 0.36 [ 17 ] to allow for easy extensibility and seamless integration of new FL workloads. 4.3 FL Workloads We use FL workloads that have been explored in previous benchmarking works. Datasets. For NILM, we use the BLOND dataset [ 33 ]. It is an office environment appliance load monitoring dataset. It contains 13,164 samples with 12 appliance classes and captures electrical appliances in building-level office environments, such as laptops, monitors, and printers. We adopt Schwermer et al.’s [ 52 ] approach to FL with BLOND. Each sample in the dataset consists of 25,600 power readings (current, voltage). Instead of generating the input features for the DL models in the data loader, we deviate from Schwermer et al. and create the Active Power, Apparent Power, Reactive Power, and MFCC ( n_mfcc = 64 ) input features offline [ 4 ]. This results in a reduced sample size of 68 x 1. The main reason for reducing the dataset size is to fit the dataset onto our RPi devices. We employ the FEMNIST dataset [ 6 ] for CV. FEMNIST consists of 814,000 samples of hand-written digits and letters. The dataset contains 62 classes of handwritten digits (26 lower case letters, 26 upper case letters, and 10 digits). We randomly crop the grayscale samples to 28x28 and perform a random flip before training. Character recognition is frequently used by mobile clients to convert images to editable text documents. We use the Shakespeare dataset [ 23 ] for NLP. It consists of the complete works of William Shakespeare. It is preprocessed in the exact same way as introduced in the LEAF benchmark [ 6 ]. The dataset was preprocessed with a sliding window of 80 characters and a stride of 1 to prepare it for the next-character prediction task. The vocabulary was generated over the alphabet, including special characters, resulting in a total size of 80. Overall, the nature of the Shakespeare dataset resembles a task like a next-word prediction on smartphone keyboards. We complement experiments in the NLP domain with the SAMSum dataset, which contains 16,000 pairs of chatmessage-like dialogue and summary pairs that may be used for sequence-to-sequence modeling tasks [ 21 ]. With SAMSum, we introduce a realistic use case that can be used on mobile clients to provide a quick overview of their chat history, as it is frequently found in applications like Slack. Apart from splitting the dataset into 10 Dirichlet subsets ( 𝛼= 1), we do not apply additional preprocessing. Researching FL applications on physical devices usually requires fitting the data distribution to the number of devices on our testbed. We opt to sample client subsets to 45 based on a Dirichlet distribution ( 𝛼= 1) to fit the number of same-type clients in our testbed (Figure 2). Models. We train a total of 7 DL models. We use four different architectures to train on the BLOND datasets to showcase the sensitivity of different model sizes and architectures to real-world environmental conditions: a CNN, an LSTM, a ResNet, and a DenseNet architecture [ 52 ]. To train on the FEMNIST dataset, we train a small and efficient CNN architecture, originally presented in the LEAF benchmark [ 6 ]. For the Shakespeare dataset, we use an LSTM model with 256 hidden dimensions and zero dropout that has been well explored to solve the next character prediction task [ 23 ]. Additionally, we use the SAMSum dataset to evaluate the hardware performance of the NVIDIA Jetson AGX Orin devices. To do so, we employ FLAN-T5-Small, an 80M parameter state-of-the-art transformer model [ 12 ]. When finetuned for a specific downstream task, the FLAN-T5 model family has proven to deliver on-par performance with significantly larger foundation models such as LLama2 [ 20 ]. 94 MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Woisetschläger et al. Table 5. Global model validation accuracy across FL strategies with varying client dropout rates after 100 FL training rounds with a client selection rate of 20% per training round. Bold highlights the best-performing FL strategy. BLOND FEMNIST Shakespeare 𝑝Strategy CNN DenseNet LSTM ResNet CNN LSTM Loc. Baseline (𝑝=0%) 0.96±0.02 0.89±0.01 0.95±0.01 0.91±0.01 0.75±0.02 0.59±0.01 0% FedAdaGrad 0.76±0.0 0.17±0.07 0.71±0.03 0.70±0.0 0.56±0.01 0.52±0.01 FedAdam 0.73±0.01 0.03±0.0 0.64±0.05 0.64±0.02 0.38±0.05 0.5±0.03 FedAvg 0.75±0.02 0.74±0.0 0.77±0.01 0.73±0.01 0.7±0.0 0.53±0.0 FedProx 0.75±0.01 0.74±0.0 0.74±0.01 0.73±0.01 0.7±0.0 0.52±0.0 FedYogi 0.8±0.01 0.79±0.0 0.81±0.01 0.78±0.0 0.53±0.0 0.25±0.0 qFedAvg 0.73±0.0 0.7±0.0 0.79±0.0 0.71±0.0 0.03±0.0 0.47±0.0 10% FedAdaGrad 0.76±0.03 0.03±0.0 0.69±0.02 0.68±0.02 0.55±0.04 0.51±0.02 FedAdam 0.73±0.01 0.03±0.0 0.54±0.1 0.41±0.22 0.35±0.01 0.48±0.01 FedAvg 0.7±0.04 0.73±0.01 0.69±0.04 0.72±0.0 0.69±0.01 0.52±0.01 FedProx 0.74±0.05 0.73±0.01 0.68±0.03 0.73±0.01 0.68±0.01 0.52±0.01 FedYogi 0.76±0.02 0.78±0.01 0.8±0.0 0.76±0.03 0.53±0.01 0.23±0.01 qFedAvg 0.73±0.0 0.69±0.0 0.77±0.0 0.71±0.0 0.03±0.0 0.47±0.0 20% FedAdaGrad 0.64±0.1 0.04±0.0 0.48±0.3 0.69±0.03 0.49±0.06 0.49±0.02 FedAdam 0.73±0.0 0.03±0.0 0.53±0.13 0.63±0.0 0.34±0.05 0.35±0.06 FedAvg 0.32±0.27 0.69±0.05 0.74±0.03 0.72±0.01 0.68±0.02 0.51±0.02 FedProx 0.37±0.3 0.69±0.05 0.68±0.04 0.72±0.01 0.67±0.01 0.51±0.02 FedYogi 0.76±0.02 0.76±0.03 0.76±0.0 0.75±0.03 0.47±0.08 0.21±0.03 qFedAvg 0.72±0.01 0.68±0.0 0.75±0.0 0.71±0.0 0.03±0.0 0.47±0.0 50% FedAdaGrad 0.42±0.24 0.03±0.0 0.35±0.27 0.68±0.04 0.45±0.06 0.45±0.03 FedAdam 0.25±0.25 0.03±0.0 0.18±0.13 0.62±0.01 0.2±0.09 0.35±0.02 FedAvg 0.46±0.16 0.67±0.06 0.66±0.01 0.7±0.02 0.66±0.03 0.49±0.03 FedProx 0.52±0.16 0.67±0.06 0.68±0.04 0.7±0.02 0.64±0.02 0.49±0.03 FedYogi 0.72±0.01 0.71±0.09 0.74±0.04 0.74±0.04 0.43±0.12 0.2±0.04 qFedAvg 0.68±0.01 0.67±0.0 0.72±0.01 0.71±0.0 0.03±0.0 0.47±0.0 For the models used on BLOND, FEMNIST, and Shakespeare, we train one local epoch on clients and then send updates for aggregation to the server to eliminate potential risks of client drift [ 41 ]. The exact model configuration and hyperparameters are available from Table 3. FL strategies. We explore all models in conjunction with six stateof-the-art FL strategies. We include FedAvg, the first communication efficient FL strategy, aggregating client updates over an unweighted average [ 41 ]. We also include adaptive strategies introduced by Reddi et al . [49] , namely Fed-Adam, FedYogi, and Fed-AdaGrad. The adaptive strategies introduce a server-side learning rate to better account for data heterogeneity. We further adopt two strategies that aim to increase fairness and robustness in an FL system. FedProx [ 37 ] introduces a method for weighting client updates in the aggregation process based on the amount of data a client has processed for an update. q-fair FedAvg (qFedAvg) [ 38 ] is a derivative of FedAvg introducing a factor 𝑞 that determines the level of fairness. Fairness in this context is defined by how well a model generalizes across clients. Higher generalizability is achieved by overweighting those clients that have the highest loss. This is done to gear a model stronger towards the high-loss clients and reduce the accuracy variance across clients. We fix the number of FL rounds to 10 for all FL strategies. Further details on the datasets and DL models are available in Table 2. Network. We modify our testbed communication to realistically reflect real-world scenarios with ERRANT [ 56 ]. We use a 1 GBit/s synchronous network link as well as the global average 4G LTE connection characteristics of 40 MBit/s download and 15 MBit/s upload [56]. 5 Results To show the practical utility of FLEdge, we run extensive experiments evaluating FL workloads on client behavior, differential privacy, energy efficiency, and hardware diversity. 5.1 Client behavior Existing FL strategies work well with unreliable clients. Unreliability is a core challenge for edge computing systems. Especially for FL workloads, this bears the potential for significant information loss whenever model updates are not transmitted to the server, regardless of the root cause. As embedded devices do not have power redundancy, run in sub-optimal environments w.r.t. heat dispersion, and may suffer from outside damages, they are considered unreliable by nature. Our experiments show that existing state-of-theart FL strategies work well in systems with unreliable clients, i.e., high dropout rates. However, for each dataset, we see that one FL strategy always consistently performs best. This suggests that the FL strategy choice overall depends on the dataset, not the model. Yet, we also identify FedAdaGrad and FedAdam as particularly sensitive to client dropout. Thus, careful strategy selection and federated hyperparameter optimization are critical for systems that involve unreliable clients. 5.2 Differential Privacy Heterogeneous client behavior has significant negative effects on the model quality of differentially private workloads. Since DP is an integral component of FL workloads, the question arises: how does it change the model quality with varying client reliability levels? Interestingly, the model quality decreases significantly when introducing client dropout in a system, and the negative effects across 95 FLEdge: Benchmarking Federated Learning Applications in Edge Computing Systems MIDDLEWARE ’24, December 2–6, 2024, Hong Kong, Hong Kong Table 6. Computation and communication cost overview for our seven FL workloads across varying network conditions and 100 FL communication rounds (20% selection rate, 9 training clients per round). Wireless communication drives communication costs by one order of magnitude. Communication Computation 𝐺 4G LTE 1 GBit/s fiber RPi 4 Nano Orin 4G LTE 1 Gbit/s fiber Dataset Model Communicable parameters Time Power (kWh) Time Power (kWh) Time Power (kWh) Time Power (kWh) Time Power (kWh) RPi 4 Nano Orin RPi 4 Nano Orin FEMNIST CNN 0.1 MB 20s 0.0006 0.8s 0.0001 56s 0.0451 31s 0.0352 11s 0.0059 791 438 579 35000 19375 25625 BLOND CNN 0.1 MB 20s 0.0003 0.8s 0.0001 65s 0.0530 23s 0.0269 11s 0.0059 918 325 155 40625 14375 6875 BLOND LSTM 0.2 MB 20s 0.0007 0.8s 0.0001 80s 0.0652 21s 0.0243 13s 0.0074 565 148 92 25000 6563 4063 BLOND ResNet 0.4 MB 30s 0.0016 1.2s 0.0002 588s 0.5197 36s 0.0417 15s 0.0086 2076 127 53 91875 5625 2344 BLOND DenseNet 1.0 MB 70s 0.0041 2.8s 0.0006 586s 0.5125 36s 0.0416 16s 0.0086 828 51 23 36625 2250 1000 Shakespeare LSTM 3.2 MB 240s 0.0134 9.6s 0.0021 981s 0.9172 59s 0.0724 19s 0.0103 433 26 8 19160 1152 371 SAMSum FLAN-T5 Small 308 MB 21800s 1.3124 872s 0.2008 OOM OOM OOM OOM 324s 2.1492 OOM OOM 1.5 OOM OOM 65.7 strategies become significant at a low DP level. Since we use a server-side user-level DP algorithm, we adjust the noise level based on the model updates received per training round. This provides appropriate privacy guarantees. Yet, the level of noise we have to add to cover for failing clients reduces the model quality significantly (Table 4). For instance, if we want to provide a loose privacy budget of 𝜖= 8on the BLOND dataset and suffer from a high client dropout rate ( 𝑝= 0 . 5), the accuracy almost halves compared to a system without client dropouts ( 𝑝= 0). As such, allowing re-training with another client in a training round or accounting for late arrivals could mitigate the effects DP has on the model quality. Also, the evaluation of DP in unreliable systems is a strong indicator of whether deploying an FL workload into an edge computing system w.r.t. to the estimated model quality in relation to the privacy requirements is worthwhile. However, training quality and efficiency are equally relevant for an effective deployment. 5.3 Communication Efficiency Granularity helps quantify the practical utility in relation to the network when deploying FL workloads in edge computing systems. Before discussing the hardware characteristics of diverse embedded platforms, the question is whether it is even worthwhile deploying an FL workload to them under given network conditions. With larger models, FL workloads become more communication intensive as we have to send more model weights. From a practical perspective, quantifying whether it’s worthwhile to consider including embedded devices in a system that runs an FL workload is essential. For all of our stateof-the-art FL workloads, 𝐺 is significantly above 1 (Table 6). This indicates the high suitability of such small models to be trained as an FL workload, regardless of the communication technology. This is particularly useful for highly specialized tasks like object detection. Yet, for more generalizing tasks like in the NLP space, we require larger models to run on embedded devices and be trained at the edge. For larger models like FLAN-T5 Small, we see 𝐺= 1 . 5. With 𝐺 close to 1, the practical utility is limited as communication takes approximately as much time as computation. Beyond the computation/communication trade-off, 𝐺 is also suitable for quantifying the net effect of communication optimization methods for an FL workload. Further, reliability is a major concern in edge computing systems with embedded clients. We want to spend as little time communicating as possible to get the model updates to the server and not risk failures during long communication times. However, computational performance and efficiency are key challenges as well. 5.4 Energy Efficiency Energy efficiency is a well-suited indicator for computational bottlenecks on embedded devices and does not require extensive client monitoring that could potentially infringe privacy. As we will see in Section 5.5, the measurements of local step times on clients unveil inefficiencies and scalability limits. Yet, micro-benchmarks are often difficult to facilitate when having a variety of devices in a system. While micro-benchmarks are useful for an in-depth exploration of bottleneck root causes on a client, they require significant efforts and interfere with the DL training process. So, how can we identify bottlenecks without interfering with the client’s training process? We find energy efficiency to be a well-suited estimator for the suitability of a device for a given task and also for the identification of bottlenecks (Figure 3). The main benefit of energy efficiency is that it measures the throughput per Watt of power consumption, two available metrics without interference with the training process, i.e., they are convenient to measure. The device comparison shows the significant advancements in hardware for FL workloads on the edge (Figure 3a). It is also an indicator that we should look more at models with multi-million parameter models for FL workloads. The Orins have proven to become more energy efficient with increasing parameter size and outperform all other embedded device platforms as well as data center resources for state-of-the-art FL workloads. We identify the computational capabilities of the Orins by running the 80M parameter FLAN-T5 model for the SAMSum text summarization task (Figure 3b). By scaling the batch size, we 96