LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe

LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe Philippe Buschmann [email protected] Siemens AG Garching, Bavaria, Germany Arne Broering arne.br[email protected] Siemens AG Garching, Bavaria, Germany Georg Carle [email protected] Technical University of Munich Garching, Bavaria, Germany Andreas Blenk [email protected] Siemens AG Garching, Bavaria, Germany Abstract Large Language Models (LLMs) have revolutionized Natural Language Processing and now find their ways into various deployments such as end customer appliances or industrial settings. Their deployment at the edge, however, presents unique challenges, particularly regarding network infrastructure and resource constraints. While existing research has focused on LLM distribution in cloud environments, there is a lack of studies addressing the specific requirements and characteristics of edge computing scenarios. Accordingly, there is a rise of distributed LLM frameworks that aim to optimize the deployment of LLMs in edge environments. Due to their work in progress nature, these frameworks lack comprehensive measurements in a real testbed w.r.t. networking. This paper presents a comprehensive analysis of distributed LLM frameworks in edge computing environments, focusing on their networking behavior and deployment requirements. Our measurement results reveal non-obvious behaviors from performance degradation by adding compute nodes to significant traffic pattern complexities. CCS Concepts •Networks → Network measurement;•Computing methodologies → Distributed algorithms;Distributed artificial intelligence; Natural language generation. Keywords Network Measurement; Distributed Large Language Models ACM Reference Format: Philippe Buschmann, Arne Broering, Georg Carle, and Andreas Blenk. 2025. LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe. In 2nd Workshop on Networks for AI Computing (NAIC ’25), September 8–11, 2025, Coimbra, Portugal. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3748273.3749213 This work is licensed under a Creative Commons Attribution-NonCommercialNoDerivatives 4.0 International License. NAIC ’25, Coimbra, Portugal ©2025 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2082-6/2025/09 https://doi.org/10.1145/3748273.3749213 1 Introduction Large Language Models (LLMs) have brought significant improvements to Natural Language Processing [ 36 ], leading to an increased adoption of LLM agents across various applications from personal assistants [ 1 ] at home to industrial agents [ 4 ] on the manufacturing floor. The deployment of generative AI presents unique challenges in industrial environments where operational efficiency and data security are paramount. These environments restrict cloud-based offloading due to privacy requirements [18, 29] and cost optimization [ 24 ]. Therefore, the deployment of LLMs on the edge is particularly important to enhance on-device intelligence [ 33 ]. While it is possible to deploy a small model on a constrained device, deploying large models is usually not possible due to the lack of resources. Distributing a single model over multiple devices enables the use of large models, which are generally known to be more powerful and accurate [ 10 ]. Several frameworks for distributed LLMs have been proposed and published in the literature, each offering distinct approaches of distributed inference. The main approaches of distributed inference is to load layers / tensors of the model onto multiple devices with pipeline / tensor parallelism. The deployment of LLMs in a distributed manner faces networking challenges. Data needs to be transmitted during distributed inference. This communication may interfere with concurrent network traffic and impact Quality of Service (QoS) constraints, especially when operating alongside latency-sensitive real-time applications. We currently see a small but increasing number of distribution frameworks. Accordingly, users are now faced to choose between multiple LLM distribution frameworks, which is inherently tied to the underlying network infrastructure. To the best of our knowledge, a study of the networking behavior and detailed requirements of LLM distribution frameworks is missing so far. Measuring and revealing the networking behavior of LLM distribution is complicated: (1) the prevalence of high-end GPU cluster evaluations that lack resource constraints of low-end, non-GPU devices [ 17 , 26 , 32 , 35 ]; (2) traditional deployment biased towards cloud-based LLM providers [ 6 , 11 , 19 , 36 ]; (3) limited coverage of edge computing environments where heterogeneous hardware and network variability predominate. In this paper, we investigate the network behavior of distributed LLM frameworks and make the following key contributions: We study eight LLM distribution frameworks, show their applicability towards the edge, and select two compatible frameworks for 108 NAIC ’25, September 8–11, 2025, Coimbra, Portugal Philippe Buschmann, Arne Broering, Georg Carle, and Andreas Blenk Pipeline Parallelism Generation Text Dataset Prompt X XC1 C2 Tensor Parallelism Dataset Prompt Generation Text12 rpi2 send result 3 Data Parallelism GenerationsDataset Prompt 1 Prompt 2 Prompt 3 Llama 3.2 1B Llama 3.2 1B Llama 3.2 1B Text 1 Text 2 Text 3 rpi1 rpi1 rpi3 rpi2 rpi3 rpi1 rpi2 rpi3 C2 = Y2 Y2 C1 = Y1 Y1 multiply with send result Figure 1: Comparison of data, pipeline and tensor parallelism exemplified on a Raspberry Pi cluster (rpi1, rpi2, rpi3). detailed analysis: distributed-llama and llama.cpp . We characterize the traffic structure, trace complexity, and inference scalability of both frameworks, and provide guidelines on deployment strategies, cluster configuration, and network infrastructure. 2 Distributed LLMs under the Microscope Distributed LLM inference primarily employs three parallelization strategies [ 25 ]: data parallelism, pipeline parallelism, and tensor parallelism. Figure 1 illustrates an exemplified deployment of all three parallel processing strategies in a cluster environment with three Raspberry Pis (rpi1, rpi2, rpi3). If the LLM fits on a single device (if the device has enough resources), we can use data parallelism to split the data into batches and distribute these batches to different devices running the same model, e.g., Llama 3.2 1B. Figure 1 depicts three prompts distributed to three devices (rpi1, rpi2, rpi3), which generate three outputs independent of each other. This can speed up the processing of large datasets [25]. When data parallelism is no longer applicable due to the entire model not fitting on a single device, pipeline parallelism serves as a viable alternative. Pipeline parallelism splits the model into stages and distributes these stages sequentially across multiple devices [ 20 ]. Figure 1 shows the distribution of 3 stages. Each stage contains at least one layer of the LLM and needs to communicate its results with other stages over the network. If a single layer does not fit on a single device, we can use tensor parallelism. Tensor parallelism distributes the underlying matrices / vectors (also named tensors) and their computations across devices [ 30 ]. In Figure 1, Raspberry Pi rpi1 distributes vector X to Raspberry Pi rpi2 and rpi3, which multiply X with Y1 and Y2, resulting in C1 and C2 correspondingly. The partial results C1 and C2 are sent back to rpi1 for combination. All three parallelism techniques are independent of each other and can be combined. We focus our analysis on pipeline and tensor parallelism, since data parallelism is not applicable to the deployment of large models on constrained devices. Table 1 lists a selection of the opensource distributed LLM frameworks and depicts their pipeline parallelism (PP) and tensor parallelism (TP) support. Petals , LLM-PQ , HexGen , vLLM , exo , prima.cpp support pipeline parallelism, while tensor parallelism is only supported by HexGen , vLLM , and distributed-llama . llama.cpp supports tensor parallelism on one device over multiple GPUs (not multiple devices) and implements its own sequential model distribution over multiple devices, which is similar to pipeline parallelism. Framework CPU GPU PP TP C Petals [16] ✗ ✓ ✓ ✗ ✗ LLM-PQ [35] ✗ ✓ ✓ ✗ ✗ HexGen [21] ✗ ✓ ✓ ✓ ✗ vLLM [13] ✓1✓ ✓ ✓ ✗ exo [3] ✓ ✓ ✓ ✗ ✗ prima.cpp [23] ✓ ✓ ✓ ✗ ✓ llama.cpp [5] ✓ ✓ ? ? ✓ distributed-llama [31] ✓ ✓2✗ ✓ ✓ 1Not available as pre-built 2Vulkan (experimental) Table 1: Selection of distributed LLM frameworks with support for CPU / GPU inference, pipeline parallelism (PP), tensor parallelism (TP), and deployment compatibility (C). Additionally, Table 1 outlines the compatibility (C) with our measurement environment (described in Section 3). Petals , LLM-PQ , HexGen , and vLLM require dedicated GPUs, which are unavailable on Raspberry Pis. However, vLLM provides documentation for building vLLM with CPU-based inference but does not support the architecture of Raspberry Pis. exo supports combined GPU and CPU distributed inference but lacks support for Raspberry Pis [ 2 ] and the Jetson AGX Orin [ 12 ]. prima.cpp , a fork of an older llama.cpp version, supports GPU and CPU inference but crashes during token generation in our environment. vLLM , exo , prima.cpp , llama.cpp , and distributed-llama are still work-in-progress with varying degrees of project maturity. While some features may not work reliably, those who do, give insights into the inner workings of the frameworks. Since the work on these frameworks continues, future updates may resolve missing support and current issues. We select llama.cpp as an example of pipeline parallelism and distributed-llama as an example of tensor parallelism for our measurement environment. llama.cpp operates sequentially and distributed-llama parallelizes its computations. As our measurements later reveal, both distributed LLM frameworks work in two phases: the initial distribution of the model with a warmup prompt and prompt handling itself. 3 Measurement Environment Target of the measurement environment is to evaluate LLM distribution frameworks in an edge environment with heterogeneous devices. Figure 2 showcases our measurement testbed: the devices in use, their hostnames, and their connectivity. We use: 6x Raspberry Pi 4Bs (rpi1, rpi2, rpi3, rpi4, rpi5, rpi6), 1x Jetson AGX Orin (jetson), 1x PC with a RTX 2080 Ti (aorus), and 1x HPE 1820-24G Switch J9980A. Table 2 outlines the available resources of the cluster. The Raspberry Pi 4B is the least powerful device in our setup based on CPU performance and RAM resources. It is limited to 4GB of RAM, while aorus and jetson provide 32GB and 64GB respectively, and it has no inference support for its internal GPU. In contrast, aorus has a dedicated GPU with 11GB of VRAM and jetson shares its 64GB LPDDR5 RAM with its internal GPU. aorus is the most powerful device in terms of CPU due to its 32 Core AMD Ryzen Threadripper 109 LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe NAIC ’25, September 8–11, 2025, Coimbra, Portugal Raspberry Pi 4B PC (aorus) Jetson AGX Orin (jetson) Processor Broadcom BCM2711 Quad core Cortex-A72 (ARM v8) 64-bit SoC AMD Ryzen Threadripper 2990WX 32-Core Processor 12-core Arm® Cortex®-A78AE v8.2 64-bit CPU RAM 4 GB (LPDDR4-3200) 32 GB (DDR4-3200) 64GB (LPDDR5) GPU in SoC - no inference support NVIDIA GeForce RTX 2080 Ti 2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores GPU VRAM – 11 GB (GDDR6) Shared 64GB (LPDDR5) Table 2: Hardware specifications of devices in measurement environment. 2990WX CPU. jetson is the device with the largest amount of RAM resources available. The cluster has a theoretical maximum of 120 GB of RAM and 11GB of VRAM as memory for a distributed LLM. Worker WorkerWorkerWorkerWorker Worker Head rpi2 rpi6 jetsonrpi3 rpi4 tcpdump aorus tcpdumptcpdump tcpdump tcpdump tcpdump Switch Worker rpi5 tcpdump rpi1 tcpdump Benchmark Figure 2: Measurement environment with heterogeneous devices. All devices connect via a 1 Gbit/s link in a star topology, even though this topology might not be optimal for all frameworks. For instance, prima.cpp communicates in a ring pattern and might benefit from a ring topology, but the authors still measure their performance in a wireless scenario [ 23 ]. A star topology is an easy network topology to start with and still reveals the overlay connectivity of the frameworks. Figure 2 outlines the head (rpi1) and worker devices. The rpi1 splits and distributes the LLM to worker devices centrally. Due to the central distribution, rpi1 has to store a copy of the LLM locally. Furthermore, rpi1 acts as entrypoint for prompts via an OpenAIcompatible API, which we use to benchmark the LLM distribution frameworks. We store and start the benchmark locally on rpi1 to avoid additional network traffic. We capture all outgoing network traffic on each device in pcap files with tcpdump for the model distribution and prompt processing phase separately. For prompt handling after the initial distribution, we select one multiple choice question from Ollama-MMLU-Pro [ 22 ] to benchmark the distributed LLM. Since the correctness / accuracy of the response only correlates with the selected type, version, and parameter number of the LLM and not the distribution approach, we disregard the accuracy results of the benchmark. Instead, we focus on the network traffic patterns, traffic volume, trace complexity [ 15 ], inference throughput, and inference time, which are influenced by the distribution method. For calculating the trace complexity, we follow the work of Avin et al. [ 15 ] to break the traffic structure down into temporal and non-temporal complexity (see Section A.1), since it can facilitate a systematic optimization of network protocol implementations [ 15 ]. We distribute the inference on 1 (head device only - no distribution), 2, 4, and 8 devices as distributed-llama operates exclusively with 2 𝑛 devices. The measurements with 1, 2, and 4 devices only include Raspberry Pis ( 1-r , 2-rr , 4-rrrr , with rpi1 as head); measurements with 8 devices consist of all devices including jetson and aorus ( 8-rrrrrrja - 6x rpi, aorus, jetson - with rpi1 as head). In addition, we capture the behavior with aorus as a head for 2 devices ( 2-aj - aorus, jetson), and 4 devices ( 4-ajrr - aorus, jetson, rpi2, rpi3). The models of llama.cpp and distributed-llama have the same parameter size but may slightly differ in their quantization due to a custom conversion done by distributed-llama . We use Llama-3.2-1B-Instruct-Q4_K_M.gguf [ 8 ] (808 MB), Llama-3.1-8BInstruct-Q4_K_M.gguf [ 7 ] (4.92 GB), and Llama-3.3-70B-InstructQ4_K_M.gguf [ 9 ] (42.5 GB). Llama 3.2 1B fits on a single Raspberry Pi, Llama 3.1 8B needs at least two, and Llama 3.3 70B needs at least five Raspberry Pis. 4 Analysis of Distributed LLM Inference This sections presents our analysis of distributed LLMs. Our investigation focuses on three key aspects: scalability, traffic patterns, and traffic complexity of distributed inference. 4.1 Scalability Figure 3 visualizes the inference throughput in tokens per seconds and the inference duration in seconds for both distributed LLM frameworks with Llama 3.2 1B and Llama 3.1 8B based on the same prompt of the benchmark dataset. We observe a strong relation between inference throughput and inference duration regardless of the used model and framework, which confirms proper system functionality. For Llama 3.2 1B, Figure 3a depicts a significant performance difference between distributed-llama (2.7 tokens/s) and llama.cpp (8.24 tokens/s) in the non-distributed scenario 1-r . Adding additional devices does not improve the performance of llama.cpp 110 NAIC ’25, September 8–11, 2025, Coimbra, Portugal Philippe Buschmann, Arne Broering, Georg Carle, and Andreas Blenk 0 20 Throughput (tokens/s) 1-r 2 - rr 4 - rrrr 8 - rrrrrrja 2 - aj 4 - ajrr Number of Devices 2.70 6.53 10.54 14.33 19.19 12.88 8.24 7.95 7.60 16.81 28.46 20.29 0 500 Time (seconds) 10:28 4:14 2:34 1:51 1:26 2:10 3:20 3:29 3:38 1:33 57.00 1:21 0.0 2.5 Throughput (tokens/s) Fails Fails 2.45 3.96 4.37 2.56 Fails 1.13 1.12 2.23 4.29 3.29 0 1000 Time (seconds) Fails Fails 11:33 7:05 6:27 11:08 Fails 25:25 25:33 12:46 6:36 8:42 (a) Llama 3.2 1B (b) Llama 3.1 8B distributed-llama llama.cpp Figure 3: Inference performance based on frameworks, models, and devices. given identical hardware specifications ( 1-r , 2-rr , 4-rrrr ). Conversely, the addition of low-performance devices reduces inference throughput and increases the inference time comparing scenario 2-aj , 4-ajrr , and 8-rrrrrrja . This effect is also visible for distributed-llama ( 2-aj and 4-ajrr ). distributed-llama benefits mostly from the increase in device count leading to a faster inference time ( 2-rr , 4-rrrr , and 8-rrrrrrja ). The ratio between number of devices and number of low compute devices impacts the inference throughput of distributed-llama with 19.19 tokens/s ( 2-aj ) compared to 12.88 tokens/s ( 4-ajrr ) and 14.33 tokens/s ( 8-rrrrrrja ). On 1, 2, and 8 devices, independent of the head device and for 4 device with aorus and jetson, llama.cpp outperforms distributed-llama . On 4 Raspberry Pis, distributed-llama outperforms llama.cpp . distributed-llama achieves performance parity with llama.cpp ’s baseline once the device count is high enough. Figure 3b reveals that Llama 3.1 8B exceeds the RAM of one Raspberry Pi and thus fails for both distributed-llama and llama.cpp . Similar failure emerges in scenario 2-rr for distributed-llama , because distributed-llama requires more resources than available with 2 Raspberry Pis. distributed-llama works properly in scenario 2-aj , since both devices provide more resources. llama.cpp behaves similar for Llama 3.1 8B and Llama 3.2 1B with no performance gain between 2 and 4 Raspberry Pis (1.13 vs 1.12 tokens/s) and the best performance with only aorus and jetson. llama.cpp demonstrates a need for high computational capabilities, leveraging these resources to overcome its sequential processing limitations. In homogeneous clusters, llama.cpp only benefits from the increase of resources and accuracy of larger models but not in terms of inference performance. In its current implementation, llama.cpp lacks optimized worker selection that could enhance performance. distributed-llama outperforms llama.cpp for 2, 4, and 8 devices with rpi1 as head and unexpectedly outperforms llama.cpp in scenario 2-aj for Llama 3.1 8B. distributed-llama seems to be more optimized than llama.cpp for Llama 3.1 8B. We also execute llama.cpp and distributed-llama in scenario 8-rrrrrrja with Llama 3.3 70B. While distributed-llama exceeds the RAM capabilities of Raspberry Pis due to equal model splitting (40 𝐺𝐵/ 8 = 5 𝐺𝐵 > 4 𝐺𝐵 ) and fails, llama.cpp finishes after around 2 hours and 21 minutes with an inference throughput of 0.2 tokens/s. We leave a deeper investigation for future work, however, need to raise doubts about this behavior and the current stabilities of the frameworks. 4.2 Traffic Patterns Figure 4a illustrates outgoing traffic of rpi1 during the model distribution for llama.cpp and distributed-llama . It shows the strictly sequential data transfer patterns of llama.cpp and distributed-llama with parallel traffic and no distinct pattern. Both llama.cpp and distributed-llama transmit at a rate of 100 to 150 Mbit/s during the model distribution of Llama 3.1 8B. Figure 4b depicts the outgoing traffic of rpi1 during prompt handling using llama.cpp and distributed-llama . It depicts the sparse communication over time due to the sequential computation efforts of llama.cpp , until the output generation near the end, at which all nodes communicate with rpi1 simultaneously. During prompt handling, llama.cpp has asymmetric traffic rates with incoming traffic at 3 Mbit/s and outgoing traffic varying between 3-7 Mbit/s. The output generation stage maintains stable traffic rates with constant incoming traffic of 1 Mbit/s and outgoing traffic of 7 Mbit/s. In addition, Figure 4b shows the unstructured, parallel traffic of distributed-llama with a prevailing communication to jetson. This behavior manifests consistently across all devices including traffic to rpi1. This is due to an increased offload of tensors to jetson. The incoming and outgoing traffic rate of all devices for distributed-llama is 75-80 Mbit/s. Figure 5 visualizes the normalized (between 0 - 714 MB) traffic volume of handling one prompt with Llama 3.1 8B over all eight 111 LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe NAIC ’25, September 8–11, 2025, Coimbra, Portugal Figure 4: Outgoing traffic patterns of rpi1 in scenario 8-rrrrrrja during model distribution and prompt handling. devices ( 8-rrrrrrja ). Figure 5a illustrates the star communication pattern of llama.cpp , in which all devices only communicate directly with the head device - here rpi1. It depicts a slightly higher traffic volume from rpi1 towards jetson (163 MB) in comparison to other hosts (115 - 122 MB) and the outlier of rpi6 (4 MB), which is last in the pipeline sequence. Furthermore, the communication between rpi1 and other devices is asymmetric with more outgoing than incoming traffic. In contrast, Figure 5b outlines an inverse pattern with higher incoming traffic towards rpi1 (head). distributed-llama uses a peer-to-peer communication strategy and has a more uniform traffic volume distribution based on the performance of devices. The Raspberry Pis have lows of 323 MB and peaks of 415 MB and aorus / jetson have lows of 579 MB and peaks of 714 MB. In the case of prompt handling, communication between devices using distributed-llama is more frequent and consumes higher bandwidth (323 MB - 714 MB) than llama.cpp (4 MB - 163 MB). This is due to the many-to-many communication pattern of distributed-llama , which differs from the star topology communication pattern of llama.cpp. In networks with limited bandwidth capacity, distributed-llama ’s throughput demands can lead to suboptimal performance or create resource conflicts with established network services, affecting their operational stability. Despite llama.cpp ’s lower traffic volume, the centralized topology through a single head device creates a single point of failure. 4.3 Traffic Structure We analyze the trace complexity of model distribution and prompt handling in scenario 8-rrrrrrja . Figure 6 depicts the complexity of the traffic traces during model distribution and prompt handling. Four different types of traffic complexity exist: (1) Traffic with a high time dependency / temporal structure has low temporal complexity (bursty). (2) Traffic with more frequent communication between some devices only has low non-temporal complexity (skewed). (3) Traffic that is time dependent and showcases a communication pattern between devices has a low temporal and non-temporal complexity (skewed and bursty). (4) Uniform, high-entropy traffic has a high temporal and high non-temporal complexity. Figures 6 shows llama.cpp as consistently more bursty and skewed than distributed-llama , regardless of models and phase. Due to the star topology communication pattern which is independent of the temporal order of packets, llama.cpp demonstrates a low non-temporal complexity / skewed traffic. Furthermore, the sequential nature of packet transmission in llama.cpp and its temporal structure signifies a low temporal complexity / bursty traffic. In contrast, distributed-llama ’s traffic is more uniform. Since all devices communicate simultaneously instead of sequentially, distributed-llama has less temporal structure / higher temporal complexity than llama.cpp . Due to its more uniform peer-to-peer communication involving all devices, distributed-llama has less non-temporal structure / higher non-temporal complexity. Model size impacts traffic patterns differently across frameworks: distributed-llama exhibits more skewed / bursty traffic for larger models during model distribution due to the increased outgoing traffic from the head (rpi1), but displays almost no difference during prompt handling as traffic volumes are fundamentally limited by unchanged hardware specifications. By comparison, llama.cpp demonstrates lower skewness when processing larger models, both in model distribution and prompt handling, since leveraging additional resources across all devices leads to more balanced traffic patterns. In general, the traffic of llama.cpp is more structured in comparison to distributed-llama , especially during prompt handling. Therefore, llama.cpp’s traffic is more predictable. 5 Related Work Most closely aligned with our research, Zerwas et al. [ 34 ] investigate network traffic patterns in distributed machine learning frameworks. Their findings on communication patterns and network resource utilization provide valuable context for our analysis of distributed LLM frameworks. In the domain of LLM inference, Abstreiter et al. [ 14 ] evaluate the performance of LLM inference on constrained / specialized edge devices, establishing a baseline for single device inference. Our research extends beyond this to examine distributed inference architectures. Through comprehensive analysis of data, pipeline, and tensor parallelism strategies, Repaka et al. [ 27 ] evaluate distributed LLM training performance to determine optimal parallelization approaches. Our work complements this by providing recommendations specifically for inference scenarios. Roy et al. [ 28 ] provided valuable insights into flow characteristics, traffic demand and traffic patterns within Facebook’s data center infrastructure. While their scale differs from our edge 112 NAIC ’25, September 8–11, 2025, Coimbra, Portugal Philippe Buschmann, Arne Broering, Georg Carle, and Andreas Blenk aorus jetson rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 Destination aorus jetson rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 Source 0 0 0.04 0 0 0 0 0 0 0 0.04 0 0 0 0 0 0.17 0.23 0 0.17 0.16 0.16 0.16 0.01 0 0 0.05 0 0 0 0 0 0 0 0.05 0 0 0 0 0 0 0 0.05 0 0 0 0 0 0 0 0.01 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0.0 0.2 0.4 0.6 0.8 1.0 Traffic Volume (a) llama.cpp with star topology communication pattern and asymmetric (higher outgoing, lower incoming) traffic. aorus jetson rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 Destination aorus jetson rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 Source 00.82 1 0.81 0.82 0.81 0.81 0.81 0.81 01 0.81 0.81 0.81 0.81 0.81 0.56 0.58 00.51 0.56 0.56 0.51 0.57 0.52 0.49 0.68 00.49 0.49 0.52 0.48 0.52 0.49 0.68 0.46 0 0.47 0.45 0.51 0.47 0.57 0.68 0.46 0.5 0 0.46 0.47 0.52 0.54 0.72 0.55 0.52 0.53 00.52 0.5 0.49 0.75 0.5 0.56 0.48 0.49 0 0.0 0.2 0.4 0.6 0.8 1.0 Traffic Volume (b) distributed-llama with peer-to-peer communication and traffic volume linked to hardware performance. Figure 5: Normalized traffic volume of Llama 3.1 8B during prompt handling. 0.0 0.2 0.4 0.6 0.8 1.0 Temporal Complexity 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Non-temporal Complexity llama3.2-1b llama3.1-8b llama3.2-1b llama3.1-8b UniformBursty Skewed Skewed& Bursty llama.cpp distributed-llama (a) Model distribution 0.0 0.2 0.4 0.6 0.8 1.0 Temporal Complexity 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Non-temporal Complexity llama3.2-1b llama3.1-8b llama3.2-1b llama3.1-8b UniformBursty Skewed Skewed& Bursty llama.cpp distributed-llama (b) Prompt handling Figure 6: Trace complexity of frameworks and models of scenario 8-rrrrrrja. computing environment, their methodologies for analyzing traffic characteristics inspire our approach to understanding communication patterns in distributed LLM deployments. 6 Conclusion This paper presents a first measurement analysis of LLM distribution frameworks for small edge computing settings. The analysis reveals operational characteristics and deployment considerations for two frameworks. llama.cpp exhibits structured, predictable behavior through its sequential processing approach, making it optimal for environments prioritizing reliable resource scheduling. Its performance heavily depends on individual device capabilities, with slower devices degrading overall system performance. As a result, llama.cpp ’s current implementation benefits from the deployment on fewer but more powerful devices. distributed-llama demonstrates superior performance through parallel processing, particularly in homogeneous environments. Its performance improves with device count, though performance benefits may diminish in small clusters with weak computational nodes. distributed-llama ’s parallel architecture generates higher traffic volume overall in comparison to llama.cpp . These architectural decisions require careful evaluation during network infrastructure planning and framework selection. We believe that our work opens interesting avenues for future research, such as larger settings or the deployment of other parallel distribution frameworks. Acknowledgment This research work has received funding from the Horizon Europe Framework Programme under Grant agreement No 101135775 (PANDORA). References [1] [n. d.]. ChatGPT. Retrieved 2025-04-29 from https://openai.com/chatgpt/ overview/ 113 LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe NAIC ’25, September 8–11, 2025, Coimbra, Portugal [2] [n. d.]. Compatibility with Raspberry Pi ·Issue #111 ·Exo-Explore/Exo. Retrieved 2025-04-29 from https://github.com/exo-explore/exo/issues/111 [3] EXO 2025. Exo-Explore/Exo. EXO. Retrieved 2025-04-03 from https://github.com/ exo-explore/exo [4] siemens.com Global Website [n. d.]. Generative AI-powered Assistant: Your Siemens Industrial Copilot. siemens.com Global Website. Retrieved 2025-0520 from https://www.siemens.com/global/en/products/automation/topic-areas/ industrial-ai/industrial-copilot.html [5] ggml-org 2025. Ggml-Org/Llama.Cpp. ggml-org. Retrieved 2025-04-20 from https://github.com/ggml-org/llama.cpp [6] [n. d.]. GitHub - TIGER-AI-Lab/MMLU-Pro: The Code and Data for "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" [NeurIPS 2024]. Retrieved 2025-05-01 from https://github.com/TIGER-AI-Lab/ MMLU-Pro [7] 2025. Llama-3.1-8B-Instruct-Q4_K_M.Gguf ·Unsloth/Llama-3.1-8B-Instruct-GGUF at Main. Retrieved 2025-05-19 from https://huggingface.co/unsloth/Llama-3.18B-Instruct-GGUF/blob/main/Llama-3.1-8B-Instruct-Q4_K_M.gguf [8] 2025. Llama-3.2-1B-Instruct-Q4_K_M.Gguf ·Unsloth/Llama-3.2-1B-Instruct-GGUF at Main. Retrieved 2025-05-19 from https://huggingface.co/unsloth/Llama-3.21B-Instruct-GGUF/blob/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf [9] 2025. Llama-3.3-70B-Instruct-Q4_K_M.Gguf ·Unsloth/Llama-3.3-70B-InstructGGUF at Main. Retrieved 2025-05-22 from https://huggingface.co/unsloth/Llama3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_K_M.gguf [10] [n. d.]. MMLU-Pro Leaderboard - a Hugging Face Space by TIGER-Lab. Retrieved 2025-05-22 from https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro [11] OpenAI 2025. Openai/Simple-Evals. OpenAI. Retrieved 2025-05-21 from https: //github.com/openai/simple-evals [12] [n. d.]. Support for Jetson? ·Issue #285 ·Exo-Explore/Exo. Retrieved 2025-04-29 from https://github.com/exo-explore/exo/issues/285 [13] vLLM 2025. Vllm-Project/Vllm. vLLM. Retrieved 2025-04-23 from https://github. com/vllm-project/vllm [14] Maximilian Abstreiter, Sasu Tarkoma, and Roberto Morabito. 2025. Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge. doi:10.48550/arXiv.2503.09114 arXiv:2503.09114 [cs] [15] Chen Avin, Manya Ghobadi, Chen Griner, and Stefan Schmid. [n. d.]. On the Complexity of Traffic Traces and Implications. 4, 1 ([n. d.]). [16] Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2023. Petals: Collaborative Inference and Fine-tuning of Large Models. doi:10.48550/arXiv.2209. 01188 arXiv:2209.01188 [cs] [17] Fenglong Cai, Dong Yuan, Zhe Yang, and Lizhen Cui. 2024. Edge-LLM: A Collaborative Framework for Large Language Model Serving in Edge Computing. In 2024 IEEE International Conference on Web Services (ICWS) (Shenzhen, China, 2024-07-07). IEEE, 799–809. doi:10.1109/ICWS62655.2024.00099 [18] Baotong Chen, Jiafu Wan, Antonio Celesti, Di Li, Haider Abbas, and Qin Zhang. 2018. Edge Computing in IoT-Based Manufacturing. 56, 9 (2018), 103–109. doi:10.1109/MCOM.2018.1701231 [19] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. doi:10.48550/arXiv.2009.03300 arXiv:2009.03300 [cs] [20] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. In Advances in Neural Information Processing Systems (2019), Vol. 32. Curran Associates, Inc. Retrieved 2025-05-01 from https://proceedings.neurips.cc/paper_files/paper/2019/hash/ 093f65e080a295f8076b1c5722a46aa2-Abstract.html [21] Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. 2024. HexGen: Generative Inference of Large Language Model over Heterogeneous Environment. doi:10.48550/arXiv.2311.11514 arXiv:2311.11514 [cs] [22] Chi Kim. 2025. Chigkim/Ollama-MMLU-Pro. Retrieved 2025-05-01 from https: //github.com/chigkim/Ollama-MMLU-Pro [23] Zonghang Li, Tao Li, Wenjiao Feng, Mohsen Guizani, and Hongfang Yu. 2025. PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters. doi:10.48550/arXiv.2504.08791 arXiv:2504.08791 [cs] [24] Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. 2025. SkyServe: Serving AI Models across Regions and Clouds with Spot Instances. doi:10.1145/3689031. 3717459 arXiv:2411.01438 [cs] [25] Alex McKinney. [n. d.]. A Brief Overview of Parallelism Strategies in Deep Learning | Alex McKinney. Retrieved 2025-05-21 from https://afmck.in/posts/2023-02-26parallelism/ [26] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam Netherlands, 2025-03-30). ACM, 586–602. doi:10.1145/3669940.3707215 [27] Kishan Teja Repaka, Manikanta Anirudh Bondugula, and Sreekar Sashaank Adibhatla. [n. d.]. Benchmarking Distributed Machine Learning Systems with Large Language Models on Human vs. LLM Text Corpus. ([n. d.]). [28] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C. Snoeren. 2015. Inside the Social Network’s (Datacenter) Network. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (London United Kingdom, 2015-08-17). ACM, 123–137. doi:10.1145/2785956.2787472 [29] Sumudith Sadeepa, Keshara Kavinda, Emmanuel Hashika, Chamara Sandeepa, Tharindu Gamage, and Madhusanka Liyanage. 2024. DisLLM: Distributed LLMs for Privacy Assurance in Resource-Constrained Environments. In 2024 IEEE Conference on Communications and Network Security (CNS) (Taipei, Taiwan, 202409-30). IEEE, 1–9. doi:10.1109/CNS62487.2024.10735498 [30] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. doi:10.48550/arXiv.1909.08053 arXiv:1909.08053 [cs] [31] Bart Tadych. 2025. B4rtaz/Distributed-Llama. Retrieved 2025-04-03 from https: //github.com/b4rtaz/distributed-llama [32] Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, and Xiaowen Chu. 2024. FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression. doi:10.48550/arXiv. 2410.12707 arXiv:2410.12707 [cs] [33] Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge. In Proceedings of the Twentieth European Conference on Computer Systems (2025-03-30). 278–292. doi:10.1145/3689031.3696099 arXiv:2407.00088 [cs] [34] Johannes Zerwas, Kaan Aykurt, Stefan Schmid, and Andreas Blenk. 2021. Network Traffic Characteristics of Machine Learning Frameworks Under the Microscope. In 2021 17th International Conference on Network and Service Management (CNSM) (Izmir, Turkey, 2021-10-25). IEEE, 207–215. doi:10.23919/CNSM52442. 2021.9615524 [35] Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. LLMPQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization. doi:10.48550/arXiv.2403.01136 arXiv:2403.01136 [cs] [36] Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2025. A Review on Edge Large Language Models: Design, Execution, and Applications. 57, 8 (2025), 1–35. doi:10.1145/3719664 A Appendix A.1 Trace Complexity Calculation To calculate the trace complexity based on the equations of Avin et al. [ 15 ], we extract the tuple of source and destination IP addresses in chronological order from the pcap files and store them separately in the text file ( 𝜎 ). We then shuffle the order of entries - not the tuple itself - and store the results as traffic without temporal structure ( Γ(𝜎) ). Next, we use the existing source and destination IP addresses form our environment but randomize all entries based on these addresses. We store this as as uniform traffic ( U(𝜎) ). We compress ( 𝐶(...) ) all files and use their size for temporal ( 𝑇(𝜎) ) and nontemporal (𝑁𝑇 (𝜎)) complexity values: 𝑇(𝜎)= 𝐶(𝜎) 𝐶(Γ(𝜎)) (1) 𝑁𝑇 (𝜎)= 𝐶(Γ(𝜎)) 𝐶(U(𝜎)) (2) 114