Benchmarking Performance of Various MQTT Broker Implementations in a Compute Continuum

Benchmarking Performance of Various MQTT Broker Implementations in a Compute Continuum Jasenka Dizdarević∗, Marc Michalke∗, Admela Jukan∗, Xavi Masip-Bruin†and Francesco D’Andria‡ ∗Technische Universität Braunschweig, Germany †Universitat Politècnica de Catalunya, Barcelona, Spain ‡Eviden, Barcelona, Spain {j.dizdarevic, m.michalke, a.jukan}@tu-bs.de, x, y Abstract—With the increasing adoption of IoT devices and applications, significant research and development efforts have been centered around engineering novel ecosystems referred to as the IoT-edge-cloud compute continuum. In this article, we implement, analyze and present a case study for performance benchmarking of five well known and select open source MQTT broker implementations in an open-source compute continuum testbed. The proposed MQTT broker implementations are evaluated in terms of response time, different payload sizes and throughput. Measurements and results show that the hardware platform used, the message size, as well as the network parameters (latency, packet loss and jitter) have a significant impact on the resulting performance of various broker implementations and therefore have to be carefully considered in the selection process for the building blocks of the continuum. All implementations and measurements are made to be fully reproducible and free and open source. I. Introduction With the increasing expansion of Internet of Things (IoT) applications, edge and cloud computing evolved as pivotal technologies, with a large spectrum of service offerings. Combined with IoT technologies and devices, edge and cloud computing create a novel ecosystem, often referred to as IoT-edge-cloud compute continuum [1], [2], - a combination of technologies that jointly process and store data in a variety of the so-called verticals, including health care [3], manufacturing [4], and transportation [5]. Engineering a functional compute continuum today is an open challenge, both conceptually and at the system level. Conceptually, application requirements drive the optimal placement of software components on a heterogeneous IoT-edge-cloud system that needs to act as single distributed entity (a ”meta operating system”). At the system level, the major challenge is in the heterogeneity of devices, software tools and protocols. To us, the choice of an application layer communication protocol is of particular interest. Since services previously deployed at the cloud premises can now also be located at the edge to support latency-sensitive IoT applications, the optimal placement of communication protocol broker/server instances regarding their performance is an open challenge [6]. Over the past years, a number of the protocol solutions, including Message Queuing Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), Hypertext Transfer Protocol (HTTP), Extensible Messaging and Presence Protocol (XMPP), Advanced Message Queuing Protocol (AMQP), and Data Distribution Service (DDS), have found their use in IoTedge-cloud continuum systems, each with various degree of adoption in various application scenarios. In a compute continuum, we not only need to consider these protocols for the exchange of messages between the different system’s components, but also the corresponding message brokers and/or servers. Of special interest are open-source MQTT broker implementations and their integration into the compute continuum, as MQTT remains the currently most widely adopted publish/subscribe protocol despite its known shortcomings, such as limited scalability of centralized broker architectures or its TCP-based transport. In this paper, we study and implement a performance benchmarking of MQTT in a compute continuum testbed. We deploy and test five well-known open source MQTT broker implementations, i.e. Mosquitto, EMQX, RabbitMQ, VerneMQ, and HiveMQ and their integration into edge-cloud network eco-systems. The developed testbed allows for interoperability with various MQTT protocol frameworks, whose utilization can be of interest to both, researchers and developers. It consists of two hardware setups; a cluster of two virtual machines to measure the performance when typical AMD64 nodes are used, as well as a cluster of two Raspberry Pis to analyze the performance if these nodes are replaced by less powerconsuming ARM64 devices. To compare the performance of different broker implementations, we configure the underlying network system that interconnects the continuum in the testbed by adjusting the parameters network delay, variance and packet loss. The broker performance is then benchmarked in terms of testbed performance offset, message exchange latency for different payload sizes and throughput under a high load, both on the publisher and subscriber sides. The conducted experiments showcase the impact on the resulting performance when using different hardware platforms, message sizes, network parameters and number of clients. Among them, the largest performance differentiator is showcased to be hardware with ARM64 based processors resulting in much higher response time for all tested brokers. The most unexpected results have been noted for the throughput measurements, where the broker implementations with a higher publishing throughput (e.g. RabbitMQ) provide a significantly lower throughput on the subscriber side compared to the other solutions. The rest of the paper is organized as follows. Section II presents the background in terms of related work and communication protocol context in the continuum. Section III describes the testbed and its configurations. Experimental benchmarking tests and results are presented in section IV. Section V concludes the paper and provides an outlook. II. Background A. Communication protocols in a continuum In a compute continuum, computing and data processing can happen at various degrees of intensity and performance in all parts of the system. For the sake of this paper, let us assume that the highest level of processing capabilities is offered by the cloud, the lowest level can be found in IoT, with edge scaling in between from reasonably powerful fog computing nodes close to the cloud, to low-energy mist computing nodes close to the IoT. There is a wide range of devices in the edge computing category, often leveraging almost any central processing unit (CPU) architecture, depending on their environment and power supply. For instance, devices with low power consumption and considerable computing capabilities such as Raspberry Pi single board computers or other ARM based devices can fit into both, the IoT and mist (and therefore edge) computing categories, depending on the deployment. For simplicity, we opt for the definition of the devices in the edge as more resource constrained in comparison to the cloud, hence assuming medium processing, storage and communication capabilities. Due to the storage and processing requirements and overhead imposed by containerization and container orchestration, we assume, without loss of generalization, that containerized services can only be used in the edge and the cloud parts of the system, as the IoT devices lack the respective capabilities in these areas. For the assumptions regarding the networking performance, lower network delay values are assumed in the edge part of the continuum, as it is expected that the edge computing resources are closer to the end devices. The communication in the continuum is achieved by using different communication protocols. Regarding the choice of these protocols for the different interfaces across the continuum, a variety of solutions is considered due to the variety of Quality of Service (QoS) requirements of the applications and the capabilities of the system. As the cloud assumes computationally intensive tasks and fewer resource constraints, making it less sensitive to the communication overhead, the developers can choose protocols simply based on their popularity and compatibility with the expected infrastructure. In fact, the most prominent protocols here include (RESTful) HTTP(S) and the plethora of protocols based on it, as well as publish-subscribe protocols like AMQP. To enable message exchange through these protocols, software components in the cloud have to host corresponding message brokers and/or servers. On the other hand, to establish communication with the IoT devices and collect their data, the devices in the edge have to host message brokers and/or servers with less computationally intensive protocols, such as MQTT and CoAP, thus minimizing the communication overhead. It is important to note that for the communication aspect, we must not only consider the protocols for the exchange of messages between the different system’s components in the continuum, but also the corresponding message brokers and/or servers. In this work we focus on MQTT protocol solutions, as it remains a highly relevant protocol in IoT related continuum systems [7]. When choosing MQTT broker applications to analyze, there are no restrictions introduced upon them by the cloud resources of the continuum. We do however consider the ability to deploy the brokers on resource constrained architectures and under lower latency restrictions as a crucial feature for their deployment in the edge part of the continuum, as this allows for placement of the communication solution on ARM64 devices, closer to the end devices, instead of restricting the placement options to the cloud and its more resource capable AMD64 devices. From all available MQTT broker implementations, we evaluate the five most widely adopted open-source options. Table I shows the specific details of these chosen MQTT brokers. TABLE I Supported MQTT broker implementations Name Source Language ARM64 v3.1.1 v5.0 Mosquitto mosquitto.org C ✓ ✓ ✓ EMQX emqx.io Erlang ✓ ✓ ✓ RabbitMQ rabbitmq.com Erlang ✓ ✓ – VerneMQ vernemq.com Erlang – ✓ ✓ HiveMQ hivemq.com Java – ✓ ✓ We focus on brokers with official container images that support the ARM64 architecture, which currently includes three implementations. Namely, these are the most widely used Mosquitto, the highly scalable EMQX as well as RabbitMQ which was initially designed for the AMQP protocol but also supports MQTT. In addition, we analyze two other broker applications that have been used in both distributed edge and cloud computing; HiveMQ and VerneMQ [8]. B. Related work While the compute continuum makes it possible for different applications and systems to benefit from the advantages offered by both edge and cloud computing paradigms, it results in a complex heterogeneous execution environment, which itself presents a number of challenges and open issues. There have been various significant efforts in defining generalized continuum reference architectures [9]–[11]. Other works have focused on the orchestration of the application workflows across the continuum resources [12]. Increasingly important directions for the continuum are related to service scheduling [13], [14], as well as the development of frameworks and testbeds that facilitate application and service deployments in edge-cloud environments, focusing on automated deployment of both scientific and production workflows [15], [16]. Further work [17]–[19] addresses the placement of applications across the continuum with machine learning and artificial intelligence, indicating the complexity of the application placement problem. These automated placement attempts however have not yet resulted in production-ready solutions, which leaves predefined placement of services as the current method of choice. Such decisions therefore still need reference values regarding the expected relative performance when services are placed at different positions among edge and cloud. While these reference values have started to be explored for processing through serverless functions [1], [20], [21] or machine learning [22], [23], this gap still exists for messaging, which we contribute to closing here. In general, the most promising angle to generate performance indicating numbers for an application is the benchmarking of the software under circumstances that resemble the production environment to be expected. To this end, the authors in [11], developed a framework to create an edge-cloud network and benchmark ML applications across it. They experiment with different offloading models and test the resulting performance of the ML application under different conditions. This is done by emulating the characteristics of different parts of the continuum. Similar benchmarking work has been conducted by the authors of [24] for the OpenWhisk serverless platform. Our work aims to provide a similar framework that can be freely repurposed for other networking environments. For the communication aspect of the continuum, papers [25]–[27] compare different communication protocol solutions and lay out their respective advantages and disadvantages. Regarding the MQTT protocol chosen in this paper, related work compares the performance of different message brokers similar to [28]. The latter study focused on two testing environments; a local setup and a proprietary cloud solution, with the Mosquitto broker solution outperforming others for most of the observed metrics. Other work focuses on smart cities regarding the performance [29] of MQTT brokers when connected through different bandwidth-limited links or an architecture to integrate them [30] albeit without consideration of link delay despite the focus on IoT devices. While there is no lack of studies of different aspects of MQTT communication in the cloud environments, particularly in terms of various whitepapers provided by broker vendors, studies focused on the edge or the IoT-edge-cloud continuum are few and far between. Work [31] conducts benchmarking of the performance of multiple MQTT brokers in clustered configuration while considering different locality setups, meaning the publishing and subscribing clients being connected to brokers on either a different or the same site as the publishers. It also introduces either 5 or 50 ms of total delay between the sites. It tests and compares the most known MQTT brokers EMQX, VerneMQ, RabbitMQ, HiveMQ, Mosquitto. These aspects are similar to the study in this paper. We do not however exclusively focus on conventional x86/AMD64 hardware but also consider the ARM64 architecture in the context of non-clustered broker deployments. In addition, we focus on the message size and concurrent requests to close the knowledge gaps for these dimensions. Finally, our measurements are also focused on the subscriber throughput, which is in stark contrast to other related work which mostly focuses on publisher throughput. This is critical for broker implementations under various traffic and load conditions. We create a testbed that allows for automated scheduling of different benchmarks and manipulation of link metrics like delay, delay variance or packet loss to asses the performance of various MQTT message brokers under different networking circumstances. We choose the networking metrics based on the performance of the cloud and edge analyzed in [32] and the values used in [21] and evaluate the performance of broker implementations under various traffic and load conditions including different emulated locations across the compute continuum. Our novel contribution can therefore be summarized as follows: •Creation of performance reference values for messaging/communication solutions across the continuum. •Creation of a reproducible framework to evaluate MQTT brokers at different locations across the compute continuum by emulating cloud and edge networking characteristics. •Focus on both; conventional x86/AMD64 hardware and the ARM64 architecture in the context of nonclustered broker deployments. •Consideration of the impact that the number of concurrent requests has on the message throughput, not just on the publisher but also on the subscriber side. •Publishing this open-source benchmarking framework such that it can be freely repurposed for evaluation of different communication protocol broker/server applications for other networking environments or broker solutions. III. Testbed Fig. 1a shows the network testbed engineered for experimental benchmarking of different MQTT broker implementations as presented in Table I. The hardware configuration of our testbed consists of two Raspberry Pis (ARM Cortex A72, 8GB) and two Virtual Machines (4vCPU, 8GB), as shown in Fig.1a, which are deployed on a hypervisor (Intel i9-10900X, 64GB), as well as a Desktop PC (Intel i5-6500, 16GB) as a tester node. All devices are interconnected through a Gigabit Ethernet switch. The operating systems include Ubuntu version 22.04.2 for the tester, Ubuntu 20.04.6 for all Raspberry Pis (ARM64) and VMs (AMD64), and Arch Linux for the hypervisor. When a benchmarking test is executed through the benchmarking tool on the tester node, one of two infrastructure setups is used as a benchmarking target. The first setup consists of two virtual AMD64 machines; one Headnode and a worker node. The second setup consists of two ARM64 Raspberry Pis with the same kind of role assignment. Across either setup, the nodes are connected through an overlay network (Nebula1) on top of which the container orchestration (k3s) forms a Kubernetes cluster. This cluster is then leveraged to deploy each of the respective containerized broker implementations through Kubernetes deployment and service definitions, one broker at a time, which ultimately are targeted for the performance test. Once a test is completed, the next broker implementation is deployed and used as a target, with all of these tests being repeated on both platform setups; AMD64 and ARM64. For the benchmarking procedure, our setup uses the two benchmarking tools jmeter and mqtt-stresser, with the tool and parameters being defined per benchmarking description that is applied to the testbed. Next, we provide a detailed overview of the testbed deployment workflow as shown in Fig.1b for the benchmarking of the different open source MQTT broker applications. This workflow is executed through the automation tool Ansible2based on so-called Ansible playbooks (YAML files), that define all necessary requirements and steps to be executed against the actual testbed infrastructure. The workflow can be divided into seven steps which are explained as follows: 1) Installation of all requirements onto the machines, includes the benchmarking tools for the tester node and basic utilities like curl for the Headnode and worker nodes 2) Installation of the Nebula overlay network to ensure connection between the nodes that shall form a Kubernetes cluster, even if they should not reside in the same LAN 3) Installation of the Kubernetes cluster with the Headnode hosting all control services and all potential applications restricted to be exclusively deployed onto the worker nodes 1https://github.com/slackhq/nebula 2http://ansible.com/ Tester node Benchmarking Tools Virtual machine AMD64, 8GB Headnode Worker node Virtual machine AMD64, 8GB Headnode Raspberry Pi ARM64, 8GB Raspberry PI ARM64, 8GB Worker node Container orchestration Overlay network Broker/server applications Container orchestration Overlay network Broker/server applications (a) Testbed infrastructure 5. Deploy containerized broker applications 3. Initiate and manage k3s cluster from headnode 4. Modify the interfaces of the overlay network Tester node 1. Initialize deployment on the VM/RP Nebula Worker node Headnode Ansible SSH SSHSSH Infrastructure inventory (YAML file) Nebula (YAML file) 2. Deploy overlay network k3s orchestrator (YAML file) Nebula k3s master k3s worker netem (YAML file) latency, variance, p. loss latency, variance, p. loss Broker containers (YAML file) Mosquitto, HiveMq, EMQX, RabbitMQ, VerneMQ 6.1. Run performance offset test against brokers Bechmarking tools Jmeter 6.2. Run payload size test against brokers 6.3. Run scalability test against brokers mqtt-stresser Results (.csv and .txt) 7. Generate results and initialize reset (b) Deployment workflow Fig. 1. Testbed for experimental benchmarking of MQTT broker implementations 4) Manipulation of the interface parameters for introduction of additional delay, packet loss or delay variance 5) Deployment of the containerized application (one of the five brokers) as Kubernetes deployment and service onto the worker node 6) Execution of the three performance tests against the Headnode (who forward the requests to the application) 7) Generation of the result files and cleanup of the infrastructure Here, steps 6.1 to 6.3 represent the individual benchmark operations to be executed against the cluster. They are described in more detail in the respective subsections under section IV. A. Infrastructure and workflow initialization (Step 1) In order to initialize the deployment of required configurations and installations on the targeted devices that form the testbed infrastructure, the prerequisites defined as part of the testbed are installed to the devices based on their role in the overall system. These include the necessary benchmarking tools onto the tester node, basic utilities used for download scripts to the worker nodes and Headnode, as well as copying files that are needed at later stages of the process to the respective nodes, creation of results folders with current timestamps, preparation of Nebula certificates and cleaning up previous deployments from all machines, in case they are still running. B. Containerization and networking (Steps 2, 3 and 4) To connect the Headnode and all worker nodes considered in each setup through a Nebula overlay network with established virtual interfaces, a Nebula lighthouse is installed onto the Headnode first. Once this succeeded, the worker nodes join the overlay network by leveraging the certificates previously generated for them. Once the overlay network is established, the Headnode creates a Kubernetes cluster by installing k3s with installation arguments like the version to be used being defined in the dedicated YAML configuration file. The worker nodes then join this cluster. A Kubernetes node selector restriction is created for the namespaces such that applications are later only deployed onto the worker nodes and not the headnode. This ensures that the resources for the Kubernetes control plane are consumed on the Headnode, which frees the worker nodes to run the broker implementations. Therefore, the impact of Kubernetes control plane operations onto the test results can be minimized. These steps can be seen in Fig.1b as steps 2 and 3, respectively. Step 4 consists of applying the link metrics specified in the playbook to the corresponding network interfaces of the machines to emulate different networking environments, connections or distances across the edge-cloud continuum. For the testbed to be used in the continuum computing context, there is an additional dimension to consider besides potentially different CPU architectures and capabilities of hardware platforms; the networking performance impacted by the different traffic parameters between all nodes. To explore potential performance differences between the brokers when this factor is considered, the testbed enables running tests over different network configuration scenarios, designed to approximate realworld scenarios. The emulation of different networking configurations is modified through the netem configuration YAML file (step 4 in the deployment workflow), where different latency, jitter and packet loss values are defined, which are then applied to the virtual overlay network interfaces. Higher amounts of delay can for example show how a cloud node has a higher distance to other nodes or an end device, while higher delay variance and packet loss could also replicate a wireless connection. The exact values for the benchmarking of MQTT broker implementations are listed in Table II, with each listed value representing the parameters that can be measured across the connection of any two nodes of the network, as well as the connection of the tester to any node. Based on the latency values measured in [32] and the scenarios defined in [21], we assume a total delay of 2.5 ms, a delay variance of 0.5 ms and a packet loss of 0.04% for our edge scenario and a delay of 6.25 ms with a delay variance of 1.25 ms and a packet loss of 0.1% for cloud data centers as shown in Table II. TABLE II Emulated parameters for network configurations Scenario Latency (ms) Variance (ms) Packet Loss (%) local 0 0 0 edge 2.5 0.5 0.04 cloud 6.25 1.25 0.1 According to these network parameters, we define three testbed network scenarios in which the broker implementations will be benchmarked: local, edge, and cloud case. While the local scenario serves as a baseline without any additionally introduced latency, variance or packet loss, the edge scenario represents what we consider an achievable best-case scenario for an edge computing deployment with very low latency. The cloud scenario on the other hand represents the lowest latency that would be reasonable to assume for a cloud deployment. C. Deploying broker applications (Step 5) We deploy the five open-source broker implementations shown previously in Table I (Mosquitto, EMQX, RabbitMQ, VerneMQ and HiveMQ) and Fig. 1b in step 5. To deploy the selected brokers in the testbed, the corresponding containers are deployed on the worker node of each testbed setup. Mosquitto, EMQX and RabbitMQ are analyzed on the Raspberry Pi setup since they are the only solutions with support for the ARM64 architecture, while all five brokers are analyzed on the AMD64 setup. D. Benchmarking tools (Steps 6 and 7) The benchmarking process itself is executed in step 6, which loops over the different test definitions, executing them one by one. For this task, the respective testing tool is used along with the parameters defined in each individual benchmarking task. Here, we use two testing tools with MQTT support; an MQTT plugin for the JMeter load testing tool3for the performance offset and payload size tests, as well as the mqtt-stresser scalability testing tool4for the scalability test. The parameters required for the utilization of these benchmarking tools (MQTT ports to be used, payload message size, number of clients, etc.) are also provided through YAML configuration files, and are run against the respective brokers (see steps 6.1, 6.2 and 6.3 of Fig.1b). Once this is accomplished, the results are retrieved and the environment is cleaned up by uninstalling the overlay network and the container orchestrator which also deletes all brokers deployed on top of it. An example of Ansible playbooks for scalability testing of EMQX broker application is shown in a following YAML file: 3https://github.com/emqx/mqtt-jmeter 4https://github.com/inovex/mqtt-stresser 1name: Playbook example for emqx and s tr e s s e r 2hosts : testers 3vars : 4application : emqx 5benchmark_tool : mqtt - s t r e s s e r 6mqtt_stresser : 7url : ”{{ address_testbed }}” 8port : 31884 9num_clients_set : [800 , 700 , 600 , 500 , 400] 10 num_messages : 1000 11 global_timeout : 60s 12 network_sets : 13 name: lo c a l 14 delay : 0 15 variance :0 16 loss : 0 17 registry : 18 user : user_name 19 ip : ip_address 20 port : port_number 21 tasks : 22 - name: Launch emqx/mqtt - s t r e s s e r workflow 23 ansible . bui l tin . include_tasks : workflows/ emqx/main . yml 24 This playbook file is configured to set up an EMQX broker instance (emqx) to be first installed on the testbed infrastructure (address_testbed) and then tested with the mqtt-stresser tool by specifying the port on which the broker application can be accessed in the testbed. The parameters of the tester itself are also configured, including number of emulated concurrent clients (num_clients_set), number of messages (num_messages) and the timeout of the entire operation (global_timeout). Here we also specify which network scenario is to be deployed across the testbed via network_sets. Detailed instructions on the testbed used and deployment of MQTT brokers over the AMD64 or ARM64 setup are ready to be published on Github, with all results, software tools and code fully reproducible and free and open source. IV. Experimental results In this section, we describe the benchmarking tests conducted as well as the measurements obtained. We define three types of tests, i.e., (i) testbed performance offset, (ii) payload size response time and (iii) throughput. The detailed specifications of each tests are shown in Table III. We utilize MQTT protocol version 3.1.1. For the first two tests we measure the latency as the time elapsed from sending a message from the MQTT publisher to the MQTT subscriber. For this type of measurements we use an MQTT plugin for the tester tool JMeter with customized test plans. For the throughput tests, we stress the MQTT broker with an increasing number of concurrent clients publishing a number of messages, and then measure the throughput in form of the messages per second successfully published to the broker and the messages per second successfully received by the subscribers. In this case, we use the mqtt-stresser scalability testing tool, configuring its parameters for the number of MQTT clients, as well as the messages per client. The mqttstresser tool is selected in this case as it allows for easier test configurations than is currently possible with the open source version of the JMeter MQTT plugin. We consider two hardware setups for the first two tests, i.e., AMD64 and ARM64, for three different network scenarios, i.e., local, edge and cloud. In each of these tests, we use one subscriber thread and adjust the number of publisher threads, with each publisher thread sending a single message to the target broker. The published messages are set to include timestamps, which are then subtracted from the corresponding timestamps on the subscriber side. Each test is repeated at least 10 times to minimize the potential margin of error and ensure reproducibility of the results. For the third test, we use the AMD64 setup only, and a local scenario. A. Testbed Latency Offset This initial measurement serves to evaluate the performance offset introduced when publishing a simple helloworld string to the MQTT broker deployed on the testbed. JMeter is used to measure the response times in terms of latency, as previously defined, and collect them in a separate .csv file per executed test. Here, we conduct two sets of measurements of the response time on the subscriber side. In the first set, we analyze broker implementations with official ARM64 container support (Mosquitto, EMQX and RabbitMQ). In the second set, we compare all five of the initially selected implementations (Mosquitto, EMQX, RabbitMQ, HiveMQ and VerneMQ). On the publisher side, we configure a utilization value of 100 threads, each sending the same message with an interval of 250ms. The QoS value of both, the MQTT subscriber and the publisher, in the protocol is set to the reliable level 1, which implies that a successful message transfer to the broker is guaranteed. It should be noted that neither the authentication nor the security mechanism was used. For the first set of measurements, the response time values for the brokers running on the ARM64 setup are shown in Fig.2a with boxplot graphics showing the median value for each of the brokers as well as variations in the data obtained. Here, we can see that the median value for the performance offset tests is around 5 ms in the local scenario, 10 - 12 ms in the edge scenario, and 18 - 19 ms in the cloud scenario, for all implementations. The figure also shows that the boxplots are almost of the same size for each scenario with low variation of the values. While Mosquitto seems to preform slightly better, the comparable delays in all cases are very low and can be considered negligible. For the second set of measurements, the response time values for all five brokers running on the AMD64 setup are shown in Fig.2b. The median value for performance offset is around 4.8 - 8 ms (local), 8 - 13 ms (edge), and 11 - 18 ms (cloud scenario), for all implementations. This case TABLE III Benchmarking tests specifications Test type Testing Setup Network No. of No. of QoS Level Payload No. of messages tool scenarios publishers subscribers size per publisher Performance JMeter AMD64, local, edge, cloud 100 1 1 10B 1 offset ARM64 local, edge, cloud 100 1 1 10B 1 Payload size JMeter AMD64, local, edge, cloud 100 1 1 1KB, 10KB, 1MB 1 response time ARM64 local, edge, cloud 100 1 1 1KB, 10KB, 1MB 1 Throughput mqtt-stresser AMD64 local 100 - 800 100 - 800 0 10B 100, 1000 mosquitto_l emqx_l rabbitmq_l mosquitto_e emqx_e rabbitmq_e mosquitto_c emqx_c rabbitmq_c 0 5 10 15 20 Performance offset [ms] mosquitto emqx rabbitmq (a) ARM64 setup mosquitto_l emqx_l rabbitmq_l mosquitto_e emqx_e rabbitmq_e mosquitto_c emqx_c rabbitmq_c 0 5 10 15 20 Performance offset [ms] mosquitto emqx rabbitmq (b) AMD64 setup Fig. 2. Testbed performance offset however shows higher variation of the values, particularly in the cloud scenario, and especially for the HiveMQ broker, which shows the highest deviation of all brokers, even under comparably good conditions. This behavior becomes more pronounced the worse the networking conditions (delay, packet loss and jitter) become, notably exceeding the other brokers’ deviations while maintaining a similar median response time. The opposite effect can be observed with the RabbitMQ broker, which shows slightly elevated median response times while maintaining significantly more predictable response times than the other brokers. When comparing the median values of the three broker implementations that have been tested on both setups in Fig.2a and Fig.2b, we can also see a slightly lower response time by roughly one ms which can be considered negligible. This difference can also be traced back to the slight performance advantage that Virtual Machines have over Raspberry Pis since VM’s inter-node traffic does not have to pass through a physical network medium but remains in the same physical device instead. Overall when analyzing the performance offset of the testbed itself, we found comparable response times for all brokers, when deployed in the same network scenarios and on the same hardware platform setups. B. Latency vs. payload size For the payload tests, we also configure a utilization of 100 threads on the publisher side, sending messages of different payload sizes with an interval of 250ms. We modify the message size by defining strings of different fixed lengths, considering 1KB, 10KB and 1MB sizes. Next, we measure the latency on the subscriber side for each payload size and for each broker implementation in all three network scenarios, for the AMD64 and for the ARM64 setup. The QoS of both, the subscriber and publisher, remains on level 1. For the first set of broker implementations tested on both setups, i.e., ARM64 and AMD64, the median latency values and inter-quartile range (IQR), which shows how spread the boxplot is, are shown in Table IV, with their values in milliseconds. When sending a 1KB message, the resulting median values in both setups are very close in all three measurement scenarios (local, edge, cloud), with negligible differences between the broker implementations. Even the differences in terms of latency values are low between the local and cloud scenario (e.g., Mosquitto, local median 5.39 ms and cloud case median 18.6 ms). If we compare these median values with those obtained as performance offset and shown in Fig.2a (e.g., approximately 4.8 ms and 17 ms median values for local and cloud case scenario for Mosquitto) we can see that the 1KB payload size introduces a very low additional latency. TABLE IV Payload size: latency median and IQR - Mosquitto, EMQX and RabbitMQ Mosquitto EMQX RabbitMQ local edge cloud local edge cloud local edge cloud 1KB ARM64 5.39 - 1.00 10.67 - 1.00 18.60 - 2.00 6.04 - 0.25 11.54 - 1.00 18.93 - 2.00 7.00 - 1.00 12.00 - 1.00 19.87 - 2.25 AMD64 3.94 - 0.00 8.59 - 2.00 15.23 - 6.00 4.24 - 0.00 9.14 - 3.00 15.25 - 7.00 4.83 - 1.00 9.65 - 3.00 15.59 - 6.00 10KB ARM64 10.68 - 1.00 15.91 - 1.00 24.90 - 2.00 11.44 - 2.00 16.56 - 2.00 24.71 - 2.00 13.16 - 2.00 17.73 - 1.00 26.15 - 2.00 AMD64 4.33 - 0.00 8.83 - 2.00 16.85 - 7.00 4.66 - 1.00 9.43 - 2.00 16.67 - 7.00 5.76 - 1.00 9.85 - 1.00 17.25 - 6.00 1MB ARM64 272.48 - 32.00 332.42 - 48.00 643.17 - 181.25 274.11 - 35.00 333.04 -49.00 641.02 - 193.75 297.74 - 53.00 379.31 - 55.00 679.16 - 167.00 AMD64 66.27 - 13.00 153.96 - 75.00 437.10 - 388.25 65.2 - 14.00 149.60 - 69.25 418.81 - 356.50 71.97 - 13.00 161.38 - 64.25 429.83 - 379.50 vm_l rpi_l vm_e rpi_e vm_c rpi_c 0 200 400 600 800 1000 Latency [ms] RPi setup VM setup Fig. 3. Payload size 1MB: ARM64 setup vs AMD64 setup The IQR values also are in a low range, with slightly higher variance for the cloud case network scenario of RabbitMQ and EMQX implementations. Similar trends continue for 10KB message size, with the expected increase of latency in all cases compared to a message size of 1KB. This increase remains in the range of approximately 5 ms in all scenarios and all implementations. Moving to 1MB message size, we can observe significant performance differences. First, when comparing the same broker implementations on the AMD64 and ARM64 setup, we can observe that in each broker implementation and in each network scenario, VMs perform better with noticeably lower median response times, as well as lower variance. The difference in latency between these two hardware setups is approximately 200ms for all implementations, which can be a limiting factor in low-latency demanding IoT applications, where MQTT is often the protocol of choice. On the other hand, when comparing the median response times for the same scenario on the ARM64 setup, e.g. 643.17 ms (Mosquitto), 641.02 ms (EMQX) and 679.16 ms (RabbitMQ) for the cloud scenario, we can see that the broker implementation itself does not play a key role in the overall results. The same can be said for AMD64 setup. In contrast to what was observed for the small payload sizes, in the case of 1MB, the IQR range is higher, indicating a higher variance of the obtained values. TABLE V Payload size: latency median and IQR - HiveMQ and VerneMQ HiveMQ local edge cloud 1KB AMD64 9.70 - 5.00 13.46 - 5.00 18.40 - 7.00 10KB AMD64 9.95 - 4.00 13.94 - 5.00 19.75 - 8.00 1MB AMD64 125.00 - 24.00 194.72 - 73.00 374.80 - 320.00 VerneMQ local edge cloud 1KB AMD64 4.37 - 1.00 8.30 - 3.00 13.64 - 6.00 10KB AMD64 5.18 - 1.00 9.23 - 2.00 15.23 - 7.00 1MB AMD64 76.25 - 12.00 144.85 - 71.00 345.64 - 292.50 The effect that the specific testbed setup has is also visually shown on the example of the RabbitMQ broker in Fig. 3 for 1MB message payload size. As it can be seen, the difference in the response time on the AMD64 and ARM64 setup is noticeable in each of the three scenarios. The median latency values and IQR for the remaining broker implementations, which are tested only on the AMD64 setup, are shown in Table V. For a message size of 1KB we can see the same trend as in the previously discussed implementations, with the median latency values being slightly higher in the local network scenario for HiveMQ. If we compare these median values with the ones obtained as performance offset and shown in Fig.2b (e.g. approximately 8 ms and 18 ms median values for local and cloud scenario for HiveMQ) we can again see that 1KB payload size introduces negligible latency. The same trend continues for 10KB message size, with the increase of latency compared to 1KB message being less than 1 ms. Moreover, this result can be observed in all five broker implementations tested on the AMD64 setup. As shown previously, the more significant deteriorations in performances of both median latency values and IQR range can be noticed with 1MB payload size. However, if we now compare all median response times for the edge scenario on the AMD64 setup, e.g., 153.9 ms (Mosquitto), 149.6 ms (EMQX), 161.38 ms (RabbitMQ), 194.72ms (HiveMQ) and 144.85 ms (VerneMQ), we can see that the differences are not very pronounced, with HiveMQ performing slightly worse than other implementations. However, if we consider the cloud scenario (AMD64 setup), we can see a significant difference between the two additional brokers VerneMQ and HiveMQ and the remaining brokers. Namely, their mean values of 345.64 ms (VerneMQ) and 374.80 ms (HiveMQ) are considerably lower than 429.83 ms (RabbitMQ), 418.81 ms (EMQX), and 437.10 ms (Mosquitto) scored by the remaining brokers. This is especially interesting given that HiveMQ preformed worse than the others when using smaller payloads. C. Throughput and Scalability In the final tests, we increase the number of concurrent publishing and subscribing clients, scaling from 100 to 800 clients. Each publisher client is set to send a number of messages (100 and 1000 per client) of 10B payload size. These parameters are specified in the playbook configuration file for each of the brokers (example shown in Section III-D), and each test has been repeated at least 10 times. It is important to note, that the default pause between messages was kept at 0s, in order to increase the load for the brokers. The QoS of both types of MQTT clients is set to level 0 (no guarantee of the delivery), as we assume a local network scenario with a stable connection and thus without added latencies, jitter and packet loss. The measurements were performed on the AMD64 setup, in order to be able to compare all five brokers. For the first set of measurements with each client publishing 100 messages, the published and received throughput in terms of number of messages published and received per second, are shown in Fig.4a and 4b, respectively. The results in Fig.4a, show rather constant publishing throughput values when scaling from 100 to 800 clients for Mosquitto (around 80000 msg/s), EMQX (around 75000 msg/s) and VerneMQ (around 60000 msg/s). On the other hand, both RabbitMQ and VerneMQ show a sharp increase in publishing throughput, with the increased number of clients. Overall the highest publishing throughput values are measured for RabbitMQ, while the worst performances are measured for HiveMQ. In order to fully understand broker behavior under these high loads, we also study the received throughput, which is a salient feature of our approach. It is measured on the side of subscriber clients, shown here in Fig.4b. The received throughput for all brokers follows almost a linearly decreasing trend, with the delivered message rates significantly lower than the one published to the brokers for each number of clients. In addition to the reduced received throughput performance, the brokers with the highest publishing throughput do not guarantee highest received throughput. This is most obvious in the case of the RabbitMQ broker, which while allowing for highest published throughput, performs almost the worst, with the received throughput around 400 msg/s, with lower values measured only for HiveMQ. The highest received throughput is measured for VerneMQ (2000-3000 msg/s) and EMQX (1800-2500 msg/s). When scaling the number of messages from 100 to 1000 for the clients, the published throughput trends were comparable. However, here the received throughput for all brokers follows a more noticeable decreasing trend, as seen in Fig.5, which is shown in logarithmic scale due to extreme difference in the received throughput between the EMQX and VerneMQ implementations (in range 1×104), RabbitMQ and Mosquitto implementations (in range 1×103), and the HiveMQ implementation (less than 1×102). As we can see there is a difference of more than 10 000 msg/s in case of 100 concurrent clients. For small numbers of concurrency (100-300), the difference from EMQX and VerneMQ to the other brokers is very high, with the gap closing the more the client number increases. The most surprising result is that the broker implementation with the highest publishing throughput - RabbitMQ, provides a much lower received throughput than the brokers with lower published throughput - EMQX, Mosquitto and VerneMQ. These results showcase the importance on measuring throughput on both MQTT client sides, as high throughput performances in terms of messages arriving at the broker might not get reflected in the number of messages that the broker is capable of forwarding to the end users. V. Conclusion and outlook We experimentally benchmarked the performance of five open-source MQTT Broker implementations at different locations across the compute continuum. To this end, we engineered a reproducible network testbed by emulating cloud and edge networking characteristics, supporting devices of both, AMD64 and ARM64 hardware architectures. When analyzing the performance offset of the testbed itself, we found comparable response times for all brokers when deployed in the same network scenarios and on the same hardware platform. In cases of small message payload sizes, with degradation of network conditions, a small latency increase could be noticed for each of the broker implementations. Under the same conditions, the latency differences between them could be considered negligible. Once the message payload size increased to 1MB, the performance of all implementations in all network scenarios deteriorates with respect to both; the median latency and its deviation. However, the largest performance differentiator has been in the testbed’s hardware setup itself, with the AMD64 setup resulting in much lower response times than the ARM64 setup. Finally, the most unexpected results were obtained for the throughput measurements under a high message and client load. The results showcased the importance of measuring throughput on both MQTT client sides, as the high published throughput performance were not reflected in the same number of messages the broker was capable of forwarding