An extensible lightweight framework for distributed telemetry of microservices

Contents lists available at ScienceDirect Sustainable Computing: Informatics and Systems journal homepage: www.elsevier.com/locate/suscom An extensible lightweight framework for distributed telemetry of microservices Manuel Otero a, José María García b, Pablo Fernandez b,∗ aUniversidad de Sevilla, Spain bSCORE Lab, Universidad de Sevilla, Seville, Spain ARTICLE INFO Keywords: Telemetry OAS API Microservices ABSTRACT Microservice architectures have become the standard for developing scalable distributed systems that offer significant benefits in managing the integration and evolution of complex applications. However, they face challenges in effectively diagnosing and resolving performance and reliability issues. Traditional centralized telemetry models and cloud-based monitoring platforms often require complex or costly configurations and are not optimized for RESTful microservices. In fact, although the OpenAPI Specification (OAS) has become a key standard for describing microservice APIs, existing telemetry tools do not leverage this information to enhance service analysis and diagnostics. This paper introduces a lightweight and distributed approach to telemetry that uses OAS-based API information, offering an automated, configuration-free system that enables developers and operations teams to perform root cause analysis more efficiently. Moreover, we propose a plugin system to incorporate intelligent behavior into the telemetry system, such as an adaptive proactive alert mechanism when response-time anomalies are detected. By incorporating this extensibility mechanism, the framework paves the way to address issues such as energy consumption and performance, allowing the system to dynamically adjust its monitoring activities to optimize resource usage and minimize the carbon footprint of microservice deployment and execution. This adaptability reduces operational overhead and supports sustainable computing practices. To validate our approach, we present a proof-of-concept in the form of a ready-to-use package for the NodeJS ecosystem, demonstrating that this distributed telemetry model can operate with minimal impact on system performance and resource usage, proving its effectiveness to support more robust and sustainable IT systems. 1. Introduction The rise of microservice architectures has established them as the standard for building scalable and continuous systems [1]. This approach offers considerable advantages when it comes to managing the integration and evolution of multiple components. However, as with any sophisticated architecture, issues such as performance deviations and component failures can result in cascading impacts throughout the system. In fact, despite the effectiveness of microservice architectures in addressing various concerns, they also introduce significant hurdles, particularly in detecting and resolving issues within the system. These challenges have been thoroughly examined in existing research [2], ∗Corresponding author. E-mail addresses: [email protected] (M. Otero), [email protected] (J.M. García), [email protected] (P. Fernandez). 1https://prometheus.io. 2https://elastic.co. 3https://datadoghq.com. 4https://newrelic.com. 5https://kubernetes.io. with solutions such as the circuit breaker pattern [3] being proposed to mitigate the risks of failure. An essential requirement for effectively managing microservice architectures is the ability to collect and analyze system data, which plays a key role in identifying the root causes of architectural problems. Traditionally, both academia and industry have leaned toward a centralized model to collect diagnostic data, where system components transmit their information to a central hub for analysis. Popular tools such as Prometheus1and ELK2follow this approach. Furthermore, platform-as-a-service (PaaS) offerings, including Datadog3and NewRelic,4provide cloud-based solutions for visualizing and analyzing diagnostics. In more complex environments, deployment on container orchestration platforms like Kubernetes5allows the implementation https://doi.org/10.1016/j.suscom.2025.101100 Received 14 September 2024; Received in revised form 28 December 2024; Accepted 10 February 2025 Sustainable Computing: Informatics and Systems 46 (2025) 101100 Available online 26 February 2025 2210-5379/© 2025 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC license ( http://creativecommons.org/licenses/bync/4.0/ ). M. Otero et al. of Service Mesh models, which centralize data collection across various services and offer the tools needed for managing elasticity and resilience within. However, these centralized approaches typically require complex configurations and ongoing maintenance or incur additional development and maintenance costs. In addition, the learning curve is steep and specific to each application. Furthermore, the type of metrics collected is often low-level and not integrated into a RESTful paradigm of the applications, necessitating custom configurations for diagnosing API-based architectures. These issues are prominent in solutions that use a cloud continuum approach, where a distributed telemetry framework is crucial to enable seamless monitoring of cloud continuum scenarios [4] and, to this extent, some approaches advocate providing orchestration frameworks with telemetry agents already integrated [5,6]. In recent years, with the advent of the Distributed Computing Continuum (DCC) perspective [7], a distributed telemetry model could also be a key element due to the inherent complexity and distributed nature of these systems, which span across various computing layers such as cloud, fog, edge, and IoT. Traditional centralized telemetry solutions are often inadequate for such environments because they do not efficiently handle the latency, bandwidth, and data sovereignty issues that arise when data must traverse multiple network segments and administrative domains. In this article, we present a lightweight decentralized telemetry approach that aligns with the DCC paradigm by enabling data to be processed and analyzed closer to its point of generation. This proximity minimizes latency, reduces transmission costs, and enhances the system’s responsiveness to detect and rectify issues in real-time. Furthermore, following principles from Distributed Computing research [7, 8], our proposed model supports a more flexible approach to monitoring and managing microservices health. This ensures that our telemetry system can adapt to the dynamic nature of DCC environments, where resources and services may fluctuate, scale, or migrate across different layers depending on demand, connectivity, or other operational considerations [9]. Such adaptability is a key factor in maintaining system performance, as highlighted in the discussion on adaptable business models in DCC systems [10]. Moreover, this need to measure distributed performance becomes even more challenging and critical in cross-organizational scenarios, where interdependencies between services provided by different organizations necessitate a comprehensive QoS enforcement or a global service level agreement (SLA) [11,12]. An additional motivation for our approach lies in the growing need for flexible, intelligent telemetry solutions capable of optimizing resource usage and energy consumption. In particular, in DCC environments, where microservices might operate in resource-constrained or dynamic nodes, a rigid telemetry model is inefficient. Therefore, we introduce a plugin-based extension mechanism that allows the telemetry system to adjust its behavior according to the context. For example, the system could save energy by reducing the frequency of monitoring during periods of low activity while scaling up its monitoring precision during peak loads to prevent performance issues. This energy-efficient and adaptive approach supports sustainable computing practices in modern microservice architectures. From a technological point of view, it is also important to highlight the role of the OpenAPI Specification (OAS)6as a critical standard for describing APIs in microservice architectures, particularly in complex DCC environments. OAS enables an explicit model of the interface of a microservice, defining operations and expected inputs and outputs. Despite this, current telemetry approaches often do not take full advantage of the detailed information provided by OAS to improve the analysis and reporting of microservice performance. To address this gap, we propose a lightweight distributed telemetry management model that takes full advantage of the OAS to develop tools that allow developers and operations teams to analyze 6https://github.com/OAI/OpenAPI-Specification. root causes of performance issues or service disruptions with minimal configuration. Using the OAS-defined API structure, our system can automatically infer operational parameters and generate detailed reports tailored to the behavior of individual microservices. This design eliminates the need for extensive manual setup, offering developers a transparent, fully operational system from the start. The shift from a centralized telemetry model to a more distributed, developer-friendly approach enables a faster and more efficient analysis of service health without the burden of additional configuration. Specifically, we provide a first proof of concept consisting of a ready-to-use package (OAS Telemetry) for the NodeJS ecosystem, one of the most popular technology stacks for implementing microservices. This package demonstrates the practicality of our distributed telemetry model and shows that it can operate with minimal overhead while delivering significant improvements in flexibility and intelligent monitoring. Moreover, from an extensibility standpoint, we propose a plugin model that enables dynamic load of more complex monitoring intelligence. Specifically, we present this model with an adaptive anomaly detection plugin that identifies abnormal response times in microservice requests. This plugin works by analyzing historical data to establish dynamic thresholds for expected response times and adjusting these thresholds based on real-time observations. When the system detects response times outside these acceptable ranges, an alert is generated, allowing rapid intervention. The plugin-based architecture ensures extensibility, allowing developers to integrate additional plugins in the future to further optimize energy consumption, monitoring precision, and system resilience. The rest of the article is structured as follows. Section 2discusses related work in the area compared with our proposal, which is described in Section 3. We validate our solution in three different scenarios to empirically assess its performance impact in Section 4. Finally, Section 5provide our concluding remarks and identify various areas for further research on this topic. 2. Related work Controllability has been identified as a key issue in the context of Microservice Architectures [13] and consequently monitoring and telemetry have gained attention in both the academy and industry [14]. Specifically, in the context of microservice telemetry, several works can be found that propose approaches to extract telemetry data to analyze software metrics [15,16]. However, they do not provide a concrete model or specification for application-level telemetry. In turn, other contributions focus on monitoring system architecture using telemetry data, defining their own model [17–19] or integrating widely used approaches such as Prometheus or ELK [20,21]. By using these frameworks, all the telemetry data are centralized, depending on an additional back-end for analyzing the data, such as commercial platforms like NewRelic or Datadog, or Open-Source alternatives like Jaegger or SigNoz. In turn, our approach leverages the distributed nature of microservice architectures to allow distributed instrumentation and analysis of this type of software systems, while relying on standard specifications and models such as OpenTelemetry and OAS. Concerning monitoring, there are also approaches that use OpenTelemetry to introduce agents into a service mesh to instrument the components [22]. However, they do not allow for a fine-grain configuration of the telemetry collection in a distributed manner. Recently, in [23] the authors proposed an interesting low-overhead, transparent tracing and performance debugging of Node.js-based microservices, however, this approach still relies on a centralized architecture to gather the traces and perform the analysis. One of the key benefits of a distributed telemetry approach is its ability to enable real-time adaptation of both monitoring strategies and system configurations. By decentralizing data collection and processing, systems can dynamically adjust their monitoring strategies, such as altering the frequency of data collection or even reconfiguring system Sustainable Computing: Informatics and Systems 46 (2025) 101100 2 M. Otero et al. components to better align with current conditions. This capability is particularly valuable for meeting service level objectives (SLOs) in complex environments. For instance, as discussed in [24], active inference models can be applied to adjust system configurations in response to real-time data, ensuring that the system operates efficiently and meets performance goals even under fluctuating conditions. This dynamic adaptation is most effective when telemetry data is distributed and kept close to the source, allowing for immediate actions and adjustments. 3. OAS telemetry framework OAS Telemetry is a framework designed to enhance observability within Node.js applications by collecting telemetry data that integrate into the OAS-Tools ecosystem, which is dedicated to providing tools and utilities for managing applications based on OpenAPI Specification (OAS), the standard for describing APIs within a microservice architecture. In such a context, the main aim of OAS Telemetry is to provide a supporting tool for developers looking to gain insights into the operation and performance of their OAS-based APIs, enabling them to monitor, debug, and optimize their applications effectively. As a key cornerstone of our proposal, we can find OpenTelemetry,7 which provides a telemetry framework whose aim is to enable observability for a software system, allowing operators to analyze the internal state of the system by examining their external behavior. In order to do so, the system must be instrumented, so that telemetry data (traces, metrics, and logs) can be generated and later analyzed by an observability backend, such as Prometheus. To this extent, OpenTelemetry focuses on collecting and managing the telemetry data, providing a specification and standard protocol for telemetry management that can be used regardless of the concrete implementation of the systems’ components. OpenTelemetry provides a standardized SDK that can instrument traces, metrics, and logs, although in the context of our proposal, primarily focuses on trace spans, which provide crucial details such as status codes and response times. Here is a minimal example: {"attributes": { "http.url": "http://localhost:8080/docs", "http.method": "GET", "http.status_code": 200 }, "_duration": [ 0, 2041000 ], "_spanContext": { "traceId": "df3...", "spanId": "9e0e..." } Within the NodeJS ecosystem, the primary aim of the OpenTelemetry SDK is to streamline the debugging, profiling, and performance monitoring processes of applications. In the context of microservices, developers typically rely on frameworks to scaffold and develop restful APIs; in particular, one of the most used frameworks is Express8 that provides a lightweight web framework for Node.js that simplifies HTTP request handling and allows efficient routing and middleware support, allowing developers to focus on building robust web applications and APIs with ease. Specifically, the Express Framework adopts a middleware-centric architecture that structures each request to the microservice (or API) as a sequence of function calls (i.e. the middlewares); this sequence facilitates the modular assembly of the request-handling pipeline, where each middleware function can process the request or response, or delegate to the next function in the stack. In such a context, the role of middleware in Express is to intercept the request–response cycle, allowing operations such as modifying 7https://opentelemetry.io/. 8https://expressjs.com/. Fig. 1. Architecture diagram. request and response objects, terminating the cycle, or passing control to subsequent middleware. These functions enable critical capabilities such as logging requests, authenticating users, managing sessions, and validating input. Middleware functions are executed sequentially, providing a mechanism for decomposing the application into smaller, reusable components that perform specific tasks independently while maintaining the flow of request processing. Using this idea, the OAS Telemetry framework provides an Express middleware for gathering telemetry data and managing the export process. It is compatible with ESM and CommonJS implementations and can be seamlessly integrated into existing Express applications with minimal setup required. By simply including two lines of code, developers can enable telemetry functionality within their applications: import oasTelemetry from ’@oas-tools/oas-telemetry’; ... // Initialize Express app and load openapispec app.use(oasTelemetry(openapispec)) ... // Rest of microservice code 3.1. Internal architecture In the context of a microservice, the OAS Telemetry package functions as middleware that sits between the application logic and the incoming/outgoing HTTP requests. This middleware allows the application to collect telemetry data without modifying the original code, requiring only the import of the package and the addition of the middleware. As shown in Fig. 1, The OAS Telemetry middleware has 4 main components: The controller initializes the telemetry elements, then the management component allows the developer to start/stop/reset the telemetry processing. The persistence component allows data fetching from the DB, and the UI component renders the information in a view designed for developers. Specifically, we can outline the components as follows: •Controller: This component manages the integration of the OpenTelemetry SDK by initializing the tracer provider and configuring the exporter and the span processor. The span processor handles the processing of spans and their exportation to the configured exporter. The exporter is an instance of the InMemoryDBExporter, a custom implementation of the OpenTelemetry framework. •Management: This component manages the span exports. It provides the endpoints to stop and start the telemetry data collection dynamically at /telemetry/stop and /telemetry/start endpoints, respectively. Despite the fact that telemetry instrumentation, once initialized, cannot be stopped, the InMemoryDBExporter enables the system to stop the insertion of spans into the database. This helps mitigate memory consumption. In addition, a/telemetry/reset endpoint is provided, which clears the database from the spans. Sustainable Computing: Informatics and Systems 46 (2025) 101100 3 M. Otero et al. Fig. 2. Global Architecture comparative diagram between OAS-telemetry and traditional telemetry solutions. •Persistence: The persistence component is responsible for managing the NeDB database within the InMemoryDBExporter component. This component facilitates the storage and retrieval of telemetry data generated by the system. Specifically, it provides two endpoints: /telemetry/list and /telemetry/find (accessible via the HTTP POST method). The /telemetry/list endpoint allows users to retrieve a list of all telemetry span data stored in the database. On the other hand, the /telemetry/find endpoint enables users to perform targeted searches for telemetry data by submitting NeDB queries in the request body. This functionality empowers users to extract specific subsets of telemetry data based on their unique requirements, enhancing the flexibility and utility of the system’s data management capabilities. •UI: This component generates a lightweight HTML page based on an OAS file path (see Figs. 3and 4) that are described in Section 3.4. 3.2. Global architecture comparative In this section, we explore how OAS-Telemetry integrates within a microservices infrastructure and compare it with traditional telemetry solutions, such as Jaeger and similar platforms. This comparison will help highlight the architectural differences in the way telemetry data is captured, stored, and accessed across both approaches. This is illustrated in Fig. 2, which presents the two integration patterns side by side: on the left, we show OAS-Telemetry embedded within each microservice, with local in-memory storage, and on the right a traditional telemetry setup that relies on a centralized telemetry platform. Note that the Telemetry platform is a simplified view of all external services such as agents, collectors, and back-ends, where telemetry data are aggregated, stored, and ultimately sent to a central point for analysis. In a traditional telemetry setup, such as the one depicted on the right side of the figure, the system is typically designed around a centralized architecture. This model relies on the deployment of multiple components that work together to collect, process, and store telemetry data. In the figure, these components are simplified and labeled as the ‘‘Telemetry Platform’’. At the core of this setup is the instrumentation within each of the microservices, integrated alongside the application code. This instrumentation, typically based on OpenTelemetry SDK, generates telemetry data, which can include traces, metrics, and logs that provide insights into system performance and behavior. Once generated, these data are sent to the centralized telemetry platform. One challenge with traditional setups like Jaeger is the need for a networked infrastructure where all microservices must connect to a central collection point. This often results in higher energy consumption and network overhead, as telemetry data must travel across the network to reach the centralized back-end. Moreover, this is a simplified view; a traditional telemetry process typically involves other services like collector agents that aggregate, filter, and forward telemetry data; Storage backends, that store the telemetry data; and data visualization services, which are necessary to provide users with an interface to query, analyze, and visualize the telemetry data. Each of these components (i.e. collector agents, storage backends, and data visualization services) can consume significant resources. In contrast, OAS-Telemetry introduces a fundamentally different approach by embedding the telemetry framework directly within each microservice. As shown on the left side of the figure, OAS-Telemetry operates within the same process as the microservice, storing data locally in in-memory storage (i.e., NeDB, a lightweight database based on the NodeJS ecosystem). The key advantage of this approach is that telemetry data are stored locally for each microservice, eliminating the need for separate components, which could increase resource usage. One of the key advantages of this framework is its fault tolerance. The infrastructure does not need to be connected to a centralized data collection system continuously with the consequent risk of a single point of failure. Instead, all nodes function independently (in line with the microservice paradigm). While the operator has access to each microservice, they can view the telemetry data directly. Additionally, thanks to the telemetry endpoint provided by OAS-Telemetry, there is the possibility of accessing telemetry data from other interconnected microservice nodes, even if these nodes are not externally connected. This design offers further benefits. By storing data locally in memory, the framework reduces transmission costs, as there is no need for network communication to transmit telemetry data to an external service. 3.3. Setup OAS Telemetry framework is available in the global npm registry and can be easily installed in the project with the standard package manager npm in NodeJS: npm install @oas-tools/oas-telemetry Once installed, the only necessary addition to the microservice code is a single line. For CommonJS type modules: let oasTelemetry = require(’@oas-tools/oas-telemetry’); or the following equivalent line for ES6 Modules: import oasTelemetry from ’@oas-tools/oas-telemetry’; 3.4. Telemetry UI The telemetry user interface (UI) is accessed within the Express application by navigating to the /telemetry endpoint, providing a convenient interface for viewing telemetry data and monitoring application performance (see Figs. 3and 4). Specifically, the UI presents a table that displays telemetry data in a simple format, providing key details such as the number of spans per path and the average duration. At the top of the page, users can access a panel with three main functions: toggling telemetry data collection (to reduce memory usage), resetting telemetry data, and stopping table updates for inspection. Additionally, for each endpoint, users can manually refresh the data by clicking a button, which retrieves the latest data from the database and updates the table. For deeper analysis, users can navigate to the ‘‘Details’’ section by clicking on an endpoint name, where a list of all spans is displayed for further inspection. Sustainable Computing: Informatics and Systems 46 (2025) 101100 4 M. Otero et al. Fig. 3. UI of OAS telemetry. Fig. 4. UI of traces detail. 3.5. Extensibility The extensibility of the proposed telemetry framework is a critical aspect that enables it to dynamically adapt to the evolving needs of distributed microservice environments. Through a plugin system, developers can extend its core functionality without modifying the underlying system, enhancing its flexibility and capability to perform a wide range of tasks. In this section, we describe the extensibility architecture, focusing on the plugin framework, and explore its potential applications for advanced telemetry scenarios. One of the key features of the proposed framework is the ability to dynamically load and configure plugins through simple HTTP requests to the telemetry API. This allows for real-time adaptation and customization without the need to restart or reconfigure the entire telemetry stack. As illustrated in Fig. 5, a typical scenario unfolds as follows: 1. A plugin is requested and loaded via an HTTP request to the telemetry API. 2. The telemetry middleware dynamically incorporates the plugin into its processing pipeline, allowing the plugin to intercept and analyze all traces as they flow through the system. 3. The example plugin (for anomaly detection9) collects response time traces from various microservice endpoints. It defines a behavioral window of acceptable response times based on historical data and automatically raises alerts in a Telegram channel if it detects an abnormal response time. In this plugin example, a simple outlier mechanism is developed, but more complex 9https://bit.ly/ot-anomaly-alert-plugin. techniques of anomaly detection [25] could be integrated in the same fashion. This example showcases the lightweight and configurable nature of the framework. By allowing external plugins to define and implement custom behaviors, the system can provide more tailored and proactive insights into microservice performance. To ensure consistency and interoperability, each plug-in within the telemetry framework adheres to a standardized structure. A plugin is encapsulated in a single JavaScript file that implements an interface with two key methods: (i) the load method, which is invoked when the plugin is first loaded, initializes the plugin and configures any parameters needed to perform its function. For example, the plugin may define threshold parameters for acceptable response times or specify how alerts should be generated and sent; and (ii) newTrace, which is called each time a new trace is processed by the telemetry system. The plugin can analyze the trace, apply any necessary computations, and take appropriate actions (such as logging data, triggering alerts, or adjusting monitoring thresholds). The plugin structure enables a direct integration into the telemetry framework while maintaining flexibility for developers to implement custom logic and behaviors. This design also ensures that the plugins can be reused in multiple microservices and adapted as the requirements of the telemetry evolve. The plugin system enables several advanced telemetry extensions that go beyond standard monitoring and alerting. In the following, we discuss three potential key directions in which the extensibility mechanism can be used to enhance system analysis and performance optimization. •Dynamic Metric Computation: In dynamic microservice environments, where system behavior can change rapidly, the ability to compute more sophisticated metrics on demand is essential. For example, when performance deviates from the norm, the telemetry system can activate plugins that use advanced techniques such as polling or statistical analysis to gather more detailed metrics over time. This allows for a more granular understanding of performance bottlenecks, especially during asynchronous or high-latency operations, and improves the precision of data collection during critical periods. •Cascading Alerts and Preemptive Measures: The plugin framework supports the ability to issue cascading alerts across multiple microservices when early signs of system degradation are detected. For example, if a plugin identifies a slowdown in a critical upstream service, it could trigger alerts to downstream services, prompting them to initiate fallback mechanisms or circuit breakers to prevent a potential system-wide failure. This preemptive alerting strategy enhances system resilience and helps mitigate the impact of failures before they spread across the entire infrastructure. Sustainable Computing: Informatics and Systems 46 (2025) 101100 5 M. Otero et al. Fig. 5. Telemetry plugin showcase. •Configurable Trace Feeds: Another area of future development is the dynamic configuration of trace feeds to centralized repositories such as OpenTelemetry collectors. By incorporating real-time analysis into the telemetry framework, plugins could selectively forward only the most relevant traces to these repositories for deeper analysis. This feature would reduce the load on storage and processing resources while ensuring that critical telemetry data is available when needed for performance tuning or troubleshooting. Plugins could also adapt their behavior based on real-time conditions, further optimizing resource usage and supporting efficient data collection strategies. Through these applications, the extensibility capabilities of the proposed framework open up new possibilities for intelligent, adaptive telemetry in distributed microservice environments. 4. Validation Given the critical role of performance in telemetry, we conducted a study to evaluate the impact of the OAS Telemetry package on Express microservice performance. We measured memory usage and request processing time in both a sample API microservice and a production environment, with and without the telemetry middleware enabled. In addition, we compared our package with an open source industrial telemetry solution with respect to the same metrics. All test files, sample applications, and results are available in the test folder of the OAS Telemetry repository.10 The test files include 10 https://github.com/oas-tools/oas-telemetry. a sample Express microservice (ks-api) and a test script (test.js) that measures the performance impact of the OAS Telemetry package. Furthermore, due to the complexity of validating in a production environment, where requests need to be spaced out to avoid server saturation, and considering that the server may not accept requests for several seconds due to configuration tasks, the tests were conducted in an ad hoc testing package. Specifically, these validation tests were performed in the context of the production environment Bluejay11 which consists of a microservice architecture that provides a development team practice audit platform [26]. Among the different components of the Bluejay architecture, validation tests were developed in its most critical microservice called Registry. 4.1. Methodology The following subsections describe the methodology employed to validate the performance and effectiveness of OAS Telemetry across three distinct scenarios. These include testing the performance impact of OAS Telemetry on a synthetic microservice application, evaluating its behavior within a real-world production environment, and comparing the performance of our telemetry solution against Jaeger, a widely adopted telemetry platform created by Uber Technologies. For each validation scenario, we compare the performance of the instrumented version of the application, which integrates OAS Telemetry, with the original uninstrumented version. This comparison is performed through a series of test cases (TC) designed to assess the impact on various performance metrics, particularly focusing on response time, 11 https://docs.bluejay.governify.io/. Sustainable Computing: Informatics and Systems 46 (2025) 101100 6 M. Otero et al. memory usage, and CPU usage. The tests are designed to encompass different orders of magnitude in terms of response times when executing the problems, reflecting the diversity of use cases and the range of latencies typically encountered in microservice architectures. By evaluating performance across these varied scenarios, we aim to demonstrate the robustness of our framework and its ability to handle a wide spectrum of workload intensities. 4.1.1. Synthetic microservice telemetry performance validation To measure the performance impact, we use a modified version of KS-API, an Express.js app that simulates a Knapsack problem solving API. The application includes a simple API with two endpoints: /api/v1/stress and /api/v1/problems. The /api/v1/ stress endpoint generates and solves a knapsack problem with a specified number of items and a maximum weight. In order to measure the memory usage, the index.js file of the ks-api application was modified to use the standard NodeJS module ‘‘v8’’ that provides heapStats logging; specifically, it provides information about the memory usage of the Node.js application, including the total heap size, used heap size, and heap size limit. In addition, a new version of the index.js file was created to include the OAS Telemetry middleware (including all code in the original index.js file plus the OAS Telemetry middleware); this duality allows the potential execution of two instances of the microservice: one with telemetry (TLM) and another without telemetry (NoTLM). Moreover, to test different scenarios, we vary the API load in different orders of magnitude in response time to observe its impact on performance within the KS-API. With these variability points, we have the following two configuration dimensions: (1) Telemetry vs. No Telemetry with the following options: •TLM: This configuration involves using the OAS Telemetry middleware in the KS-API calling indexTelemetry.js. •NoTLM: This configuration excludes the OAS Telemetry middleware calling index.js. (2) API Loads within the KS-API with the following options. There will be 3 different orders of magnitude in the problem size sent to ks-api: •ONES (ones of milliseconds): This load involves sending the problem sizes to the /api/v1/stress endpoint, resulting in response times ranging from 1 to 10 ms. Previous tests have determined the problem size for this category as 13. Consequently, calling the /api/v1/stress/13/13 endpoint, representing a problem size of 13, will result in a response time between 1 and 10 ms on the testing machine. •TENS (tens of milliseconds): Here, problem sizes sent to /api/v1/stress yield response times between 10 and 100 ms. Previous tests have determined the problem size for this category as 6000 (/api/v1/stress/6000/6000). •HUNDREDS (hundreds of milliseconds): Problem sizes sent to /api/v1/stress lead to response times ranging from 100 to 1000 ms. Previous tests have determined the problem size for this category as 80000 (/api/v1/stress/80000/80000). In addition, other parameters are kept constant during the testing process to ensure that results can be compared accurately. These parameters include setting concurrentUsers to 1 to focus solely on one request at a time, maintaining a fixed count of 200 for requests to ensure workload consistency, and regulating the pace of request generation with a steady requestDelay of 100 ms. The reason for using only one concurrent user is rooted in NodeJS single-threaded architecture. Since Node.js processes requests in a single thread, introducing more than one concurrent user would result in requests being queued. This queueing could introduce variability in response time and performance measurements, thus obscuring the impact of changes to the application or environment under test. Based on these dimensions of variability, we measure the impact on performance under different conditions and durations by defining the following specific test cases: •TC-01 — Short duration with different loads: This test aims to measure the memory usage and processing time of requests with and without the OAS Telemetry middleware enabled. It includes: –15 iterations with telemetry enabled (TLM) (5 per problem size order of magnitude: ONES, TENS, HUNDREDS). –15 iterations with telemetry disabled (NO_TLM) (5 per problem size order of magnitude: ONES, TENS, HUNDREDS). •TC-02 — Intermittent telemetry: This test measures the memory usage and processing time of requests when the OAS Telemetry middleware is running, then stopped, and finally started again. It involves: –15 iterations (5 for each order of magnitude: ONES, TENS, HUNDREDS). For each iteration, the following phases are executed: 1. Phase I: 66 requests with telemetry enabled (TLM). 2. Request to stop telemetry (spans are no longer exported). 3. Phase II: 66 requests with telemetry stopped. 4. Request to start telemetry again. 5. Phase III: 66 requests with telemetry started again (TLM). •TC-03 — Long running with different loads: This test evaluates the memory usage and processing time of requests with and without OAS Telemetry middleware enabled for a longer duration of 30 min. It includes: –15 iterations of telemetry enabled (TLM) for 30 min (5 for each order of magnitude: ONES, TENS, HUNDREDS). –15 iterations of telemetry disabled (NO_TLM) for 30 min (5 for each order of magnitude: ONES, TENS, HUNDREDS). 4.1.2. Realistic microservice telemetry performance validation To measure the impact of performance in a production environment, we use the registry microservice, a critical component of the Bluejay architecture [27] responsible for the computation and storage of Team Practice Agreements (TPA). The registry microservice interacts with the collector microservice, which gathers relevant pieces of evidence from GitHub and sends them back to the registry so it can evaluate whether each team member is following the practices described in the TPA. A TPA consists of metrics and guarantees. In the context of Bluejay, it measures developers’ adherence to good practices. For example, a development team practice (or TP) may analyze the number of pull requests reviewed by a team member, with the corresponding guarantee that the user must review at least four pull requests per month. See12 for examples of TPAs. For the purpose of testing, we created a specific version of the microservice which can be pulled using the following command: docker pull governify/registry:env-telemetry This version includes a telemetry package that can be activated or deactivated based on the value of the environment variable REGISTRY_TELEMETRY. When the variable is set to true, telemetry 12 https://github.com/governify/zoo/tree/main/bluejay/tpa. Sustainable Computing: Informatics and Systems 46 (2025) 101100 7 M. Otero et al. is enabled, allowing us to monitor and collect relevant performance data. In contrast, setting it to false disables telemetry, facilitating the baseline performance test without the additional overhead of monitoring. In order to improve control over execution times and request management, an ad hoc module,13 was developed for testing. This module performs tasks similar to those performed for testing the ks-api application but offers better control over execution times and the rate at which requests are sent. The package includes a dedicated module that manages the execution of requests using the apipecker tool. Additionally, the memory and CPU usage of the registry microservice are measured by a separate module that interfaces with Docker’s Stats API. The combination of these modules allows for precise performance monitoring in production environments while ensuring requests are spaced appropriately to avoid overloading the server. Using the modules described above, we can effectively monitor how the system behaves while calculating TPA compliance. The complexity of this calculation depends on several factors: 1. The number of guarantees within the TPA. 2. The number of metrics per guarantee. 3. The number of users included in the agreement (typically 1 to 5 users). 4. The window type: Guarantees can be calculated for a specific point based on data from the previous hour, day, week, or month. 5. The period size: The range (start-to-end) over which all data points are calculated, determined by the window type (hourly, daily, weekly, or monthly). To ensure accurate comparisons across all tests, we maintain certain constants: The TPA will always involve one user, with no variations in the guarantees, metrics, or window type. To evaluate the impact on performance, we will perform tests that vary the period size, directly affecting the time required for TPA calculations. Specifically, we will test period durations in hours (1, 24, and 48). These variations will allow us to observe performance impacts in the Bluejay environment. From these points of variability, we define two configuration dimensions: (1) Telemetry in Application vs. No Telemetry, with the following options: •Telemetry Enabled (TLM): This configuration enables telemetry within the registry microservice to monitor the performance of TPA calculations. •Telemetry Disabled (NoTLM): This configuration disables telemetry, measuring baseline performance without any monitoring overhead. (2) TPA Period Durations, with the following options: We will test three different period durations, corresponding to different orders of magnitude in response time: •SMALL (1 h): This load calculates TPA compliance for a 1-h period. Previous tests show response times ranging from 1 to 3 s. •MEDIUM (24 h): The period duration is set to 1 day, with expected response times between 4 and 6 s. •LARGE (48 h): The period duration extends to 2 days, generally resulting in response times between 6 and 10 s. 13 https://github.com/oas-tools/oas-telemetry-tester. The registry cannot handle parallel requests, and the collector service stops accepting requests every minute between seconds hh:mm:50 and hh:mm+1:05, with a margin of error. Therefore, all test requests will be sent sequentially within the available 45 s (starting at hh:mm:05). The delay between requests will be calculated as: delay =⌊45000 estimated-max-response-time⌋ For each order of magnitude: •SMALL (1 h): Assuming a max response time of 3 s, the delay would be floor(45/3) =floor(15.0), allowing 15 requests within the minute/iteration. •MEDIUM (24 h): With a max response time of 6 s, the delay would be floor(45/6) =floor(7.5), allowing 7 requests. •LARGE (48 h): Assuming a max response time of 10 s, the delay would be floor(45/10) =floor(4.5), allowing 4 requests. These tests will help determine the system’s performance under different load conditions within the Bluejay architecture. Based on these configuration dimensions, we define the following specific test cases, similar to the ks-api ones: •TC-01 — Short duration with different loads: This test measures response times, Docker CPU, and memory usage percentage for variants with telemetry enabled (TLM) and telemetry disabled (NoTLM) for each order of magnitude. It involves: –9 iterations with telemetry enabled (TLM) (3 per order of magnitude: SMALL, MEDIUM, LARGE). –9 iterations with telemetry disabled (NoTLM) (3 per order of magnitude: SMALL, MEDIUM, LARGE). •TC-02 — Intermittent telemetry: This test measures Docker CPU and memory usage when telemetry is first running, then stopped, and finally restarted. It involves: –9 iterations of 3 min (3 for each period duration: SMALL, MEDIUM, LARGE). For each iteration, the following phases are executed: 1. Phase I: 1 min sending requests with telemetry enabled (TLM). 2. Request to stop telemetry (spans are no longer exported). 3. Phase II: 1 min sending requests with telemetry stopped. 4. Request to start telemetry again. 5. Phase III: 1 min sending requests with telemetry restarted (TLM). •TC-03 — Long running with different loads: This test evaluates docker CPU and memory usage over a longer duration with (TLM) and without (NoTLM) telemetry enabled. It includes: –3 iterations of 28–30 min telemetry enabled (TLM) (1 for each period duration: SMALL, MEDIUM, LARGE). –3 iterations of 28–30 min telemetry disabled (NoTLM) (1 for each period duration: SMALL, MEDIUM, LARGE). 4.1.3. Jaeger performance comparative In addition to the validation cases described above, we include an additional experiment to compare the performance of our solution with a widely used telemetry platform in the industry, namely Jaeger.14 14 https://www.jaegertracing.io/. Sustainable Computing: Informatics and Systems 46 (2025) 101100 8 M. Otero et al. Jaeger is an open source project under the Cloud Native Computing Foundation that provides a telemetry platform for distributed systems with a special focus on microservice architectures. It also uses the Open Telemetry standard under the hood, though traces are centrally collected and analyzed. Thus, we design a validation case to compare the performance of our totally distributed telemetry approach with an existing telemetry platform where some components are centralized. In order to carry out this comparison, we replicated the TC-01 test case by adding another 15 iterations where we included the Jaeger instrumentation in the ks-api instead of OAS-Telemetry, calling indexJaeger.js. In addition, Jaeger needs a collector component to manage the exported telemetry. The Jaeger ‘‘all-in-one’’ system provides a convenient, singlecontainer deployment that includes several key components necessary for tracing and monitoring: the traces collector, which receives trace data from the exporter and writes it into the storage backend, and the query service, which enables the querying and visualization of trace data. To be fair and ensure consistency with our OAS-Telemetry tests, the Jaeger system utilized in-memory storage instead of an external storage back-end, mirroring the architecture of the OAS-Telemetry implementation. To facilitate these tests, we created a flexible Docker image within the test folder of the OAS-Telemetry GitHub repository. This image includes multiple files essential for comparing performance across different telemetry implementations. Each file serves a specific purpose in the testing framework: 1. indexSelector.js: A dynamic implementation selector controlled by an environment variable. This file enables seamless switching between different telemetry implementations during different test executions, ensuring that all tests are conducted under consistent conditions. 2. index.js: The original implementation of the previously tested ks-api, used as a baseline for comparison. 3. indexTelemetry.js: An enhanced version of ks-api instrumented with the OAS-Telemetry exporter, imported directly from the npm package. This implementation uses in-memory storage for trace data, providing a lightweight telemetry solution. 4. indexJaeger.js: A version of the ks-api integrated with the Jaeger OpenTelemetry exporter. This implementation exports the traces to the Jaeger ‘‘all-in-one’’ service. This setup allows for detailed comparisons of response times, resource usage, and overall system performance across the original implementation, the OAS-Telemetry-enhanced version, and the Jaegerinstrumented version, all within a controlled and reproducible environment. With respect to energy efficiency, we estimate the reductions in energy consumption based on Jaeger’s test CPU usage results. By analyzing both the CPU consumption of TC-01 and the CPU specifications, particularly the Thermal Design Power (TDP), we calculated the expected energy savings in terms of CPU power input. In such a context, it is important to highlight that this analysis represents an initial baseline for energy savings that leaves out important elements such as I/O operations on storage devices or Network Usage, which could point to even bigger savings. 4.1.4. Scenario testing process For each of the previous three scenarios, we carry out the same testing process described in the following. Server initialization. Initially, the server (TLM or NO_TLM depending on the configuration) is initiated to create the environment for conducting the tests. Then a single request is made to verify the server connection. This step ensures that the system is ready to process requests. In situations that require complementary telemetry services, such as the Jaeger comparison, additional components — like the Jaeger ‘‘all-in-one’’ service — are also initialized. Fig. 6. TC-01: Relative change in response time. Test instance execution. During each test instance of a Test Case, specific procedures are undertaken to capture and analyze relevant data points: •Measurement of Node heap statistics: The first step involves measuring Node.js heap statistics (heapStats) to assess the system’s memory usage. •Measurement of Docker CPU and Memory: For Docker containers, such as the Bluejay registry microservice, the Docker Stats API is used to record CPU and memory usage of the container. •Execution of Requests: Following the heapStats measurement, the test instance utilizes the apipecker tool15 to generate a predetermined number of requests directed towards the URL where the ks-api application is actively running. •Reassessment of Node heap statistics: After completing the request cycle, the test instance measures the Node.js heap statistics again to track any changes in memory usage. •Data logging: The collected data, including memory before, response statistics, memory after, CPU usage, and Docker memory usage, are logged into a CSV file for further analysis and review. Server termination. Once the test instance completes its execution, the server is terminated to complete the iteration. This ensures a clean state for subsequent tests and maintains the integrity of the testing environment. In situations that require complementary telemetry services, such as the Jaeger comparison, additional components (such as the Jaeger ‘‘all-in-one’’ service) are also terminated. 4.2. Validation results In this subsection, we detail the analysis of the impact of telemetry on microservice performance, focusing on response times, CPU usage, and memory usage across the different test cases (TC-01, TC-02, TC-03) described above. 4.2.1. Synthetic microservice telemetry performance results In this scenario, we evaluate the performance of the telemetry framework over a synthetic microservice (the KS-API) in the three different test cases. TC-01 — Short duration with different loads 15 https://github.com/pafmon/apipecker. Sustainable Computing: Informatics and Systems 46 (2025) 101100 9 M. Otero et al. Fig. 15. TC-01; Synthetic environment: Response times and Memory Consumption. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Sustainable Computing: Informatics and Systems 46 (2025) 101100 16 M. Otero et al. two solutions is expected, as both packages utilize OpenTelemetry for trace export. Overall, these findings demonstrate that our in-memory exporter implementation introduces no significant overhead. Memory usage.In Fig. 15(b), memory consumption is analyzed across three problem difficulty levels. This figure presents three distinct graphs, each corresponding to a different problem difficulty. The graphs compare memory consumption in megabytes (MB) against the timestamp in seconds. It is important to note that the 𝑌-axis in these graphs represents memory usage in megabytes rather than percentages, as seen in the previous test results. This distinction arises because it would not be appropriate to combine memory percentages, which are calculated per container. Therefore, the actual memory usage in megabytes is depicted to provide a fair comparison. The figure contains three key lines for each telemetry version: a black dashed line for the original KS-API application (baseline), a green dot-dashed line for the KS-API instrumented with Jaeger telemetry, and a red solid line for the KS-API instrumented with OAS-telemetry. Notably, there is a significant distinction regarding the Jaeger instrumentation. Unlike OAS-telemetry, which stores telemetry data in an in-memory database, Jaeger necessitates an external service to collect the data (all-in-one container), depicted by the dashed gray line with a light gray background. The green line, representing the total memory consumption for the Jaeger ecosystem, is the aggregate of the memory usage from both the all-in-one container (gray background) and the KS-API instrumented with Jaeger (green background). This distinction underscores the additional resource consumption associated with Jaeger’s external service requirement. In the next paragraphs, the results are analyzed for the three problem sizes: small, medium, and large. The comparison includes the original KS-API application referred as ks-api(No TLM) or baseline, the KS-API instrumented with Jaeger telemetry referred as ks-api(JGR), the KS-API instrumented with OAS-telemetry referred as ks-api(OAS-TLM), the Jaeger external service referred as all-in-one(JGR), and the total memory consumption of the Jaeger ecosystem referred as total(JGR). For small-scale problem instances, the memory consumption of the ks-api (OAS-TLM) configuration averages 73.00 MB, representing a 69.93% increase relative to the baseline of 42.96 MB. In comparison, if we only consider instrumentation mechanisms, the ks-api (JGR) configuration exhibits a slightly lower average memory usage of 65.28 MB, which corresponds to a 51.95% increase over the baseline. However, the external Jaeger service (all-in-one (JGR)) contributes an average memory consumption of 79.61 MB. When the memory usage of both the ks-api (JGR) and the external Jaeger service (all-in-one (JGR)) is aggregated, the total memory consumption rises to 144.89 MB, reflecting a substantial 237.25% increase by Jaeger services compared to the baseline. For medium-scale problem instances, the ks-api (OAS-TLM) configuration demonstrates an average memory usage of 92.45 MB, which constitutes a 47.98% increase over the baseline value of 62.47 MB. The ks-api (JGR) configuration consumes 77.96 MB on average, representing a 24.79% increase compared to the baseline. The external Jaeger service (all-in-one (JGR)) further contributes an average memory consumption of 73.96 MB. When the memory consumption of both the ks-api (JGR) and the external service (all-in-one (JGR)) is summed, the total memory usage amounts to 151.92 MB, which indicates a significant 143.17% increase relative to the baseline. For large-scale problem instances, the ks-api (OAS-TLM) configuration exhibits an average memory consumption of 117.73 MB, a 20.82% increase over the baseline of 97.44 MB. The ks-api (JGR) configuration requires slightly more memory, with an average of 125.57 MB, reflecting a 28.87% increase relative to the baseline. Additionally, the external Jaeger service (all-in-one (JGR)) consumes 78.52 MB on average. The combined memory consumption of both the ks-api (JGR) and the external Jaeger service (all-in-one (JGR)) results in a total Table 11 CPU usage percent and relative change to baseline in Jaeger comparison tests. Order Microservice Mean CPU usage (%) Relative change to baseline (%) L All-in-one 0.019 – ks-api(JGR) 6.311 4.403 Total(JGR) 6.330 4.722 ks-api(baseline) 6.045 0 ks-api(oas-tlm) 6.296 4.157 M All-in-one 0.019 – ks-api(JGR) 0.522 68.350 Total(JGR) 0.541 74.517 ks-api(baseline) 0.310 0 ks-api(oas-tlm) 0.411 32.750 S All-in-one 0.019 – ks-api(JGR) 0.221 195.176 Total(JGR) 0.239 220.293 ks-api(baseline) 0.075 0 ks-api(oas-tlm) 0.127 70.211 memory usage of 204.09 MB, which corresponds to a notable 109.45% increase when compared to the baseline. Memory consumption data reveals that the incorporation of an external service for telemetry, such as Jaeger, significantly contributes to increased memory consumption. In contrast, the ks-api (OAS-TLM) configuration exhibits a lower memory overhead by utilizing an inmemory database for telemetry storage. This distinction is of particular importance in scenarios where memory resources are constrained, as the additional memory consumption associated with external telemetry services can substantially impact overall system performance. CPU usage.Table 11 presents CPU usage data across different problem sizes: large (L), medium (M), and small (S). The table compares the CPU usage (in %) and the relative change from the baseline for various configurations, including the Jaeger all-in-one external service (all-in-one), the Jaeger instrumentation of ks-api (ks-api(JGR)), the total Jaeger services (instrumentation + all-in-one), the baseline with no telemetry (ks-api(baseline)), and OAS-TLM (ks-api(oas-tlm)) instrumentation. The observations indicate that the CPU usage for the ‘‘all-in-one’’ Jaeger service remains consistently low, as expected, since it is not heavily stressed. However, this service is essential for capturing the total energy consumption of the Jaeger ecosystem. When comparing the CPU usage between Jaeger instrumentation (ks-api(JGR)) and OAS-TLM (ks-api(oas-tlm)), a clear distinction emerges in terms of resource usage across the different problem sizes. In the large size (L), Jaeger instrumentation results in 6.311% CPU usage, a 4.4% increase from the baseline, while OAS-TLM shows slightly lower usage at 6.296%, with a 4.16% increase. This shows that both configurations have a similar impact at the large scale, with Jaeger instrumentation marginally higher. In the medium size (M), the gap widens: Jaeger’s CPU usage increases to 0.522%, a 68.4% rise from the baseline, while OAS-TLM is slightly lower at 0.411%, reflecting a 32.8% increase. This difference becomes more pronounced in the small problem size (S), where Jaeger instrumentation results in 0.221% CPU usage, a 195% increase from the baseline, compared to 0.127% for OAS-TLM, which represents a 70.2% rise. Overall, Jaeger instrumentation tends to use more CPU resources than OAS-TLM across all sizes, with the disparity becoming more significant as the problem size decreases. These results highlight the performance overhead introduced by both telemetry frameworks, with OAS-TLM offering a more CPUefficient solution. The implications for energy consumption will be explored in the discussion section. Energy consumption.The comparison of telemetry solutions revealed an energy savings potential when using our OAS-telemetry framework compared to Jaeger. In order to do this comparison we estimate the energy consumption based on CPU usage results using the Thermal Design Power (TDP) that is provided by manufacturers and represents Sustainable Computing: Informatics and Systems 46 (2025) 101100 17 M. Otero et al. the amount of heat a processor must dissipate under maximum load, typically measured in watts. Although TDP is not directly equivalent to input power, it is closely related, as nearly all the energy consumed by a processor is ultimately converted into heat. The analysis presented is based on the experimental results obtained using a machine with an Intel Core i5-12600KF processor, featuring 16 logical processors (10 physical cores: 4 optimized for energy efficiency and 6 for high performance). Specifically, the processor’s Thermal Design Power (TDP) values, provided by the manufacturer, are as follows: the Base TDP is 125 W, which we will refer to as the Breeze scenario, representing typical energy consumption under standard load. The Turbo TDP is 150 W, referred to as the Inferno scenario, reflecting maximum energy consumption under peak performance. The CPU usage results presented in Table 11 are given as percentages. However, for our calculations, we will use the CPU usage ratio 𝑢𝑖 (rather than the percentage). To convert the percentage values into the ratio, we use the following formula: 𝑢𝑖= 𝑢percent 𝑖 100 Next, to calculate the energy savings in the CPU when using our framework compared to Jaeger, we first need to determine the additional CPU usage introduced by each instrumentation method. To do this, we subtract the original (uninstrumented) CPU usage from the instrumented CPU usage for each method. The formula for this is as follows: 𝛥𝑢𝑖=𝑢instrum 𝑖−𝑢orig where: •𝛥𝑢𝑖represents the additional CPU usage caused by the telemetry instrumentation (such as OAS-TLM or Jaeger) in the microservice. •𝑢instrum 𝑖is the CPU usage ratio of the instrumented application. •𝑢orig is the CPU usage ratio of the original, uninstrumented application. This calculation allows us to quantify the impact of each instrumentation on CPU usage, enabling us to assess the potential energy savings when switching from one framework to another. To estimate the additional power consumption introduced by the instrumentation, we multiply the additional CPU usage ratio by the Thermal Design Power (TDP) of the processor. In our case, we consider two processor scenarios: Breeze (TDP =125 W) and Inferno (TDP =150 W), representing the minimum and maximum power consumption scenarios, respectively. 𝑃add,𝑖 =𝛥𝑢𝑖×TDP𝑠 where 𝑃add,𝑖 is the estimated power added by the instrumentation for microservice 𝑖, and TDP𝑠is the Thermal Design Power (TDP) for the processor in the ‘‘Breeze’’ (minimum) or ‘‘Inferno’’ (maximum) scenario. To calculate the energy consumption over the course of one hour, we multiply the additional power consumption by the time duration (in hours). This gives the energy consumed in watt-hours (Wh), which is a suitable unit for this study since we are focusing on CPU-related energy consumption. Although electrical energy is commonly measured in kilowatt-hours (kWh) in larger-scale power systems, watt-hours (Wh) are more appropriate here, as they directly reflect the increase in energy usage due to CPU overheads. The formula for this calculation is: 𝐸add,𝑖 =𝑃add,𝑖 × 1h where 𝐸add,𝑖 is the estimated extra energy consumed by the instrumentation in one hour for microservice instrumentation 𝑖. The results of applying the above formulas are presented in Table 12. The data show that, for larger problem sizes (orders of magnitude), both frameworks add more power consumption (P+). However, Table 12 Increased CPU and energy consumption due to telemetry instrumentations in ks-api. Order Instrum. Breeze (125 W) Inferno (150 W) P+(W) E+(Wh) P+(W) E+(Wh) LJaeger 0.333 0.333 0.399 0.399 oas-tlm 0.314 0.314 0.377 0.377 MJaeger 0.265 0.265 0.318 0.318 oas-tlm 0.127 0.127 0.152 0.152 SJaeger 0.182 0.182 0.219 0.219 oas-tlm 0.066 0.066 0.079 0.079 Table 13 Total CPU power and energy consumed by Jaeger all-in-one service. Order Breeze (125 W) Inferno (150 W) P (W) E (Wh) P (W) E (Wh) L 0.024 0.024 0.029 0.029 M 0.024 0.024 0.029 0.029 S 0.023 0.023 0.028 0.028 Jaeger consistently uses more power, with the largest differences observed in smaller problem sizes. Specifically, Jaeger’s additional power consumption ranges from 0.182 W to 0.219 W, while OASTLM consumes between 0.066 W and 0.079 W. This indicates that, while Jaeger introduces a higher overhead, the extra CPU energy consumption (E+) difference becomes more pronounced in smaller problem sizes, where the overheads of OAS-TLM are relatively lower. To quantify the energy savings when using OAS-TLM instead of Jaeger, we first calculate the difference in energy consumption between the two telemetry solutions. This difference is given by the following formula: 𝐸diff =𝐸i=jaeger −𝐸i=oas-tlm where 𝐸diff represents the energy saved (in CPU) by using the OASTLM telemetry solution instead of Jaeger. 𝐸OAS-TLM and 𝐸Jaeger refer to the extra energy consumed by the OAS-TLM and Jaeger telemetry solutions, respectively. It is important to consider that Jaeger’s ecosystem requires an additional microservice, the ‘‘all-in-one’’ service. This service plays a critical role by serving as both a collector and a query engine for telemetry data, and it is responsible for storing the collected data. As a result, this additional microservice introduces extra CPU energy consumption, which is not present in the OAS-TLM setup. This added energy consumption must be factored into our overall energy savings calculation when comparing the two telemetry solutions. The total power (P) and energy (E) consumed by the ‘‘all-in-one’’ service in one hour are presented in Table 13. The energy consumed by this service is calculated using the same formula as the energy consumed by the individual microservices. Specifically, the energy consumption of the ‘‘all-in-one’’ service is given by: 𝑃all-in-one =𝑢all-in-one ×TDP𝑠(1) 𝐸all-in-one =𝑃all-in-one × 1h (2) where 𝑢all-in-one is the CPU usage ratio for the ‘‘all-in-one’’ service, and TDP𝑠is the Thermal Design Power of the processor. This additional energy consumption is counted only once in the calculation, as it applies to the entire Jaeger setup, regardless of the number of microservices being instrumented. To better understand the impact of energy savings in real-world scenarios, we calculate the energy savings based on the number of microservices in the application; in fact, the magnitude of this number can greatly vary from dozens to thousands of microservice in certain Sustainable Computing: Informatics and Systems 46 (2025) 101100 18 M. Otero et al. Table 14 Additional CPU energy consumption saved when instrumented with OAS-Telemetry instead of Jaeger. Order Micro-services Breeze (125 W) Inferno (150 W) E saved (Wh) E saved (Wh) L 1 0.043 0.051 20 0.395 0.474 300 5.591 6.710 1000 18.582 22.298 M 1 0.162 0.194 20 2.781 3.337 300 41.380 49.656 1000 137.877 165.453 S 1 0.140 0.168 20 2.359 2.831 300 35.053 42.064 1000 116.789 140.147 scenarios (e.g. Netflix had over 700 microservices back in 201516). Consequently, in our analysis, we compute the energy savings for different numbers of microservices: 1, 20, 300, and 1000. This enables us to capture how the energy savings scale as the number of microservices increases. The total energy savings across all microservices is then calculated using the following formula: 𝐸saved,𝑛 =𝐸diff ×𝑛+𝐸all-in-one where 𝐸saved,𝑛 is the total CPU energy saved by OAS-TLM in watthours for 𝑛microservices, 𝐸diff is the energy difference between Jaeger and OAS-TLM, 𝑛is the number of microservices, and 𝐸all-in-one is the energy consumed by the Jaeger ‘‘all-in-one’’ service. By calculating the energy savings across different microservice scales, we can more effectively compare the energy efficiency of the two telemetry solutions in large-scale, real-world applications. The results in Table 14 demonstrate the energy savings achieved by the OAS-telemetry framework compared to Jaeger across different workloads and processor configurations. The data highlights the potential for significant energy reduction, with variations based on both the number of microservices and the thermal design power (TDP) of the processor. 4.3. Discussion The validation results provide a comprehensive analysis of the impact of telemetry on microservice performance. By examining various test cases (TC-01, TC-02, and TC-03) in different environments (node.js in a Windows pc, docker in a Linux server, and docker in a windows pc) under different load conditions (small, medium, and large), we have been able to assess the effects on response times, CPU usage, and memory consumption. This diverse set of validations ensures that our findings are robust and applicable to a wide range of scenarios. In the following discussion, we will interpret the results, put them in context, and explain their significance. We will highlight key features such as energy efficiency, reliability, and the trade-offs of using a distributed telemetry approach. The synthetic environment tests demonstrated that our distributed telemetry framework has minimal impact on response times, CPU usage, and memory consumption. The results confirmed that our approach is both lightweight and efficient, with response times remaining consistent under varying load conditions. The telemetry system successfully collected data without significantly affecting the performance of the instrumented microservices. Additionally, the framework’s ability 16 https://www.youtube.com/watch?v=-mL3zT1iIKw. to start and stop telemetry collection further highlights its resourceefficient design, minimizing system resource usage when telemetry is not needed. The capability to dynamically adjust monitoring collection provides a significant advantage in resource management. This flexibility opens up opportunities for integrating AI-driven plugins that could automatically adjust data collection methods, similar to the manual adjustments we currently support. Furthermore, the integration of AI could extend to modifying system configurations — such as triggering circuit breakers or automatically scaling Kubernetes resources — to further optimize performance. In our current system, plugins are already running within the instrumented microservice, integrated with the plugin system. This setup allows the plugins to have direct access to the data being collected, enabling real-time analysis and actions. With the integration of AIdriven plugins, these plugins could modify configurations and data collection methods dynamically. This ensures a rapid response to any performance issues, reducing latency and enhancing the system’s overall responsiveness. By enabling direct interaction with the system’s data and configurations, the AI plugins could detect and rectify issues in real time, further optimizing performance without manual intervention. To further validate the efficiency of our framework, we instrumented a real production environment with our telemetry package. The results showed that our approach is capable of monitoring microservices in a real-world setting without introducing significant overhead. The telemetry system was able to collect data effectively, providing valuable insights into the performance of the instrumented microservices. In TC-02, the results (see Figs. 12(b),12(d),12(f)) indicated that no significant changes were observed in memory usage when telemetry was stopped. As mentioned in the results section, this is likely due to the low number of requests, which helps prevent server overload, as the registry manages large, costly data. However, in TC-02 of the synthetic validation, the ks-api received requests every 200 ms and a noticeable difference in memory usage was observed (see Fig. 8(b)) when telemetry collection was halted. The comparison with Jaeger highlighted the advantages of our distributed telemetry approach. Our framework consumes fewer resources than centralized solutions like Jaeger, especially under smaller loads, while maintaining similar performance in terms of response times. This demonstrates the energy efficiency of our approach, making it well-suited for dynamic and distributed microservice architectures. The results show that our improved performance is partly attributed to eliminating the reliance on external services such as Jaeger. Storing data in-memory reduces the need for continuous communication with a centralized system, thereby minimizing latency and network overhead. This approach not only cuts transmission costs but also boosts overall system efficiency. While the advantages are clear, a more precise measurement of this impact would be valuable in future evaluations. One of the trade-offs of our approach is the potential loss of data when a node goes down or when there is a large volume of telemetry data being generated. To address this challenge, we are developing the dynamicExporter — a module that allows our system to function like standard OpenTelemetry instrumentation, with the ability to export data to centralized telemetry services. This exporter will enable the system to decide whether to store data in memory or, when necessary, ‘dump’ it back to a centralized service. This hybrid approach provides the advantage of offering telemetry data close to the source, while still maintaining the functionality of traditional telemetry systems, thus offering greater flexibility to framework users. Another trade-off to consider is the resource consumption associated with running multiple nodes for telemetry data collection. However, telemetry instrumentation is always integrated into existing microservices, and the cost incurred by the telemetry system is primarily the difference in resource usage between instrumented and non-instrumented Sustainable Computing: Informatics and Systems 46 (2025) 101100 19 M. Otero et al. microservices. Furthermore, traditional centralized systems often require additional intermediaries, such as agents or gateways, alongside the backend storage, making our system potentially more efficient. We plan to validate this efficiency in future tests to compare the performance and resource consumption of our distributed model with that of traditional centralized systems. With respect to energy consumption, the results show how OAStelemetry framework consistently delivers energy savings across small, medium, and large workloads when compared with a widely used alternative (Jaeger). Although the absolute energy savings per microservice are smaller for larger problem sizes, the framework achieves important reductions in energy consumption for small to mediumsized workloads. As the number of microservices increases, the overall energy savings grow substantially, further emphasizing the scalability of the framework. As expected, the energy savings are more pronounced under higher TDP conditions (Inferno) compared to the lower TDP (Breeze), highlighting the critical role of hardware efficiency in optimizing overall energy consumption. These findings underscore the advantages of an in-memory telemetry storage solution, particularly in scenarios where energy efficiency is critical. In addition to not requiring an ‘‘all-in-one’’ service, our framework avoids the overhead of deploying additional components such as collector agents, gateways, and backend services. This contributes to a more streamlined architecture and further reduces energy consumption. It is important to note that our analysis focuses solely on CPU usage, and does not account for the additional energy consumption required by the deployment of other services needed in other telemetry frameworks, which also influence the total energy consumed when using telemetry frameworks. While our analysis provides an initial foundation for energy efficiency, it is essential to validate these findings with real-world hardware measurements. Future work will involve implementing a testbed to measure actual power consumption during the execution of telemetry tasks. Such an experimental setup will provide a more precise evaluation of energy efficiency and enable us to refine our framework further. 5. Conclusions This article presents a novel, lightweight distributed telemetry framework specifically tailored for microservice architectures, leveraging the OpenAPI Specification (OAS). Unlike traditional centralized telemetry systems, which often face challenges of complexity and rigidity in distributed environments, our approach utilizes decentralized telemetry data storage. This improves system resilience in the event of network failures and reduces the bottlenecks typically associated with centralized systems. The effectiveness of our approach was validated in three distinct environments and application scenarios. Experimental results show that our distributed telemetry framework performs with minimal overhead compared to uninstrumented applications and other monitoring tools (i.e. Jaeger, a widely used open source alternative). This highlights its potential to enhance observability while maintaining system efficiency and demonstrating advantages in terms of energy savings, particularly for distributed and dynamic microservice architectures. Future work will focus on expanding the framework’s capabilities by enabling distributed access to telemetry data across multiple nodes in the user interface. To complement this, new plugins will be introduced to not only collect data but also enable the framework to adapt in real-time using artificial intelligence, improving anomaly detection and optimizing system performance. Additionally, we will validate the framework in new environments and compare it with different monitoring tools such as OpenZipkin, while measuring additional metrics such as network usage and its related energy consumption. CRediT authorship contribution statement Manuel Otero: Writing – review & editing, Writing – original draft, Software, Formal analysis, Data curation, Conceptualization. José María García: Writing – review & editing, Conceptualization. Pablo Fernandez: Writing – review & editing, Writing – original draft, Supervision, Funding acquisition, Conceptualization. Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Pablo Fernandez reports financial support was provided by Spain Ministry of Science and Innovation. Pablo Fernandez reports financial support was provided by Government of Andalusia. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments This publication is part of the R&D project PID2021-126227NB-C22, funded by MICIU/AEI/10.13039/501100011033/ERDF/EU, Spain and of projects TED2021-131023B-C21 and PDC2022-133521-I00 funded by MICIU/AEI/10.13039/501100011033/European Union NextGenerationEU/PRTR, Spain. Data availability Data will be made available on request. References [1] T. Salah, M. Jamal Zemerly, C.Y. Yeun, M. Al-Qutayri, Y. Al-Hammadi, The evolution of distributed systems towards microservices architecture, in: 2016 11th International Conference for Internet Technology and Secured Transactions, ICITST, IEEE, 2016, http://dx.doi.org/10.1109/ICITST.2016.7856721. [2] S. Hassan, R. Bahsoon, Microservices and their design trade-offs: A self-adaptive roadmap, in: 2016 IEEE International Conference on Services Computing, SCC, 2016, pp. 813–818, http://dx.doi.org/10.1109/SCC.2016.113. [3] F. Montesi, J. Weber, Circuit breakers, discovery, and API gateways in microservices, 2016, arXiv:1609.05830. [4] Y. Verginadis, A review of monitoring probes for cloud computing continuum, in: L. Barolli (Ed.), Advanced Information Networking and Applications, Springer International Publishing, Cham, 2023, pp. 631–643, http://dx.doi.org/10.1007/ 978-3-031-28694-0_59. [5] C. Misa, R. Durairajan, R. Rejaie, W. Willinger, Revisiting network telemetry in COIN: A case for runtime programmability, IEEE Netw. 35 (5) (2021) 14–20, http://dx.doi.org/10.1109/MNET.201.2100064. [6] A. Ferikoglou, A. Kokkinis, D. Danopoulos, I. Oroutzoglou, A. Nanos, S. Karanastasis, M. Sipos, J.F. Ghotbi, J.J. Vegas Olmos, D. Masouros, K. Siozios, The SERRANO platform: Stepping towards seamless application development & deployment in the heterogeneous edge-cloud continuum, in: 2023 Design, Automation & Test in Europe Conference & Exhibition, 2023, pp. 1–4, http: //dx.doi.org/10.23919/DATE56975.2023.10137039. [7] S. Dustdar, V. Casamayor Pujol, P.K. Donta, On distributed computing continuum systems, IEEE Trans. Knowl. Data Eng. 35 (4) (2022) 4092–4105, http://dx.doi. org/10.1109/TKDE.2022.3142856. [8] V. Casamayor Pujol, P.K. Donta, A. Morichetta, I. Murturi, S. Dustdar, Edge intelligence—research opportunities for distributed computing continuum systems, IEEE Internet Comput. 27 (4) (2023) 53–74, http://dx.doi.org/10.1109/ MIC.2023.3284693. [9] S. Laso, J. Berrocal, P. Fernandez, J.M. García, J. Garcia-Alonso, J.M. Murillo, A. Ruiz-Cortés, S. Dustdar, Elastic data analytics for the cloud-to-things continuum, IEEE Internet Comput. 26 (6) (2022) 42–49, http://dx.doi.org/10.1109/MIC. 2021.3138153. [10] V. Casamayor Pujol, A. Morichetta, I. Murturi, P. Kumar Donta, S. Dustdar, Fundamental research challenges for distributed computing continuum systems, Information 14 (3) (2023) 198, http://dx.doi.org/10.3390/info14030198. [11] R. Engel, P. Fernandez, A. Ruiz-Cortes, A. Megahed, J. Ojeda-Perez, SLA-aware operational efficiency in AI-enabled service chains: challenges ahead, Inf. Syst. EBus. Manag. 20 (1) (2022) 199–221, http://dx.doi.org/10.1007/s10257-02200551-w. Sustainable Computing: Informatics and Systems 46 (2025) 101100 20 M. Otero et al. [12] S. Laso, J. Berrocal, P. Fernández, A. Ruiz-Cortés, J.M. Murillo, Perses: A framework for the continuous evaluation of the QoS of distributed mobile applications, Pervasive Mob. Comput. 84 (2022) 101627, http://dx.doi.org/10. 1016/j.pmcj.2022.101627. [13] P. Jamshidi, C. Pahl, N. Mendonça, J. Lewis, S. Tilkov, Microservices: The journey so far and challenges ahead, IEEE Softw. 35 (2018) 24–35, http://dx. doi.org/10.1109/MS.2018.2141039. [14] M. Waseem, P. Liang, M. Shahin, A.D. Salle, G. M’arquez, Design, monitoring, and testing of microservices systems: The practitioners’ perspective, J. Syst. Softw. 182 (2021) 111061, http://dx.doi.org/10.1016/j.jss.2021.111061. [15] J. Mertz, I. Nunes, Tigris: a DSL and framework for monitoring software systems at runtime, 2021, CoRR abs/2103.15986,arXiv:2103.15986. URL https://arxiv. org/abs/2103.15986. [16] A. van Hoorn, J. Waller, W. Hasselbring, Kieker: A framework for application performance monitoring and dynamic software analysis, in: Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering, ICPE ’12, Association for Computing Machinery, New York, NY, USA, 2012, pp. 247–248, http://dx.doi.org/10.1145/2188286.2188326. [17] J. Enes, R.R. Expósito, J. Touriño, BDWatchdog: Real-time monitoring and profiling of big data applications and frameworks, Future Gener. Comput. Syst. 87 (2018) 420–437, http://dx.doi.org/10.1016/j.future.2017.12.068. [18] P. Faymonville, B. Finkbeiner, M. Schledjewski, M. Schwenger, M. Stenger, L. Tentrup, H. Torfah, StreamLAB: Stream-based monitoring of cyber-physical systems, in: I. Dillig, S. Tasiran (Eds.), Computer Aided Verification, Springer International Publishing, Cham, 2019, pp. 421–431, http://dx.doi.org/10.1007/ 978-3-030-25540-4_24. [19] A. Noor, D. Jha, K. Mitra, P. Jayaraman, A. Souza, R. Ranjan, S. Dustdar, A framework for monitoring microservice-oriented cloud applications in heterogeneous virtualization environments, in: 2019 IEEE 12th International Conference on Cloud Computing, CLOUD, 2019, pp. 156–163, http://dx.doi.org/10.1109/ CLOUD.2019.00035. [20] A. Tundo, M. Mobilio, M. Orrù, O. Riganelli, M. Guzmàn, L. Mariani, VARYS: an agnostic model-driven monitoring-as-a-service framework for the cloud, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, in: ESEC/FSE 2019, Association for Computing Machinery, New York, NY, USA, 2019, pp. 1085–1089, http://dx.doi.org/10.1145/3338906.3341185. [21] A. Ullah, H. Dagdeviren, R.C. Ariyattu, J. DesLauriers, T. Kiss, J. Bowden, MiCADO-Edge: Towards an application-level orchestrator for the cloud-to-edge computing continuum, J. Grid Comput. 19 (4) (2021) 47, http://dx.doi.org/10. 1007/s10723-021-09589-5. [22] A. Al Maruf, A. Bakhtin, T. Cerny, D. Taibi, Using microservice telemetry data for system dynamic analysis, in: 2022 IEEE International Conference on Service-Oriented System Engineering, SOSE, 2022, pp. 29–38, http://dx.doi.org/ 10.1109/SOSE55356.2022.00010. [23] H.M. Kabamba, M. Khouzam, M. Dagenais, Vnode: Low-overhead transparent tracing of node.js-based microservice architectures, 2023, http://dx.doi.org/10. 48550/arXiv.2311.11095, ArXiv arXiv:2311.11095. [24] B. Sedlak, V.C. Pujol, P.K. Donta, S. Dustdar, Equilibrium in the computing continuum through active inference, Future Gener. Comput. Syst. 160 (2024) 92–108, http://dx.doi.org/10.1016/j.future.2024.05.056. [25] L. Meng, F. Ji, Y. Sun, T. Wang, Detecting anomalies in microservices with execution trace comparison, Future Gener. Comput. Syst. 116 (2021) 291–301, http://dx.doi.org/10.1016/j.future.2020.10.040. [26] R. Fresno-Aranda, J. Ojeda-Perez, P. Fernandez, A. Ruiz-Cortés, Governify. An agreement-based service governance framework, Softw. Impacts 19 (2024) 100629, http://dx.doi.org/10.1016/j.simpa.2024.100629. [27] C. García, A. Guerrero, J. Zeitsoff, S. Korlakunta, P. Fernandez, A. Fox, A. Ruiz-Cortés, Bluejay: A cross-tooling audit framework for agile software teams, in: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), 2021, pp. 283–288, http://dx.doi.org/10.1109/ICSE-SEET52601.2021.00038. Sustainable Computing: Informatics and Systems 46 (2025) 101100 21