SaaMS: The Synopses-as-a-MicroService Paradigm for Scalable Adaptive Streaming Analytics across the Cloud to Edge Continuum

SaaMS: The Synopses-as-a-MicroService Paradigm for Scalable Adaptive Streaming Analytics across the Cloud to Edge Continuum Georgios Panagiotis Kalfakisa, Nikos Giatrakosa,∗ aSchool of Electrical and Computer Engineering, Technical University of Crete, University Campus, Kounoupidiana, Chania, GR-73100, Greece Abstract The use of data synopses in Big streaming Data analytics can offer 3 types of scalability: (i) horizontal scalability, for scaling with the volume and velocity of Big streaming Data, (ii) vertical scalability, for scaling with the number of processed streams, and (iii) federated scalability, i.e. reducing the communication cost for performing global analytics across a number of geo-distributed data centers or devices in IoT settings. Despite the aforementioned virtues of synopses, no state-of-the-art Big Data framework or IoT platform provides a native API for stream synopses supporting all three types of required scalability. In this work, we fill this gap by introducing a novel system and architectural paradigm, namely Synopses-as-a-MicroService (SaaMS), for both parallel and geo-distributed stream summarization at scale. SaaMS is developed on Apache Kafka and Kafka Streams and can provide all the required types of scalability together with (i) the ability to seamlessly perform adaptive resource allocation with zero downtime for the running analytics and (ii) the ability to run both across powerful computer clusters and Java-enabled IoT devices. Therefore, SaaMS is directly deployable from applications that either operate on powerful clouds or across the cloud to edge continuum. Keywords: Data summarization, Data streams, Big data analytics, Cloud, Edge, Adaptivity 1. Introduction Many modern applications from the financial, maritime, civil protection and other diverse domains base their business value on real-time, online Big Data analytics either at powerful data centers or over a number of, potentially geo-dispersed, devices across the cloud to edge continuum. In the financial domain, extreme scale stock streams, from various markets across the world, stream in investment companies’ data centers. Global and continuous analytics over thousands, rapidly evolving stock streams need to be performed in real-time to timely pinpoint interor intra-market investment opportunities and risks [1]. In turn, such companies provide personalized portfolio management and investment services that span the cloud to fog to edge continuum to allow rapid trading on investors mobile devices. In the maritime domain, thousands of vessels across the globe are being monitored via satellite images, AIS receiver stations at regional coastal areas or unmanned vehicles swarming at sea to enable authorities to detect illegal activities or safety incidents [2] and act accordingly. In civil protection scenarios, drones, robots and sensors of various types operate in forest areas or along river banks. Such devices collect voluminous streams of relevant data in order to monitor environmental conditions and provide early warnings in case of forest fires or floods, among other events. Over the years, there is an established consensus in the streaming data management community [3, 4, 5, 6, 7, 8, 9, 10] that in ∗Corresponding author Email addresses: [email protected] (Georgios Panagiotis Kalfakis), [email protected] (Nikos Giatrakos) order to deal with the volume and velocity of such unbounded streams of data, data stream summaries including sketches [8, 9, 7, 10, 11], samples [6, 12, 12, 13], wavelets[14], histograms[15, 16] and dimensionality reduction techniques [17, 18, 19] can combine the potential to scale the computation by reducing the processing and memory load, while controllably sacrificing the accuracy of streaming analytics tasks, with predefined quality guarantees. Such summaries can provide analytics answers to a variety of commonly used, continuously executed queries that include, but are not limited to, distinct count, cardinality, frequency moment, correlation, set membership or quantile estimation [5]. To deliver Big streaming Data analytics at scale, stream synopses can provide three types of scalability: Horizontal scalability: scaling the computation with the volume and velocity of data streams by reducing the processing and memory load via data summarization. In public or private cloud environments, this property can be combined with parallelization opportunities where each worker is assigned to process a disjoint subset of the incoming streams or a portion of an entire streaming dataset. In that, each worker operates independently on a portion of the incoming load and partial query results per worker can be combined (if needed) based on the mergeability property of many of such summaries [20]. In that, the virtues of both stream summarization and parallelization are exploited. Despite this fact, Big Data frameworks, like Apache Spark [21] or Flink [22], provide no Native API for data stream synopses [23]. Besides, these Big Data platforms can only aid in horizontal scalability at the cloud side as described above, while two additional types of scalability are required for realPreprint submitted to Information Systems November 6, 2025 time data stream processing at the cloud, and beyond the cloud side, as detailed below. Vertical scalability: scaling the computation with the number of processed streams. For instance, in cross-stream correlation (stock-, vessel-, or sensor-streams in the aforementioned scenarios) computation, the complexity of the problem at hand is exponential to the number of processed streams. Therefore, mere parallelism cannot help by itself to reduce the processing load. On the contrary, stream summarization provides a scalable solution. In particular, sketch summaries [18, 24, 25] have been used for correlation/distance-aware hashing of streams to respective processing units. Based on the synopses, using the most significant DFT coefficients or compressed Locality Sensitive Hashing signatures as the hash key respectively, highly uncorrelated/dissimilar streams are hashed to be processed for pairwise comparisons at parallel processing units. Stream correlations are pruned for streams that do not end up nearby in the hashing space by exploiting such localityand similarity-aware hashing schemes. Federated scalability: This type of scalability involves scaling out the computation beyond single computer clusters or clouds to fully geo-distributed settings. Across the cloud to edge continuum, there exist a number of devices and intermediate nodes (sensors, robots, drones, Raspberry Pis) that do possess some processing capacity. Having these nodes naively relaying raw data depletes the available bandwidth and causes network latencies, thus hindering the delivery of real-time, continuous responses to running analytics [26, 27, 28]. By pushing the computation of stream summaries across the cloud to edge continuum, the processing capabilities of the entire network of devices can be exploited and the communication cost, as well as network latency, can be harnessed due to the use of synopses. Despite this fact, no IoT framework provides native support for stream synopses across the cloud to edge continuum [29]. The sole related work in the literature that provides a stateof-the-art parallel, stream summarization engine to support all the required types of scalability is SDEaaS [30, 31]. However, SDEaaS suffers from two inherent drawbacks: (i) it is not easily adaptable to changing stream summary maintenance conditions. This means that changing the parallelism of an SDEaaS service that runs at the cloud incurs significant downtime for scaling out or in (increasing or decreasing parallelism, respectively) upon workload changes and (ii) it cannot be deployed on devices across the cloud to edge continuum. The latter is both due to the fact that SDEaaS is built on a Big Data platform, not destined for resource constrained devices, and because of the downtime adaptation decisions incur for it. In particular, across the cloud to edge continuum, network devices may depart or connect at any given time due to sporadic connectivity or mobility characteristics. If synopses computation is assigned on such devices, every time they enter or leave the network an adaptation of the synopses computation assignment should be performed. Hence, continuous adaptation of the distribution of processing load among the devices is required in such cases. But the downtime that would be incurred by SDEaaS in order to adapt, would dominate the benefits of distributing the processing load among a dynamically changing population of devices. In this work, we introduce a novel paradigm for parallel and geo-distributed stream summarization, SaaMS (Synopsesas-a-MicroService). SaaMS operates across the cloud to edge continuum and deals with all the aforementioned drawbacks. We describe an open-source [32] realization of SaaMS built on Apache Kafka [34] using the Kafka Streams API [35]. At the cloud side, SaaMS retains the ability to provide all the required types of scalability along with zero downtime adaptation to changing workloads. Across the cloud to edge continuum, contrary to prior art, SaaMS is directly deployable on Java-enabled devices incurring zero downtime adaptation upon changes in the populations of available devices which perform stream synopses computation. Therefore, SaaMS can simultaneously scale stream synopses computation at the cloud side and distribute synopses computation across network devices. In that, it fully exploits the lot of the processing capacity of the continuum. Our experimental evaluation using hundreds of real stock and vessel data streams shows that SaaMS (i) scales faster than linearly with increasing stream volumes and velocities (horizontal scalability), (ii) maintains linear scaling trends upon increasing the number of processed streams from 10s to 100s (vertical scalability) and (ii) can save more than two orders of magnitude communication-wise across the cloud to edge continuum compared to naively using network devices at the edge as relay nodes (federated scalability). Compared to the state of the art SDEaaS approach which operates only at the cloud side, SaaMS exhibits an up to 3 times higher average throughput due to the fact that it diminishes downtime. Moreover, this especially happens in cases where the volume, velocity and the number of processed streams is significantly increased. 2. Related Work From a research viewpoint, there is a large number of related works on data synopsis techniques. Please refer to [5, 40, 3] for comprehensive reviews on relevant issues. Such prominent techniques, cited in Table 2, are incorporated in SaaMS, which already includes a rich set of data summaries serving a wide variety of analytics tasks. What is more important is that SaaMS can incorporate any data summarization technique abiding by a simple, yet effective, software architecture (Section 4.3). With respect to data summarization engines and libraries, Table 1 provides a comparison of SaaMS against prior art, regarding their scalability features, the level of the cloud to edge continuum where these scalability features are supported and their potential for adaptivity. Apache DataSketches [36] and Stream-lib [37] are software libraries of stochastic streaming algorithms and summarization techniques. These software libraries are detached from parallelization aspects. Therefore, they cannot provide horizontal, vertical or federated scalability at the cloud side. Moreover, they do not provide the primitives for IoT devices to coordinate and merge their partial synopses within the context of a unified synopses service. In other words, the application should manually install, program, synchronize and coordinate the IoT architecture from scratch. 2 Features →Horizontal Scalability Vertical Scalability Federated Scalability Zero-downtime Adaptivity IoT Level → Related Approach ↓@CLOUD @FOG @EDGE @CLOUD @FOG @EDGE @CLOUD @FOG @EDGE ANY DataSketch [36] ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ Stream-lib [37] ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ StreamApprox [38] ❑ (Stratified Sampling) ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ SnappyData [39] ❑ (Simple Aggregates) ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ Condor [23] ✔✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ SDaaS [30, 31] ✔✘ ✘ ✔✘ ✘ ✔✘ ✘ ✘ SaaMS (this work) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Table 1: Scalability and Zero-downtime Adaptivity of Prior Art across the Cloud to Edge Continuum SnappyData’s [39] stream processing is based on Spark and incorporates a limited set of synopses serving simple SUM,COUNT and AVG queries. Similarly, StreamApprox [38] offers only sampling as a pipeline operator over Spark and Flink. Thus, these are deprived of vertical scalability features and federated scalability provisions at the cloud side. Besides their limited support for synopses techniques (therefore the orange square in Table 1), SnappyData’s and StreamApprox operate on top of Big Data frameworks which are not deployable across the cloud to edge continuum. The prominent work of Condor [23] elegantly optimizes the parallel computation of stream summaries at the cloud side, though neglecting vertical and federated scalability. SDEaaS [31, 30] covers all aspects of scalability at the cloud side only. Since Condor and SDEaaS run on top of Flink, they are restricted at the cloud level. No related technique provides zero-downtime adaptivity at any level of the cloud to edge continuum. Indicatively, since SDEaaS and Condor are developed on Flink, they are running as Flink jobs at the cloud side. Our experience with adapting, even such simple stateless jobs says that it takes 10 to 15 seconds in case of adaptation (scaling in or out). This downtime intervals can considerably increase when state should be stored before and loaded after adaptation. SaaMS also overcomes other limitations of SDEaaS including: (i) SDEaaS, by design, cannot use the native windowing operators of the DataStream API of Flink, while SaaMS incorporates the windowing operator of the Kafka Streams API, thus, offering additional parallelization strategies (Section 4.2), (ii) SaaMS allows to save and load saved synopses to reuse summaries from past microservices, while SDEaaS always starts maintaining synopses from scratch, (iii) Adding code of new synopses on-the-fly, at SDEaaS runtime, is cumbersome due to Java ClassLoader usage, raising potential security issues in YARN-like clusters. On the contrary, adding code of new synopses on-the-fly, at SaaMS runtime, just creates new MicroService instances without the need for ClassLoader usage. 3. Background 3.1. Apache Kafka Apache Kafka [34, 35] constitutes the de-facto standard for stream ingestion in large scale, distributed, streaming applications [41, 42]. Kafka is a fast, scalable, durable, and faulttolerant publish-subscribe messaging system. A Kafka cluster is composed of a number of workers coordinated by a ZooKeeper. Workers are runtime instances ran on Virtual Machines (VMs) of a Kafka cluster undertaking the execution of data ingestion tasks. Kafka receives and sends data using producers and consumers. A producer is a data source that initializes input by sending records to the Kafka log. The log typically consist of <key, value>pairs and, as its name suggests, it operates on an append only fashion. The records that arrive at a Kafka cluster are categorized into topics based on the context the corresponding logs refer to. For instance, based on the running examples of Section 1, a Kafka topic may involve a group of stocks coming from the same market, in the financial domain scenario. The producer in this scenario generates and publishes financial market data, such as stockID (key) along with volume and prices per stock (value), or other relevant market events, into Kafka topics. In the maritime activity monitoring scenario, a Kafka topic may log vessel positions of a specific area of the oceans generating vesselID (key) and longitude, latitude (value) pairs. The actual context stored in the Kafka topics should be determined and defined by the application. Automatically, each record of a topic is assigned a unique ‘offset’, which is a value that determines its position in the log. Kafka topics include one or more partitions for parallel processing purposes. A topic with multiple, non-overlapping partitions can be read (consumed) by different application instances, each reading and processing only a portion of the ingested data. Each partition has a leader instance held at one broker and replicas kept at the same or other brokers. Application instances should be equipped with Kafka consumers which read records using the offset as a pointer to identify where they previously stopped. A main characteristic of Kafka consumers is the organization into groups, allowing the distribution of partitions across the consumers of the same group. For instance, a topic with Xpartitions can have its records distributed among these partitions in a round robin fashion and a group of Xor fewer consumers can operate on a separate subset of partitions each, reading and processing the records in parallel. In this case, a parallelization degree of Xis achieved and the parallelism can increase in case a topic has more partitions. In general, the number of partitions of a Kafka topic sets an upper bound on the allowed processing parallelism by the applications built on top of Kafka. The above 3 Kafka Cluster ZooKeeper Worker 1 Worker 2 Broker 1 Broker 2 Broker 3 Nikkei225 Topic S&P500 Topic NASDAQ Topic AIS Vessel Data Mediterranean Topic Leader Partition 0Leader Partition 0 Replica Partition 0 offset Leader Partition 1 Leader Partition 2 Replica Partition 1 Replica Partition 2 Replica Partition 0 Replica Partition 1 Replica Partition 2 Leader Partition 0 ReplicaPartition 0 ReplicaPartition 1 Replica Partition 0 Replica Partition 1 Leader Partition 1 Replica Partition 0 Replica Partition 1 Replica Partition 0 Replica Partition 1 Leader Partition 0 Leader Partition 1 Figure 1: Kafka Cluster Exemplary Organization. Coloring of topics and partitions expresses correspondence. For instance, grey partitions belong to the grey NASDAQ topic. discussion is depicted in Figure 1. Our SaaMS framework communicates with the outside world, in the context of broader stream processing workflows, via Kafka topics which (i) consume streams and update maintained synopses, (ii) accept (consume) application/user queries and requests, and (ii) deliver (produce) the estimations provided by the maintained synopses as query answers in the output. Details follow in Section 4.1 and Section 5.1. 3.2. The Kafka Streams API The Kafka Streams API of Kafka is a Java library for building real-time streaming microservices that can consume, process, and produce data from and to Kafka topics. It seamlessly integrates with Kafka, allowing stream processing applications to directly consume and produce messages to Kafka topics. This makes it easy to incorporate stream processing into existing Kafka-based architectures. Kafka Streams is a functional programming API that ensures fault tolerance exploiting Kafka metadata automatically created in the background and exactlyonce semantics (each record is processed exactly once even in the face of failures). Kafka Streams applications are Java applications composed of one or more microservices that can run on any Java-enabled device. The developer has just to produce a .jar file and deploy it to the desired devices (cluster VM or device, across the cloud to edge continuum). Separate devices can understand that they are part of the same application since they share the same application.id. The application.id is a unique identifier assigned to a Kafka Streams application, and it plays a crucial role in ensuring that instances of the same application coordinate and work together within the Kafka cluster. A Kafka Streams application is composed of a number of processing and storing entities, besides producers and consumers described in Section 3.1. Below we summarize these entities focusing on their properties that justify the way they are positioned in the SaaMS architecture later on in our discussion (Section 5): •KStream: a high-level abstraction representing an immutable, ordered, and replayable sequence of stream records. Typically, each record is considered as a key-value pair. The term ”KStream” is derived from ”Keyed Stream”. •KTable: represents materialized views of changelog streams. KTables are immutable and maintain only the latest state of a microservice or application data, enabling efficient joining of streams or KTables. KTables are useful when the application needs to keep track of the current state of entities. •State Store: provides mutable storage within a Kafka Streams application. State Stores can be used to maintain intermediate or aggregate state during stream processing. State stores allow updates, inserts, and deletions. •Serdes: the term combines ser(ialization) and des(erialization) needed for transforming data (see stream transformations below) between microservices connected with Kafka topics and the internal representations used by the Kafka Streams application. The different workers (within a cluster) or various devices (across the cloud to edge continuum) need to communicate their transformed streams, the sender first has to serialize the data in a series of bytes so that they can travel via network channels, while the recipient has to deserialize bytes into meaningful streaming records. •Stream Transformations: implement the actual data processing functionality of a number of microservices within Kafka Streams applications. Stream transformations typically are higher order functions taking as input or having a built-in anonymous function along with one or more Kafka Streams or KTables. Below we briefly mention the functionality of the major stream transformations used in the SaaMS architecture: –flatMap: transforms each input record into zero or more output records by applying a one-to-many transformation. –mapValues: transforms the values of a stream without changing the keys. –groupByKey: groups records by their keys for further aggregation or processing. –aggregate: performs stateful aggregations on a (usually grouped) stream, maintaining results over time. –windowedBy: groups records into time-based windows for operations like counting or aggregating over specific time intervals. –join: combines records from a pair of sources based on their keys. For KStream-KStream joins, it matches records with the same key from two KStreams. For KStream-KTable joins, it combines KStream records with the latest record from a KTable based on matching keys. A JoinWindows parameter specifies time windows for temporal matching in both types of joins. –transform: allows the application of custom stateful transformations on each record in a KStream, providing flexibility for complex processing scenarios by maintaining and updating state across multiple records within a transformer. 4 Query(y) → Estimation(1,”yes”) 0 1 0 0 0 0 0 01 1 Query(w) → Estimation(0,”No”) Bloom Filter Current State Figure 2: Bloom Filter example on a current bitmap (synopsis state) with 2 queried elements (y,w) and set membership estimations. 3.3. Running Stream Synopses Examples SaaMS supports a large set of synopses over streaming data (Table 2), but is extensible and customizable to any data summarization algorithm required by an application. In this section we are going to present two synopsis techniques, namely Bloom Fliters [11] and AMS Sketches [43], which will be used as running examples in the following sections. Our focus will be on presenting (i) the way they reduce the processing and memory load and (ii) their update, estimation (querying) and synopses merging procedures. The reason for this, is that (i) and (ii) are at the core of the software technology in SaaMS design (Section 4) and the implementation of the SaaMS architectural framework (Section 5). The Bloom Filter [11] is a lightweight and space-efficient probabilistic algorithm that can provide approximate estimations on set membership queries. The algorithm uses as its structure a bitmap of Mbits with initial values set all to 0. Ldenotes the cardinality of the distinct stream elements (with L≫M, therefore, the memory efficiency), and Kare distinct hash functions used to map incoming stream elements to bitmap positions. Update Procedure and Synopsis State: When a stream element xarrives, it is processed by a total of Khash functions. Each hash function is denoted as hi(x),i=1,2,...,K. The bit positions in the bit array are represented as: h1(x),h2(x),...,hK(x) For each of these bit positions to which the hash functions points, the corresponding bit is set to 1. A bit already set to 1 by a previous stream element remains unchanged. The state of the synopsis at any given time is the bitmap itself with its set and unset bits in each position, as formed by updates received so far. Estimation (Querying): Upon an application query, the algorithm estimates set membership of a queried element yin the set, by computing Khash functions on y. If all the bit positions, resulted from the hash functions, in the bit array are 1, yis in the set as illustrated in Figure 2. On the other hand, if for another element wthere are 0 bits in any of these positions, then wis definitely not in the set (Figure 2). The reason the algorithm does not respond with certainty is the limited size of the bit array compared to the cardinality of the original set of elements. As a result, different stream elements may hash to the same set of positions. The probability +𝑐𝑔1(e) +𝑐𝑔2(e) +𝑐𝑔3(e) +𝑐𝑔𝐿(e) h1(K) hL(K) e, +𝑐 L depth M buckets AMS Sketch Current State Figure 3: AMS Sketch synopsis update and current state. that all Kpositions are 1 for a queried element and this element did not appear in the stream so far is FP =1−e−KL MK. To minimize the probability of a false positive (FP) result (the Bloom Filter erroneously replies that an element is in the set although it has not appeared in the stream so far), the number of hash functions should be set to: K=M Lln 2 Merge Procedure: In case the stream is monitored in parallel by multiple workers or devices across the cloud to edge continuum, each instance of the Bloom Filter keeps a local state (bitmap) built on the subset of stream elements that have arrived locally. Local Bloom Filters can be merged into a global Bloom Filter, retaining the aforementioned properties of the synopsis, by performing a bitwise OR operation on locally constructed Bloom Filters. A Bloom filter requires O(M) memory and O(K) update, query and merge time complexity per parallel instance. The key idea in AMS Sketch [43] is to represent a vector v, holding the frequencies of a large domain of streaming elements, using a much smaller sketch vector sk(v). This vector is updated with the streaming tuples and provides probabilistic guarantees for the quality of the data approximation. Update Procedure and Synopsis State: An AMS sketch is initialized as a matrix Swith Lrows and Mcolumns (depth and buckets in Table 2), where L=O(1/ε2), and M=O(log 1/δ), with ε, 1−δbeing the desired bounds on error and probabilistic confidence, correspondingly. Lis the number of hash functions and Mis the number of buckets. When an element eis encountered in the data stream, it is hashed Ltimes using Lindependent hash functions. Each hash function himaps eto a bucket index j(where 1 ≤j≤M), and the count in the corresponding bucket for each row iis incremented (Figure 3). The AMS sketch defines the i-th sketch entry, sk(v)[i] as the random variable PKv[K]·gi[K], where {gi}is a family of four-wise independent binary random variables uniformly distributed in {−1,+1} (with mutually-independent families across different entries of the sketch). Using appropriate pseudo-random hash functions, each such family can be efficiently constructed on-line in logarithmic space. Note that, by construction, each entry of sk(v) is essentially a randomized linear projection (i.e., an inner product) of the vvector (using the corresponding gfamily), that can be easily maintained (using a simple counter) over the input update stream. Similarly for element deletion – expiration. 5 Listing 1: Example of Streaming Tuple at SaaMS Data Topic. 1{ 2" streamID ":" Tesla Inc .", 3" dataSetKey ":" NASDAQ Stock Market " , 4"date ":"01/02/2019", 5"time ":"00:00:01", 6" price ":6.0654, 7"volume":1 8} The state of the sketch at any given time is the two-dimensional L×Marray along with its count entries. Estimation (Querying): An estimation on the cardinality of the “inner product” between two streams in the sketch-vector space is given by: sk(v1)·sk(v2)=median | {z } j=1,··· ,M        1 L L X i=1 sk(v1)[i,j]·sk(v2)[i,j]       For estimating the second frequency moment of a single stream we simply replace sk(v2) with sk(v1) in the formula above. Merge Procedure: In case the stream is monitored in parallel by multiple workers or devices across the cloud to edge continuum, each instance of the AMS keeps a local L×Mmatrix built on the subset of stream elements that have arrived locally. Local AMS Sketches can be merged into a global sketch, by entry-wise summations of the respective arrays. As mentioned above, AMS requires O(M) memory and exhibits an O(K) update, query and merge time complexity per parallel instance. 4. SaaMS Fundamentals 4.1. SaaMS API SaaMS communicates to the outside world and broader application workflows, where it is deployed, via JSON formatted Kafka messages. JSON is used both to describe the schema of the incoming and outgoing data tuples, and as an API so that it can parse and accept instructions for (i) starting maintaining new synopses, (ii) querying existing synopses, (iii) updating existing synopses, (iv) saving the current state of a synopsis or loading a past synopsis and (v) deleting an existing synopsis. Because of this design choice, SaaMS can be part of any streaming processing workflow irrespectively of other platforms and tools that are deployed in the rest of the processing pipeline. Listing 1 shows an example of a JSON formatted Kafka message which constitutes a streaming tuple destined to update an existing synopsis. We use the stock market scenario described in Section 1 as our running example: •streamID: The name of the stream the tuple belongs to. •dataSetKey: It represents a set of streams. •Field Name(s): Represents the field(s) which may be used to build the synopsis on. In this example: Listing 2: Example of New Synopses Maintenance Request. 1{ 2" streamID ":" Tesla Inc ." , 3" synopsisID ":1, 4" dataSetKey ":" NASDAQ Stock Market ", 5" param ":[" CountMin "," volume "," Construct ",0. 001,0.99,12345], 6" noOfP ":5 7} Listing 3: CountMin [7] Query Example at SaaMS Request Topic. 1{ 2" streamID " :" Tesla Inc ." , 3" synopsisID " :1, 4" dataSetKey " :"NASDAQ Stock Market", 5" param " :[6,"volume"," Queryable "," Continuous "] 6} – date & time: The timestamp at which the tuple (trade of streamID = "Tesla Inc." in the example) was captured. – price: The current price at the specified date and time. – volume: The quantity of a trade for the streamID = "Tesla Inc." at the specified date and time. As another example, in the maritime scenario of Section 1, streamID would correspond to the identifier of a vessel, dataSetKey would correspond to a sea area, while application specific fields like price and volume would be replaced with timestamped vessel coordinates. Listing 2 presents a JSON message with a request for starting maintaining a new synopses. In the example of Listing 2, the creation of a new synopsis is requested with specifications to create a CountMin synopsis on the volume attribute, along with necessary parameters (see Table 2) for that synopsis. In particular: •streamID: As in Listing 1. •synopsisID: Defines the synopsis type as a sequentially increasing integer for each of the synopses listed in Table 2. For instance, synopsisID = 1 corresponds to a CountMin sketch. •dataSetKey: As in Listing 1. •param: An array of parameters necessary to build the requested synopsis: Listing 4: Request Message for Loading a Saved Synopsis. 1{ 2" param " :["LOAD_REQUEST",PathToLoadSynopsis\\ stored_CountMin . ser " ] 3} 6 SynopsisID Synopsis Output Estimation Input Parameters 1 CountMin [7] Count, Frequency Estimation ϵ(maximum error), δprobabilityOfExceeding ϵ, seed 2 HyperLogLog [9] Cardinality, Distinct Count Relative Standard Deviation (RSD) 3 BloomFilter [11] Membership Expected #Elements, False Positive Rate 4 DFT [18] Correlation Estimation Window Size, Slide Size,#coefficients 5 LossyCounting [13] Count, Frequent Items ϵ(maximum error) 6 StickySampling [13] Count, Frequent Items Support, ϵ(maximum error), δProbab. of Exceeding ϵ 7 AMS [43] L2Norm, Inner Product Depth, Buckets 8 GKQuantiles [44] Quantiles ϵ(Maximum Error) 9 LSH [19, 45] Correlation Estimation Window Size, Compression Ratio, #Workers 10 WindowSketch Quantiles [46] Quantiles ϵ(Maximum Error), Window Size Table 2: SaaMS Built-in Synopses, Input Parameters and Output Estimation. SaaMS remains extensible and customizable by plugging in new synopses. – Synopsis Type: Specifies the type of the synopsis (e.g., CountMin) including those in Table 2 or custom synopses plugged in the SaaMS framework. – Field Name: Represents the field used to build the synopsis on (e.g., ”volume”). – Request Status: Describes the type of the request. In Listing 2 it receives the value of Construct to declare a request for start maintaining a new synopsis. – Synopsis Parameter(s): Each synopsis has different parameters as in Table 2. For example, CountMin has ϵ,δ, and seed. These are instantiated in the example of Listing 2. •noOfP: Defines the number of partitions in the synopsis data topics and the maximum parallelization degree of the synopsis microservice, to be further discussed in Section 5.1. If a synopsis already exists, it is not duplicated. In case the streamID is empty, the synopsis will be maintained on all streams with the same dataSetKey. Moreover, the request for deleting a synopsis is similar to that of Listing 2, with Delete replacing the Construct keyword. Listing 3 provides an example of a query on the previously created CountMin synopsis. The difference compared to Listing 2 is on the parameter array that is passed in the request. param is again an array of parameters, but this time includes the parameters that are necessary to query the involved synopsis: •Query Parameters: Specifies the query parameters for the involved synopsis. In the example of Listing 3 we query the CountMin sketch (because synopsisID = 1) for estimating the frequency of trades with a volume of 6. •Field Name: The queried field, volume in the example of Listing 3. •Request Status: Describes the type of the request. In Listing 3 it receives the value of Queryable to declare a request for querying a synopsis. •Query Type: Each query may either be a Continuous query, meaning that it is registered once and gets continuously executed until the synopsis is deleted, or Ad-hoc which involves one shot queries. Finally, Listing 4 expresses a request for loading a previously stored synopsis, from a file of .ser type. This kind of request just uses the path from which the corresponding file should be loaded. Having issued a Construct request as in Listing 2, a LOAD REQUEST follows in order to load a previously saved synopsis, instead of starting maintaining it from scratch. Having loaded a synopsis, it can get updated using tuples similar to the one in Listing 1, as well as it can get queried by receiving requests similar to the one in Listing 3. The request for saving the current state of a synopsis is similar, substituting LOAD REQUEST with SAVE REQUEST. 4.2. SaaMS Parallelization Schemes SaaMS employs, and when needed combines, three parallelization schemes: (i) partition-based parallelization, (ii) keybased parallelization, and (iii) window-based parallelization. Partition-based parallelization refers to the ability to process multiple partitions of a Kakfa topic concurrently. Each partition is an ordered, immutable sequence of tuples, and consumers (workers at the cloud side or devices across the cloud to edge continuum) can read from multiple partitions simultaneously, allowing for parallel processing of data. This type of parallelism is used on synopses defined based on dataSetKey as detailed in Section 4.1. In this case, tuples of the same dataset are distributed among partitions (noOfP parameter in Listing 2). Key-based parallelization, on the other hand, involves grouping records by their keys and processing records with the same key in parallel. This is particularly useful in SaaMS since we have data that may need to be partitioned based on a key, such as the streamID in our running example. These two forms of parallelization are combined, depending on the synopsis type, in SaaMS by ensuring that tuples with the same key are sent to the same partition. By doing so, SaaMS leverages both partition-based parallelization (processing multiple partitions concurrently) and key-based parallelization (processing tuples with the same key in parallel within a partition). 7 ED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED UNREGISTERED «package» Synopses «package» HyperLogLog «package» BloomFilter «package» DFT «package» GKQuantiles «package» AMSSketch «package» LossyCounting LSH «package» StickySampling «package» WindowSketchQuantiles «package» CountMin AMSSketch -depth: int -buckets: int -count: int -counts: int[*] «constructor»+AMSSketch(depth: int, buckets: int) #update(item: Double): boolean #update(item: Double, incrementCount: long): boolean #compareTo(o: AMSSketch): int #estimateCount(item: Double): long #size(): long #estimateF2(): long #estimateInnerProduct(b: AMSSketch): long #update(source: AMSSketch): boolean #subtract(source: AMSSketch): boolean #contains(item: Double): boolean AMSSketchSynopsis +amss: AMSSketch «constructor»+AMSSketchSynopsis(String []) +serde(): Serde<AMSSketchSynopsis> +add(Object): void +estimate(Object): Object {query} +merge(Synopsis): Synopsis +size(): long «package» SynopsesSerdes «package» AMSSketch AMSSketchSerde «constructor»+AMSSketchSerde() +serializer(): Serializer<AMSSketch> +deserializer(): Deserializer<AMSSketch> -readAllBytes(DataInputStream): byte[*] HyperLogLog GKQuantiles WindowSketchQuantiles CountMin BloomFilter LossyCounting AMSSketchSerializer «constructor»+AMSSketchSerializer() +configure(Map<String, ?> , boolean): void +serialize(String, AMSSketch): byte[*] AMSSketchDeserializer «constructor»+AMSSketchDeserializer() +configure(Map<String, ?>, boolean): void +deserialize(String, byte[]): AMSSketch -readAllBytes(DataInputStream): byte[*] StickSamplingLSH «package» DFT «package» «package» «package» «package» «package» «package» «package» «package» «package» Figure 4: SaaMS Library Software Technology (Package Structure). In our running example, if we have a Kafka topic with stock trades from a market (the same dataSetKey) and we partition the topic based on streamID, tuples for the same stock will be sent to the same partition. This allows SaaMS to process data for different stocks in parallel across different partitions, while still processing tuples for the same stock in parallel within a partition, based on their keys. Window-based parallelization, on the other hand, involves dividing data into fixed-size time intervals or windows and processing each window independently. This is particularly useful for handling streaming data where we want to perform operations, such as aggregation or computation, over a defined time window. These forms of parallelization can be combined in SaaMS via the Kafka Streams library. In that, SaaMS allows the definition of windowed operations (Section 3.2), enabling the processing of data in parallel across different partitions within each time window. This boosts the ability of SaaMS for efficient and scalable processing of streaming data, taking advantage of both partition-based and window-based parallelization. Such parallelization flexibility should not be taken for granted. It is based on the facilities provided by Kafka and Kafka Streams, but retaining these parallelization facilities, use and combine parallelization schemes is solely provided due to the unique Synopses-as-a-Microservice paradigm (Section 5.1) introduced by SaaMS. For instance, prior art [31, 30] is built on Flink, which supports window-based parallelization, but the SDEaaS framework built on Flink, fails to incorporate it natively and relies on manually programmed window configurations. 4.3. SaaMS Synopses Library and Software Technology The SaaMS library is built using a simple, yet effective software technology that not only allows the incorporation of a large set of stream synopsis techniques (Table 2), but, importantly, remains extensible to new synopses and easily customizable to any application specific needs. This is in contrast to other large libraries listed in Table 1, which are extremely complex and hard to maintain and extend in the long run [42]. SaaMS avoids by design this complexity via two means. First, the design choice of using JSON formatted, schema-aware Kafka messages in its input/output. This alleviates the burden of incorporating schema details in the SaaMS software technology itself. This is in contrast to other approaches which define a separate class for any synopsis specific data type. As an example, DataSketches [36] defines a separate Java interface and classes for sketches of floats, sketches of doubles and so on. Second, the SaaMS library is built based on the observation that synopsis techniques rarely are complex algorithms themselves. Stream summarization algorithms fall into the category of clever, small and elegant probabilistic mechanisms that share the following fundamental operations (i) update a synopsis with a new tuple (add operation), (ii) provide an estimation upon getting queried (estimate operation), (iii) perform a merge operation in case a synopsis is maintained in a distributed setting, provided the synopsis itself possesses the mergeability property [20]. Figure 4 and Figure 5 provide a concise view of the SaaMS Synopses Library software technology. In the Synopses package (top of Figure 4) there is the abstract Synopsis class (middle of Figure 5). The member variables of the Synopsis class are the synopsisID and the synopsisParameters explained in Section 4.1. In addition to these, the synopsisDetails 8 Figure 5: SaaMS zooming in the Class Diagram. member variable holds important information about the synopsis, including the key of the Construct Request and noOfP (Section 4.1). The abstract Synopsis class (Figure 5) provides setters and getters for these member variables and requires for every subclass that inherits from it to implement the fundamental operations for each synopsis, as previously mentioned, i.e., the add, estimate,merge operations. Moreover, there are two additional methods, size and serde, for serialization and deserialization purposes (Section 3.2) upon operating in a distributed/- parallel (within the cloud) or geo-distributed (across the cloud to edge continuum) setting. Every specific synopsis technique, such as those cited in Table 2, inherits from the Synopsis abstract class and implements add,estimate,merge,size and serde as illustrated in Figure 5. It is important to note that all classes that have the ...Synopsis suffix, arrange the details of the distributed/parallel maintenance and querying of the respective synopsis over SaaMS parallel architecture (Section 5.1). Returning to Figure 4, around the Synopses package, there are separate packages for each specific synopsis technique. Figure 4 zooms in the AMSSketch package which includes 2 classes. The AMSSketchSynopsis which, as we already analyzed, inherits from the abstract class Synopsis in the Synopses package and takes care of the distributed/parallel synopsis over the SaaMS architecture. The second class is the AMSSketch class which is a simple .java file with the code of the respective (the AMSSKetch in this example) synopsis. This class implements the logic of the synopsis, being totally deprived of parallel execution details. This is a design choice that boosts the extensibility of the SaaMS Library since any other existing library that implements a synopsis can be plugged in a separate package, as those around the Synopses one in Figure 4, and only a new class with the ...Synopsis suffix, inheriting from Synopsis, should be created implementing add,estimate,merge,size and serde. Speaking about the serde method, the SaaMS Library software technology follows a modular approach here as well, packing serdes in a separate package, SynopsesSerdes in Figure 4. The SynopsesSerdes package includes a separate sub-package for each specific synopsis technique and Figure 4 uses the AMSSketch sub-package to provide a zoomed in viewpoint and illustrate the contents of each such sub-package. In particular, the AMSSketch sub-package in the SynopsesSerdes package includes 3 classes. The AMSSKetchSerializer and the AMSSKetchDeserializer implement the actual serialization 9 the application logic. Consumers interacting with SaaMS experience minimal to no disruption, as Kafka manages the rebalancing process internally. •Atomicity and Consistency: Kafka ensures atomicity and consistency during rebalancing operations to prevent data inconsistencies or processing anomalies. Stateful operations maintain data integrity, and Kafka’s consumer group coordination ensures that all instances agree on the partition assignments and processing responsibilities. In all, SaaMS turns Kafka’s generic rebalance machinery into an application-aware, continuum-wide control plane. Each synopsis is shipped as a lightweight microservice with its own consumer group, so failures or scale in events trigger partition movement, state migration, and task activation only for the synopses that were actually running on the affected node(s). SaaMS further orchestrates automatic scale out procedures by launching new instances whenever spare capacity appears when new nodes join, letting synopses microservices that have not exceeded their noOfP parallelism cap exploit the newly introduced resources. Because the state that moves is a compact synopsis (KTables /state stores) instead of raw streams, replay and recovery overhead remain minimal, while JMX-based feeds make any autoscaling policies orthogonal to SaaMS. The result is virtually zero downtime, minimal recovery latency, and enhanced elasticity capabilities that plain Kafka Streams cannot deliver on its own for concurrently running synopses without continuous manual intervention. 6. Experimental Evaluation For repeatability purposes, we provide a detailed description of our experimental setup. SaaMS Open Source Repository: SaaMS code is available as open-source software [32], along with detailed installation instructions for both bare-metal setups (used in this experimental evaluation) as well as via Docker containers. SaaMS has been developed using Kafka and Kafka Streams 3.3.1. It has been tested with Zookeeper 3.9.0, Java 19, and Maven 3.6.3. Backward compatibility has also been verified with versions of Kafka down to 2.8.1. The Scripts folder of the repository provides the necessary scripts for starting the Kafka brokers and a Zookeeper Server. The RequestExamples folder includes the exemplary syntax for all possible requests (Section 4.1) across supported synopses, including those used in the experiments presented below. Statistics Collector: In our experiments, we monitor relevant performance metrics (throughput and communication cost) using JMX technology2. The JMX API is a standard API for monitoring and managing resources, Kafka Streams applications, devices, services, and Java Virtual Machines themselves. JMX provides tools for local and remote monitoring, as well 2https://docs.oracle.com/en/java/javase/19/jmx/java-managementextensions-jmx-user-guide.html as capabilities for collecting and exposing application performance statistics in real-time. Our lightweight statistics collector is available in the metrics directory of SaaMS code repo. It connects to every Kafka Streams JVM and polls it over JMX, so as to monitor both processing speed and application level communication cost, anywhere from the cloud down to edge devices. Inside each SaaMS task running on a device, aByteCountingSensor registers custom byte counters with the StreamsMetrics registry, which are then exposed as JMX MBeans. A command line interface (MetricsUserInterface) receives a comma separated list of JMX service URLs, one per node in the cloud-to-edge continuum, and establishes two collectors: (i) the ThroughputJMXMetrics collector aggregates the thread level throughput from every URL/device periodically (the period is expressed as a configurable number of seconds) and appends a timestamped block to a report, (ii) the CommunicationCostJMXMetrics collector snapshots the two custom byte counters once and prints the network wide totals to stdout. Because the statistics collector scrapes any JMX endpoint, it works across heterogeneous nodes (cloud VMs, fog gateways, Raspberry Pis) so long as we expose the right port and enable JMX. Hardware Setup: For our experimental set up, we have a network composed of a cloud and a network side with the following characteristics: Cloud side, running Ubuntu 22.04.3 LTS server, equipped with (a) Two Intel Xeon Silver 4310 processors with 12 cores and 24 threads each, (b) Four 64GB RAMs RDIMM 3200MT/s each, (c) One ROM 960GB SSD vSAS with read-intensive 12Gbps. Network side with 5 VMs running Raspberry Pi OS (Debian GNU/Linux 11) each with a hardware configuration of (a) One CPU, (b) 16GB RAM, (c) 30GB ROM. Datasets: We use two real datasets from the stock market [47, 48] and the maritime domain [2, 50]. The full stock market data spans from 1 January 2019 to 31 December 2019, covering more than 500 tradeable instruments: forty-nine Forex pairs, nine spot-crypto pairs, twenty ICE & CME futures contracts (energy, metals, rates, and index futures), eleven synthetic equity index contracts, four proprietary gaming tokens, the rest being cash equities partitioned by market (200+US, 130+UK, 80+FR, 70+DE, 30+ES, 10+CH, and 10+NL). Each trading day is in a quotes/ directory of tick-by-tick files and a companion history/ directory of one minute OHLCV (Open, High, Low, Close, and Volume) aggregates; the tick files contain four comma separated fields: Date, Time, Price (native currency), and Volume. A single day contributes on average 8.4×106ticks (σ=0.9×106), of which Forex accounts for 39%, US equities 29%, futures 13%, index synthetics 7%, European equities 11%, and the remaining asset classes <1%. Scaling the daily figure to the 252 trading sessions of 2019 yields roughly 3.1×109quotes and trades. The median tick frequency per instrument is 9.6×103messages per session, with a 95th percentile burst rate of 1080 messages/sec. The full maritime dataset comprises roughly 3.8×107timestamped position reports collected from 500 distinct vessels operating in the Saronic Gulf (23.1–25.4°E, 37.1–38.4°N). Each record supplies a UTC time field, a vessel identifier, geodetic coordinates (longitude, latitude) and kinematic attributes in16 cluding heading, course over ground (COG), speed over ground (SOG), ship type code and the destination of the vessel. The stream averages 7.4×104updates per ship, peaking above 1000 messages/sec, with vessel speeds spanning 0–38.6 kn (mean 1.9 kn, rising to 5.6 kn when vessels staying idle at ports are excluded). Passenger ferries (AIS ship type code 60), tugs (AIS ship type code 37), fast ships (AIS ship type code 90), pleasure boats (AIS ship type code 80) and product tankers (AIS ship type code 52) dominate the received messages. The file is a direct export of raw NMEA 0183 AIS messages decoded by shore receivers. The counters and ranges reported above rely on a light cleaning pass that removes rare latitude, longitude outliers before statistical aggregation. Synopses and Synopses Parameters: Since SaaMS ingests JSON formatted Kafka messages, a CSV to JSON converter is also provided in the SaaMS code repository [32]. In the synopses used in our experiments we maintain CountMin (denoted by CM -ϵ=0.002, δ=0.99), HyperLogLog (denoted by HLL - RS D =0.02) and Discrete Fourier Transform (denoted by DFT - WindowS ize =500,S lideS ize =200,#coe f f icients =8). We set these parameters after discussions with domain experts who share the datasets in the above cited links. As shown in Table 2, each synopses is destined to support different types of analytics related to frequency estimation, distinct count and correlation, respectively, on the <price,volume >and positional attributes of stocks and vessels for the involved domain data streams. Competitor Approaches: In Section 6.4 we perform an experimental comparison between SaaMS and the prior state-of-theart, SDEaaS apporoach [31, 30]. The code of SDEaaS is provided open source as well [33]. Note that our experiments concentrate on computational and communication performance figures. We do not provide results for the synopses accuracy, since SaaMS does not alter in anyway the accuracy guarantees of synopses. Theoretic bounds and experimental results on the accuracy of each synopsis can be found in related works cited in Table 2. In our experimental evaluation we first examine SaaMS performance with respect to the scalability type requirements motivated in Section 1. We present triplets of figures describing the horizontal, vertical and federated scalability at the cloud side, the network side and the continuum as a whole. Since we notice little deviations between the conclusions drawn from the stock and the maritime datasets, we initially focus on the stock data. Then, we provide a comparative study between the previous state-of-the-art, the SDEaaS approach [30], and SaaMS on the Maritime Dataset. Again analogous results can be extracted for the stock market data. 6.1. SaaMS Horizontal Scalability Figure 9 plots the achieved throughput (vertical axes) while varying parallelism (horizontal axes), at the cloud side (Figure 9(a)), at the network side (Figure 9(b)) and across the cloud to edge continuum as a whole (Figure 9(c)). As illustrated in Figure 9(a), at the cloud side, SaaMS exhibits horizontal scalability that is polynomial while increasing parallelism from 3 to 18, for all the types (CM,HLL,DFT) of maintained synopses as well as cumulatively (red line in Figure 9(a)). This result, as noted on the vertical axis label, stands for processing 100 streams maintaining 3 synopses per stream (300 synopses in total). Polynomial scalability, where increasing parallelism leads to a polynomial increase in throughput, indicates that as SaaMS scales out (more instances and processing threads are devoted), the overall performance improves at a rate that is faster than linear. This means SaaMS makes efficient use of the additional resources that become available to it in order to handle its workload. In addition, the exhibited polynomial scalability provides flexibility in scaling SaaMS to accommodate growing workloads. As the demand for processing power increases, SaaMS can add more resources to the system to maintain performance levels without hitting a performance plateau too quickly. Overall, polynomial scalability is a desirable characteristic in distributed systems like SaaMS because it indicates that the system can efficiently utilize resources, handle increasing workloads, and maintain performance as it scales. At the network side, plotted in Figure 9(b), we observe that in this case, because the computational resources of Raspberry Pis are more restricted, SaaMS scales almost linearly, while processing 20 streams maintaining 3 synopses per stream, i.e., 60 synopses in total. This holds both upon examining individual CM,HLL,DFT synopses, as well as in cumulative throughput represented by the red line in Figure 9(b). More precisely, the red line in the figure has a sigmoid shape while increasing the number of Raspberry Pis in the network from 2 to 5. The blue line shows how throughput would be plotted in case of absolutely linear trend while increasing parallelism. For fewer than 4 Raspberry Pis, SaaMS is below linear throughput, while for 4 and 5 Raspberry Pis, the throughput of SaaMS shows a trend that is better than linear. Nevertheless, the blue shaded areas above and below the red line are almost equivalent in surface. Therefore, in total, SaaMS exhibits a linear trend in throughput increase upon adding more devices at the network side. What is more important is that, across the cloud to edge continuum, Figure 9(c) shows that the overall performance of SaaMS is superior compared to examining the cloud (Figure 9(a)) or the network (Figure 9(b)) sides, separately. This holds both for individual types of synopses (black lines), and cumulatively (red line). In particular, in Figure 9(c) the number of parallel instances in the horizontal axis is varied between 6 to 23. These numbers are the sum of the parallelism at the cloud side and the number of Raspberry Pis used at the network side. We begin with a parallelism of 6 at the cloud side, we scale out to 6+2=8 by adding 2 Raspberry Pis. Then, we switch to a setup with a parallelism of 9 at the cloud side and 3 Raspberry Pis at the network side (9 +3=12 parallel instances) and to 12+4=16. Finally, the last value of 23 in the horizontal axis is formed by a parallelism of 18 at the cloud side and 5 Raspberry Pis at the network side. In Figure 9(c), it can be observed that for fewer Raspberry Pis (2-3), where the throughput trend at the network side is below linear (Figure 9(b)), this slight under-performance is balanced out by the cloud side. On the other hand, for parallelism between 12 to 18, the cloud side shows a more moderate increase in throughput (Figure 9(a)). But being combined with the performance of the network side, which is above linear for 17 0.0E+00 3.0E+05 6.0E+05 9.0E+05 1.2E+06 1.5E+06 361218 Throughput (tuples/sec) 100 Streams - 300 Synopses Degree of Parallelism Cumulative@Cloud CM HLL DFT (a) Horizontal Scalability @ the Cloud Side 0.0E+00 1.0E+05 2.0E+05 3.0E+05 4.0E+05 2345 Throughput (tuples/sec) 20 Streams - 60 Synopses Number of Raspberry Pi OS VMs Cumulative@Network CM HLL DFT (b) Horizontal Scalability @ the Network Side 0.0E+00 3.0E+05 6.0E+05 9.0E+05 1.2E+06 1.5E+06 1.8E+06 6 8 12 16 23 Throughput (tuples/sec) 120 Streams - 360 Synopses Total Parallel Miroservice Instances Cumulative@Continuum CM HLL DFT (c) Horizontal Scalability across the Continuum Figure 9: SaaMS Horizontal Scalability more than 3 Raspberry Pis, contributes in improving the overall throughput across the cloud to edge continuum. Therefore, both the black and the red lines in Figure 9(c) show a higher increasing trend in throughput compared to examining the network or the cloud side, separately. 6.2. SaaMS Vertical Scalability Figure 10 plots the vertical scalabiliy of SaaMS. Recall that vertical scalability refers to the ability to scale the computation with the number of processed streams. At the cloud side (Figure 10(a)) we fix parallelism to 9 and we explore the ability of SaaMS to scale by progressively increasing the number of processed streams from 30 to 230 for each type of synopsis (CM,HLL,DFT). At the network side (Figure 10(b)) we use 3 Raspberry Pis and we explore the ability of SaaMS to scale by progressively increasing the number of processed streams from 10 to 30 for each type of synopsis, due to the more resource constrained setup. Across the cloud to edge continuum (Figure 10(c)), to keep the correspondence between the graphs, we start by maintaining synopses for 40 streams, 30 streams from the cloud plot of Figure 10(a), plus 10 from the network side of Figure 10(b) (40 streams in total). Then, we change the setup to 60+20 streams, switching to 150+30 streams. Finally, to the 180 streams of the previous step, we start maintaining synopses for 80 more streams. This explains the horizontal axes among the various subfigures of Figure 10. In addition, note that by adding more streams we simultaneously increase the volume and velocity of the ingested data. Figure 10(a) illustrates that, at the cloud side, SaaMS scales linearly with increasing number of processed streams, both per synopsis type (black lines) and cumulatively (red line). This is an important result since even maintaining steady throughput with the number of processed streams would be satisfactory from a vertical scalability viewpoint. On the contrary, at the network side (Figure 10(b)), individual synopses maintain relatively steady throughput with increasing number of streams. Nonetheless, the cumulative throughput expressed by the red line in Figure 10(b) shows almost linear scaling. The reason for this behavior is that individual types of maintained synopses are not of absolutely steady throughput, but have a slightly increasing trend. Therefore, when the total throughput is computed, the small throughput increments per synopsis type accumulate to an overall linear scaling trend with increasing number of processed streams. Finally, the vertical scalability across the continuum is dominated by the linear throughput increase at the cloud side (Figure 10(c)). 6.3. SaaMS Federated Scalability To judge the federated scalability of SaaMS, we measure the amount of data communicated between microservice instances both at the cloud and at the network side, among Raspberry Pis. In Figure 11, the horizontal axes hold the number of parallel microservice instances, while there are 2 vertical axes in each graph. The leftmost vertical axes, in each of Figure 11(a), Figure 11(b) and Figure 11(c) measures the amount of communicated data between parallel microservice instances while maintaining the synopses as described for Figures 9(a), Figure 9(b) and Figure 9(c), correspondingly. Because simply mentioning the amount of data communicated by SaaMS is not informative enough by itself, we also include the corresponding communication cost for answering continuous queries (see Output Estimation column in Table 2) in case we communicate the original streams instead of synopses. Therefore, all subfigures in Figure 11 have separate clusters of bars for the communication cost of answering queries using synopses, denoted by CM+HLL+DFT, versus the bars exposing the communication cost for answering queries using the original streams, termed NoCM+NoHLL+NoDFT. Given these, the rightmost vertical axis in each subfigure shows the communication ratio of Raw Streams over SaaMS and the red line plots this ratio for the corresponding number of parallel microservice instances. As shown by the red line in Figure 11, the federated scalability virtues of SaaMS are more pronounced at the network (Figure 11(b)), rather than the cloud side (Figure 11(a)). In particular, at the cloud side, synopses can reduce the communication cost from an order of magnitude and up to 28 times compared to the naive approach of continuously communicating the raw streams in order to answer frequency estimation (for CM), distinct count (for HLL) and correlation (for DFT) queries. For the network side (Figure 11(b)), the communication cost reduction is between 170 and up to 188 times while varying the number of Raspberry Pis in the network, from 2 to 5. The overall communication cost reduction across the continuum (Figure 11(c)), therefore, is between 188 and 215 times. 18 0.0E+00 2.0E+05 4.0E+05 6.0E+05 8.0E+05 1.0E+06 1.2E+06 1.4E+06 1.6E+06 30 60 150 230 Throughput (tuples/sec) Parallelism = 9 Number of Streams Cumulative@Cloud CM HLL DFT (a) Vertical Scalability @ the Cloud Side 0.0E+00 5.0E+04 1.0E+05 1.5E+05 2.0E+05 2.5E+05 3.0E+05 1030 Throughput (tuples/sec) 3 RPis 20 Number of Streams Cumulative@Cloud CM HLL DFT (b) Vertical Scalability @ the Network Side 0.0E+00 3.0E+05 6.0E+05 9.0E+05 1.2E+06 1.5E+06 1.8E+06 40 80 180 260 Throughput (tuples/sec) Parallelism = 9 + 3 RPis Number of Streams Cumulative@Cloud CM HLL DFT (c) Vertical Scalability across the Continuum Figure 10: SaaMS Vertical Scalability 0 5 10 15 20 25 30 1.0E+00 1.0E+01 1.0E+02 1.0E+03 1.0E+04 3 6 9 12 18 Intra-cloud Communication Cost (MB - LogScale) Degree of Parallelism CM+HLL+DFT NoCM+NoHLL+NoDFT Raw Streams/SaaMS x Times Communication Cost Ratio (a) Federated Scalability @ the Cloud Side 100 110 120 130 140 150 160 170 180 190 1.0E-01 1.0E+00 1.0E+01 1.0E+02 1.0E+03 1.0E+04 2345 Communication Cost (MB - LogScale) Number of Raspberry Pi OS VMs CM+HLL+DFT NoCM+NoHLL+NoDFT Raw Streams/SaaMS x Times Communication Cost Ratio (b) Federated Scalability @ the Network Side 165 170 175 180 185 190 195 200 205 210 215 220 1.0E+00 1.0E+01 1.0E+02 1.0E+03 1.0E+04 6 8 12 16 23 Communication Cost MB - LogScale Total Parallel Microservice Instances CM+HLL+DFT NoCM+NoHLL+NoDFT Raw Streams/SaaMS x Times Communication Cost Ratio (c) Federated Scalability across the Continuum Figure 11: SaaMS Federated Scalability The communication cost reduction ratio achieved by SaaMS at the network side is much higher compared to the one at the cloud side. This is due to the data shuffling optimization performed by Kafka, according to the discussion in Section 5.2. Data shuffling optimization is possible at the cloud side of the network where the actual Kafka topics are handled, rather than at the network devices. Therefore, because SaaMS communicates compact data summaries instead of the original raw streams even at the network side, its benefits are greater. This is an important argument exhibiting that simply deploying Kafka Streams across the cloud to edge continuum and merely relying on Kafka Streams optimization for federated scalability is not sufficient by itself. It is the streams synopses microservices of SaaMS that enable federated scalability. Moreover, we note that the communication cost reduction achieved by SaaMS is expected to be even greater for larger networks. This is because, in larger network settings, the number of hops data need to travel in order to reach the query source has an aggregative effect in the total communication burden across the continuum. 6.4. SaaMS vs SDEaaS In this section we provide a comparative study on the current state of the art, the SDEaaS approach [31, 30] versus SaaMS. SDEaaS has been proven [31, 30, 42] to outperform other (nonSDEaaS competitors) on the same datasets we use in this work. We use the Maritime dataset for this experiment, keeping synopses for up to 500 vessels. SDEaaS is designed to operate at the cloud side and, therefore, we also restrict SaaMS to deploying it only at the cloud side. As discussed in Section 1, Section 2, Section 5.1 and Section 5.3, the microservice architecture of SaaMS exploits Kafka and Kafka Streams in order to re-scale with zero downtime. SDEaaS on the other hand, as a Flink job, in each re-scaling decision requires to (i) take a savepoint, (ii) stop the running SDEaaS job, (iii) restart the job with altered parallelism, loading the previously taken savepoint. Therefore, for SDEaaS we measure two versions of its throughput. “SDEaaS only Uptime” measures throughput while the job is up and running, while “SDEaaS Uptime +Downtime” measures the average throughput throughout the job’s lifespan, also accounting for the fact that during downtimes, the throughput of SDEaaS is zeroed out. In Figure 12, we have the vertical axis, which measures throughput, and two horizontal axes. The horizontal axis at the bottom shows the number of vessels for which we keep synopses (equivalent to those of the previous sections). Note that this entails both increasing the number of processed streams (vertical scalability) and the volume, velocity of the incoming streams (horizontal scalability). As the horizontal axis at the bottom shows, we begin with keeping synopses for 10 streams and we increase the number of streams up to 500. The more streams SaaMS processes, the higher the volume and the velocity of ingested streams/vessels. The horizontal axis at the top of Figure 12 shows the parallelism and how it is altered during the experiment. So, all the candidate approaches begin with a parallelism of 3, when 50 streams are processed the parallelism switches to 6, it further increases at 100 streams to a value of 9 and reaches 12 for ∼180 streams/vessels. Parallelism remains 19 0.0E+00 2.0E+05 4.0E+05 6.0E+05 8.0E+05 1.0E+06 1.2E+06 1.4E+06 1.6E+06 10 100 1000 Average Throughput (Tuples/sec) Number of Vessels SDEaaS Uptime + Downtime SDEaaS only Uptime SaaMS 3 6 9 12 Parallelism Scale out Figure 12: SaaMS vs SDEaaS Comparison. The “SDEaaS only Uptime” line plots throughput without considering donwtime due to re-scaling. “SDEaaS Uptime +Downtime” also accounts for zero throughput during re-scaling intervals. fixed to 12 for the rest (180 - 500 streams/vessels) of the experiment. Note that we decide the timepoint for re-scaling based on the number of processed streams, as detailed above. Any mechanism that performs adaptive re-scaling in a different way is orthogonal to SaaMS. Proposing an autoscaler for SaaMS or SDEaaS is out of the scope of the current work and, threfore, we leave further investigation of this issue as future work. As Figure 12 demonstrates, “SDEaaS only Uptime” can achieve between 1.5 to 3.8 times higher throughput compared to SaaMS for up to 200 streams. For more than 200 streams, SaaMS shows up to 1.5 times higher throughput. The reason for this behavior is that, SDEaaS ingests data from Kafka topics. Flink (and therefore SDEaaS) does not enforce strict collocation of Kafka topics with Flink tasks by default, as Kafka does. Therefore, when the ingestion (message consumption) load increases, the time devoted to reading messages from topics that are not necessarily collocated with the processing tasks, causes a drop in the overall throughput. On the contrary, as discussed in Section 5.2, Kafka by default performs data shuffling optimization. Nevertheless, “SDEaaS only Uptime” shows an ideal picture about the performance of SDEaaS. This is because “SDEaaS only Uptime” totally ignores the intervals of downtime throughout the SDEaaS lifespan. While switching parallelism from 3→6→9→12 SDEaaS exhibits downtime the duration of which varies from tens of seconds to minutes depending on the state size of the respective savepoints. During these time intervals, no streaming tuples are processed and therefore SDEaaS’s throughput is zero. When we take this into account, the true average throughput of SDEaaS as plotted by “SDEaaS Uptime +Downtime” is significantly deteriorated. For any number of streams/vessels above 40, SaaMS shows from 1.2 to 3 times higher throughput compared to SDEaaS. An important observation involves the greater deterioration in average throughput between “SDEaaS Uptime +DownTime” and “SDEaaS only Uptime”, SaaMS for larger number of vessels/streams. This behavior appears due to the increased state size and the increased complexity of state organization as a result of maintaining more synopses for more streams. Therefore, it takes more time in Flink to take a savepoint of a larger state, stop the job and then load that larger state in order to restart the job under a different parallelism. Finally, all 3 approaches show a considerable deterioration in throughput for any value streams/vessels above 300 due to the respective increase in the workload that progressively causes backpressures. Even in these cases, SaaMS provides 1.3 and up to 3 times higher throughput than SDEaaS. 7. Conclusion and Future Work In this work we presented SaaMS, the first engine for maintaining hundreds of stream synopses of various types across the cloud to edge continuum. SaaMS attributes horizontal, vertical and federated scalability to the applications that utilize its synopses. It also ensures adaptivity to changing network conditions and stream statistical properties, with zero downtime. SaaMS outperforms the current state of the art by ensuring up to 3 times higher throughput, for hundreds of streams, according to our experimental evaluation. Our future work concentrates on deploying SaaMS in larger, real networks composed of devices with heterogeneous resource capacities. We are further developing a, Bayesian Optimization-based, optimization module for automatically setting and adapting the parallelism of each synopsis microservice, based on our prior experience [49]. Acknowledgments Nikos Giatrakos was supported by the EU project CREXDATA under Horizon Europe agreement No. 101092749. References [1] A. Kontaxakis, A. Deligiannakis, H. Arndt, S. Burkard, C. Kettner, E. Pelikan, K. Noack, Real-time processing of geo-distributed financial data, in: A. Margara, E. D. Valle, A. Artikis, N. Tatbul, H. Parzyjegla (Eds.), 15th ACM International Conference on Distributed and Event-based Systems, DEBS 2021, Virtual Event, Italy, June 28 - July 2, 2021, ACM, 2021, pp. 190–191. doi:10.1145/3465480.3467842. URL https://doi.org/10.1145/3465480.3467842 [2] M. Vodas, K. Bereta, D. Kladis, D. Zissis, E. Alevizos, E. Ntoulias, A. Artikis, A. Deligiannakis, A. Kontaxakis, N. Giatrakos, D. Arnu, E. Yaqub, F. Temme, M. Torok, R. Klinkenberg, Online distributed maritime event detection & forecasting over big vessel tracking data, in: 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, December 15-18, 2021, 2021, pp. 2052–2057. [3] G. Cormode, K. Yi, Small Summaries for Big Data, Cambridge University Press, 2020. [4] G. Cormode, M. N. Garofalakis, Join sizes, frequency moments, and applications, in: Data Stream Management - Processing High-Speed Data Streams, 2016, pp. 87–102. [5] G. Cormode, M. Garofalakis, P. Haas, C. Jermaine, Synopses for massive data: Samples, histograms, wavelets, sketches, Foundations and Trends in Databases 4 (1-3) (2012) 1–294. [6] G. Cormode, S. Muthukrishnan, K. Yi, Q. Zhang, Optimal sampling from distributed streams, in: Proceedings of the Twenty-Ninth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS 2010, June 6-11, 2010, Indianapolis, Indiana, USA, 2010, pp. 77–86. 20 [7] G. Cormode, S. Muthukrishnan, An improved data stream summary: the count-min sketch and its applications, J. Algorithms 55 (1) (2005) 58–75. [8] G. Cormode, M. N. Garofalakis, Approximate continuous querying over distributed streams, ACM Trans. Database Syst. 33 (2) (2008) 9:1–9:39. [9] P. Flajolet, ´ E. Fusy, O. Gandouet, F. Meunier, Hyperloglog: the analysis of a near-optimal cardinality estimation algorithm, in: Discrete Mathematics and Theoretical Computer Science, Discrete Mathematics and Theoretical Computer Science, 2007, pp. 137–156. [10] P. Flajolet, G. N. Martin, Probabilistic counting algorithms for data base applications, J. Comput. Syst. Sci. 31 (2) (1985) 182–209. [11] B. H. Bloom, Space/time trade-offs in hash coding with allowable errors, Commun. ACM 13 (7) (1970) 422–426. [12] B. Babcock, M. Datar, R. Motwani, Sampling from a moving window over streaming data, in: Proceedings of the Thirteenth Annual ACMSIAM Symposium on Discrete Algorithms, January 6-8, 2002, San Francisco, CA, USA, 2002, pp. 633–634. [13] G. S. Manku, R. Motwani, Approximate frequency counts over data streams, in: Proceedings of 28th International Conference on Very Large Data Bases, VLDB 2002, Hong Kong, August 20-23, 2002, 2002, pp. 346–357. [14] M. N. Garofalakis, Discrete wavelet transform and wavelet synopses, in: L. Liu, M. T. ¨ Ozsu (Eds.), Encyclopedia of Database Systems, Springer US, 2009, pp. 857–863. doi:10.1007/978-0-387-39940-9 539. URL https://doi.org/10.1007/978-0-387-39940-9\_539 [15] M. Shekelyan, A. Dign¨ os, J. Gamper, Digithist: a histogram-based data summary with tight error bounds, Proc. VLDB Endow. 10 (11) (2017) 1514–1525. doi:10.14778/3137628.3137658. URL http://www.vldb.org/pvldb/vol10/p1514-shekelyan. pdf [16] Y. E. Ioannidis, The history of histograms (abridged), in: J. C. Freytag, P. C. Lockemann, S. Abiteboul, M. J. Carey, P. G. Selinger, A. Heuer (Eds.), Proceedings of 29th International Conference on Very Large Data Bases, VLDB 2003, Berlin, Germany, September 9-12, 2003, Morgan Kaufmann, 2003, pp. 19–30. doi:10.1016/B978-012722442-8/50011-2. URL http://www.vldb.org/conf/2003/papers/S02P01.pdf [17] D. E. Yagoubi, R. Akbarinia, F. Masseglia, D. E. Shasha, Radiussketch: Massively distributed indexing of time series, in: 2017 IEEE International Conference on Data Science and Advanced Analytics, DSAA 2017, Tokyo, Japan, October 19-21, 2017, IEEE, 2017, pp. 262–271. doi:10.1109/DSAA.2017.49. URL https://doi.org/10.1109/DSAA.2017.49 [18] Y. Zhu, D. E. Shasha, Statstream: Statistical monitoring of thousands of data streams in real time, in: Proceedings of 28th International Conference on Very Large Data Bases, VLDB 2002, Hong Kong, August 20-23, 2002, 2002, pp. 358–369. [19] M. Charikar, Similarity estimation techniques from rounding algorithms, in: Proceedings on 34th Annual ACM Symposium on Theory of Computing, May 19-21, 2002, Montr´ eal, Qu´ ebec, Canada, 2002, pp. 380–388. [20] P. K. Agarwal, G. Cormode, Z. Huang, J. M. Phillips, Z. Wei, K. Yi, Mergeable summaries, in: M. Benedikt, M. Kr¨ otzsch, M. Lenzerini (Eds.), Proceedings of the 31st ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS 2012, Scottsdale, AZ, USA, May 20-24, 2012, ACM, 2012, pp. 23–34. [21] Apache Spark v. 3.5.0, https://spark.apache.org/. [22] Apache Flink v. 1.18, https://flink.apache.org/. [23] R. P. Lemaitre, M. Kiefer, J. V. Hein, J. Quian´ e-Ruiz, V. Markl, In the land of data streams where synopses are missing, one framework to bring them all, Proc. VLDB Endow. 14 (10) (2021) 1818–1831. [24] O. Levchenko, D. E. Yagoubi, R. Akbarinia, F. Masseglia, B. Kolev, D. E. Shasha, Spark-parsketch: A massively distributed indexing of time series datasets, in: A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Z. Broder, M. J. Zaki, K. S. Candan, A. Labrinidis, A. Schuster, H. Wang (Eds.), Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, ACM, 2018, pp. 1951–1954. doi:10.1145/3269206.3269226. URL https://doi.org/10.1145/3269206.3269226 [25] O. Levchenko, B. Kolev, D. E. Yagoubi, R. Akbarinia, F. Masseglia, T. Palpanas, D. E. Shasha, P. Valduriez, Bestneighbor: efficient evaluation of knn queries on large time series databases, Knowl. Inf. Syst. 63 (2) (2021) 349–378. doi:10.1007/S10115-020-01518-4. URL https://doi.org/10.1007/s10115-020-01518-4 [26] N. Giatrakos, et al, Infore: Interactive cross-platform analytics for everyone, in: M. d’Aquin, S. Dietze, C. Hauff, E. Curry, P. Cudr´ e-Mauroux (Eds.), CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, ACM, 2020, pp. 3389–3392. doi:10.1145/3340531.3417435. URL https://doi.org/10.1145/3340531.3417435 [27] I. Flouris, N. Giatrakos, A. Deligiannakis, M. N. Garofalakis, Networkwide complex event processing over geographically distributed data sources, Inf. Syst. 88 (2020). doi:10.1016/J.IS.2019.101442. URL https://doi.org/10.1016/j.is.2019.101442 [28] G. Stamatakis, A. Kontaxakis, A. Simitsis, N. Giatrakos, A. Deligiannakis, Sheermp: Optimized streaming analytics-as-a-service over multisite and multi-platform settings, in: Proceedings of the 25th International Conference on Extending Database Technology, EDBT 2022, Edinburgh, UK, March 29 - April 1, 2022, 2022, pp. 2:558–2:561. [29] D. Giouroukis, A. Dadiani, J. Traub, S. Zeuch, V. Markl, A survey of adaptive sampling and filtering algorithms for the internet of things (2020). doi:10.1145/3401025.3403777. URL https://doi.org/10.1145/3401025.3403777 [30] A. Kontaxakis, N. Giatrakos, D. Sacharidis, A. Deligiannakis, And synopses for all: A synopses data engine for extreme scale analytics-as-aservice, Inf. Syst. 116 (2023) 102221. doi:10.1016/J.IS.2023.102221. URL https://doi.org/10.1016/j.is.2023.102221 [31] A. Kontaxakis, N. Giatrakos, A. Deligiannakis, A synopses data engine for interactive extreme-scale analytics, in: CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, 2020, pp. 2085–2088. [32] SaaMS – Synopses-as-a-MicroService Open-source Repository, https: //github.com/geok1999/Synopses-as-a-MicroService-SaaMS (2024). [33] SDEaaS – Synopses Data Engine-as-a-Service Open-source Repository, https://sdeaas.github.io/ (2023). [34] J. Kreps, N. Narkhede, J. Rao, et al., Kafka: A distributed messaging system for log processing, in: Proceedings of the NetDB, 2011, pp. 1–7. [35] Apache Kafka v. 3.3, https://kafka.apache.org/. [36] Apache datasketches, https://datasketches.github.io/. [37] Stream-lib, https://github.com/addthis/stream-lib. [38] D. L. Quoc, R. Chen, P. Bhatotia, C. Fetzer, V. Hilt, T. Strufe, Streamapprox: approximate computing for stream analytics, in: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, Las Vegas, NV, USA, December 11 - 15, 2017, 2017, pp. 185–197. [39] B. Mozafari, Snappydata, in: Encyclopedia of Big Data Technologies, 2019. [40] Data stream management - processing high-speed data streams, DataCentric Systems and Applications, Springer, 2016. doi:10.1007/978-3540-28608-0. URL https://doi.org/10.1007/978-3-540-28608-0 [41] A. Povzner, P. Mahajan, J. Gustafson, J. Rao, I. Juma, F. Min, S. Sridharan, N. Bhatia, G. K. Attaluri, A. Chandra, S. Kozlovski, R. Sivaram, L. Bradstreet, B. Barrett, D. Shah, D. Jacot, D. Arthur, M. Chawla, R. Dagostino, C. Mccabe, M. R. Obili, K. Prakasam, J. G. Sancio, V. Singh, A. Nikhil, K. Gupta, Kora: A cloud-native event streaming platform for kafka, Proc. VLDB Endow. 16 (12) (2023) 3822–3834. doi:10.14778/3611540.3611567. URL https://www.vldb.org/pvldb/vol16/p3822-povzner.pdf [42] N. Giatrakos, E. Alevizos, A. Deligiannakis, R. Klinkenberg, A. Artikis, Proactive streaming analytics at scale: A journey from the state-of-theart to a production platform, in: I. Frommholz, F. Hopfgartner, M. Lee, M. Oakes, M. Lalmas, M. Zhang, R. L. T. Santos (Eds.), Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 2125, 2023, ACM, 2023, pp. 5204–5207. doi:10.1145/3583780.3615293. URL https://doi.org/10.1145/3583780.3615293 [43] N. Alon, Y. Matias, M. Szegedy, The space complexity of approximating the frequency moments, in: Proceedings of the Twenty-Eighth Annual ACM Symposium on the Theory of Computing, Philadelphia, Pennsylvania, USA, May 22-24, 1996, 1996, pp. 20–29. [44] M. Greenwald, S. Khanna, Space-efficient online computation of quantile summaries, in: S. Mehrotra, T. K. Sellis (Eds.), Proceedings of the 2001 ACM SIGMOD international conference on Management of data, Santa 21 Barbara, CA, USA, May 21-24, 2001, ACM, 2001, pp. 58–66. [45] N. Giatrakos, Y. Kotidis, A. Deligiannakis, V. Vassalos, Y. Theodoridis, In-network approximate computation of outliers with quality guarantees, Inf. Syst. 38 (8) (2013) 1285–1308. [46] A. Arasu, G. S. Manku, Approximate counts and quantiles over sliding windows, in: C. Beeri, A. Deutsch (Eds.), Proceedings of the Twentythird ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, June 14-16, 2004, Paris, France, ACM, 2004, pp. 286– 296. [47] Burkard, Data set of correlations between stocks world wide (Mar. 2022). doi:10.5281/zenodo.6331464. URL https://doi.org/10.5281/zenodo.6331464 [48] spring, Financial data set used in infore project (Jun. 2020). doi:10.5281/zenodo.3886895. URL https://doi.org/10.5281/zenodo.3886895 [49] N. Giatrakos, E. Kougioumtzi, A. Kontaxakis, A. Deligiannakis, Y. Kotidis, Easyflinkcep: Big event data analytics for everyone, in: G. Demartini, G. Zuccon, J. S. Culpepper, Z. Huang, H. Tong (Eds.), CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, ACM, 2021, pp. 3029–3033. doi:10.1145/3459637.3482094. URL https://doi.org/10.1145/3459637.3482094 [50] I. Kontopoulos, M. Vodas, G. Spiliopoulos., K. Tserpes. & D. Zissis. Single Ground Based AIS Receiver Vessel Tracking Dataset . (Zenodo,2020,4), https://doi.org/10.5281/zenodo.3754481 22