Marea 2. Design and Optimization of a Distributed Communications Middleware

MASTER THESIS Title : MAREA 2. Design and Optimization of a Distributed Communications Middleware Master Degree: Master in Science in Telecommunication Engineering & Management Author: Santiago Pérez Fernández Director: Juan López Rubio Date: October 26, 2013 Title: MAREA 2. Design and Optimization of a Distributed Communications Middleware Author: Santiago Pérez Fernández Director: Juan López Rubio Date: October 26, 2013 Overview In recent years, the rapid growth of distributed embedded systems have been vigorously pushing middleware systems and technologies in different areas like: telecommunications, health care, automotive, defense, avionics, etc. The aim of this master thesis is to develop a new optimized version of the middleware MAREA 1, a software specifically designed to fulfill Unmanned Aircraft Systems (UAS) communications and their application to the design of complex distributed UAS avionics. This document presents the software architecture of MAREA 2 and discusses design decisions, as well as some of the techniques employed to develop the middleware. MAREA 2 provides a more modular, flexible and reusable architecture and implements some new functionalities and features proposed in Service Oriented Architecture for Embedded (Avionics) Applications (López J., 2009). Another of the main contributions of this master thesis is the performance evaluation and optimization of the middleware through the analysis of some key performance parameters. The present document provides a comparison between the new and previous version of the middleware both in terms of design and performance. Títol: MAREA 2. Design and Optimization of a Distributed Communications Middleware Autor: Santiago Pérez Fernández Director: Juan López Rubio Data: 26 d’octubre de 2013 Resum En els últims anys, el ràpid creixement dels sistemes distribuïts embeguts ha impulsat amb determinació els sistemes i tecnologies middleware a diferents àrees com: telecomunicacions, salut, automoció, defensa, aviònica, etc. L’objectiu d’aquest projecte de fi de màster es desenvolupar una nova versió del middleware MAREA 1, un software específicament dissenyat per complir amb les comunicacions dels Sistemes Aeris No Tripulats i la seva aplicació en el disseny de sistemes aviònics distribuïts complexos per Avions No Tripulats. Aquest document presenta l’arquitectura de software de MAREA 2 i aborda les decisions del seu disseny, així com algunes de les tècniques emprades pel seu desenvolupament. MAREA 2 proporciona una arquitectura més modular, flexible i reusable i inclou algunes de les noves funcionalitats i característiques presentades a Service Oriented Architecture for Embedded (Avionics) Applications (López J., 2009). Una altra de les principals contribucions d’aquest projecte de fi de màster es l’avaluació i optimització del middleware a través de l’anàlisi d’alguns paràmetres clau de rendiment. El present document fa una comparativa entre la nova i l’anterior versió del middleware tant en relació al disseny com al rendiment. Primer de tot vull agrair als meus pares i la meva germana tot el suport incondicional i els esforços que han fet perquè pugi arribar fins aquí. També vull expressar el meu agraïment més sincer i profund a Ramona Vidal per ensenyar-me el valor de l’esforç, la constància i el treball. Finalment, vull agrair especialment al meu director Juan López tota la seva paciència, dedicació i temps. La seva ajuda, comentaris, suggeriments i correccions han contribuït notablement a que aquest treball tirés endavant. CONTENTS INTRODUCTION ................................. 1 CHAPTER 1. MAREA .............................. 3 1.1 Middleware Context ............................... 3 1.1.1 Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.3 Types Of Middleware . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.4 Adaptive and Reflective Techniques . . . . . . . . . . . . . . . . . . 7 1.1.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Description ................................... 7 1.3 Communication Primitives ........................... 8 1.4 System Architecture .............................. 9 1.5 Naming Service ................................. 10 1.5.1 Address Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.6 Service ...................................... 11 1.6.1 Service Description . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.6.2 Service Implementation . . . . . . . . . . . . . . . . . . . . . . . . 13 1.7 Conclusions ................................... 16 CHAPTER 2. NETWORK LAYER ....................... 19 2.1 Architecture ................................... 19 2.2 Router ...................................... 21 2.2.1 Network Lanes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3 NetworkMessage Pool ............................. 22 2.3.1 Results................................. 23 2.4 Encoding Layer ................................. 24 2.4.1 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4.2 Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.3 Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.4 Results................................. 28 1 INTRODUCTION The main objective of this master thesis is to design and implement a new version of the middleware MAREA 1, a software specifically designed to fulfill Unmanned Aircraft Systems (UAS) communications and their application to the design of complex distributed UAS avionics. Middleware is a system software that resides between the applications and the underlying operating systems, network protocol stacks, and hardware, which provides facilities in order to build and use distributed systems [1]. The time and cost of the deployment of middleware solutions across heterogeneous environments continues to grow as technology evolves and becomes more complex. The objective of this master to address this issue regarding the design and deployment of services, is to implement a new service approach based on the deployment unit concept (López J., 2009). An important work-line inside the middleware solutions is the support of naming services, where redundant services must be targeted with load balancing and fault tolerance. In relation to this topic, the objective of this master thesis is to implement a naming service that allows to find, share and access services and its communication primitives hiding the network complexity, and providing location transparency. Actual middleware solutions have a hard time to build and deploy distributed real-time and embedded systems. In recent years, the development of efficient middleware architectures has become one of the big challenges of distributed real-time and embedded systems. One of the objectives regarding this, is to study and use software optimization techniques to improve the performance of the new design of MAREA middleware. The document is structured in five different chapters. In addition to the chapters, the document includes some appendices with further information. A brief overview of each of the chapters follows: •Chapter 1: presents a brief introduction to MAREA 2, and provides a high-level overview of the system architecture from an end user programmer’s point of view. •Chapter 2: is the first of three chapters that exclusively deals with the internal implementation details of MAREA 2. This chapter describes the design of the network system architecture and compares it with the one from MAREA 1 by contrasting results. •Chapter 3: tackles the new design and implementation of the service container and provides a comparison with service container from MAREA 1. •Chapter 4: introduces the design of the protocol layer and describes it interactions with all the other components of the architecture. •Chapter 5: provides a summary of the conclusions reached and points out some future work. 2 MAREA 2. Design and Optimization of a Distributed Communications Middleware MAREA 3 CHAPTER 1. MAREA The first part of this chapter introduces basic concepts about embedded systems and middleware technologies required for understanding the content of the present document. The second part introduces MAREA middleware architecture, the communication mechanisms available to communicate services and the MAREA naming scheme. The last section of this chapter tackles the design and implementation of MAREA services from an end user programmer’s point of view. 1.1 Middleware Context This section sketches out briefly middleware systems and technologies. The first part introduces and describes distributed systems. Next, middleware is formally defined and the different types of middleware are analyzed. The section comes to an end some of the techniques applied in the development of next-generation middleware systems. 1.1.1 Distributed Systems Distributedcomputinganddistributedsystemshave gainedpopularityandimportanceover past years. The main purpose of this type of systems is to interact and exchange data between its set of components in order to share resources. This sort of systems could be perceived as a single integrated computing facility. The common characteristics of this type of systems are: resource sharing, fault tolerance, concurrency, scalability, transparency and openness. Distributed systems offer facilities to increase the performance, availability and reliability of applications. In general, this type of systems are cheaper than a centralized single system, because a large number of small, low-power systems tend to be cheaper than single computer. This type of systems is more complex to build and maintain than an equivalent centralized system. One example of this is the software developing complexity introduced to ensure a proper coordination and communication between the distributed components. This complexity would be a major unwanted problem for application developers. Another problem is the effect of heterogeneity in distributed systems. Distributed systems may containmanydifferenttypesofcomponents (softwareand hardware)working together in cooperative way to solve problems. 4 MAREA 2. Design and Optimization of a Distributed Communications Middleware 1.1.2 Middleware Complexity and heterogeneity drawbacks of distributed systems could be solved or relived using a middleware. Middleware is a system software that resides between the applications and the underlying operating systems, network protocol stacks, and hardware, which provides facilities in order to build and use distributed systems [1]. This type of software provides a transparent and abstract vision of the low-level details (e.g. network communication, encoding, concurrency, protocol handling, etc.) facilitating end user programming. Middleware typically provides two different types of transparency to distributed systems: •Access transparency: Hides differences between remote and local operations like data representation and invocation mechanisms. •Location transparency: Hides the location of components. The different components could be redistributed (e.g. moved between computers) without changing any of the other components. 1.1.3 Types Of Middleware The following subsection presents the main types of middlewareand describe the most important requirements (scalability, reliability, heterogeneity, transparency, etc.) addressed by each alternative. The most widely known middleware implementations are also mentioned for each type of middleware. 1.1.3.1 Message Oriented Middleware (MOM) Message oriented middleware (MOM), is a middleware that allows the communication between the components through messages. In this type of middleware, the coordination between the components could be achieved synchronously or asynchronously depending on the communication model of the MOM. •Message queuing: This asynchronous indirect communicationmodel uses a queue in order to exchange messages. The messages from the producer arestored into the consumer’s queue after being sent. In this type of communication model, persistent queues are used when the reliability is required in front of performance. Quality of service (QoS) policies are also a good solution to provide reliability. •Message Passing: In this direct communication model, the messages are sent directly to the interested parts through publish-subscribe pattern. First, the different parts register interest in receiving messages on a particular message topic. Then, consumers will receive any message corresponding to the subscribed topic. MAREA 5 MOM has a limited support for data heterogeneity because marshalling has to be implemented by the programmers. This type of middleware offers location transparency, inherent from publish-subscribe model, but has a limited support for access transparency. This lack of access transparency limits replication and migration transparency, complicating the scalability. MOMs are build around a reliability paradigm that is suitable for applications where the availability of a network or all components is not warranted [2]. The most common implementation of this type of middlewareare Sun’s Java Message Queue and IBM’s MQSeries. 1.1.3.2 Procedural Middleware (PM) Procedural middleware (PM) is based on the concept of Remote Procedure Calls (RPC). RPC is an interprocess communication (IPC) mechanism which is designed to exchange data and invoke functions between client and server processes. In this type of middleware, RPC servers contain procedures which could be invoked by remote clients across the network. PM provides location transparency because clients can invoke remote procedures as if it were local. RPC communications are synchronous, because clients remains blocked until the remote procedures have been executed. RPC presents good heterogeneity because the relationships between servers an clients is defined through common interface with IDL (Interface Definition Language). Client does not need to know the language that server supports because the IDL compilers can translate the clients request. IDL compilers also marshalls and interprocess data automatically. PM has a limited scalability due to the absence of replication and load balancing mechanisms. As opposite of MOM, PM does not support group communications. This type of middleware is specially useful for simple point-to-point applications. The most widely known implementations of PM are Microsoft RPC Facility and Open Software Foundation’s Distributed Computing Environment DCE. 1.1.3.3 Transactional Middleware (TM) Transactional middleware (TM) supports the development of distributed systems through asynchronous transactions using a client/server model. A transaction is an atomic and logical sequence of events or operations. This implies that all set of operations are either executed or not at all. TM use a request message to ask the system to execute a transaction. The transaction processing (TP) monitor is the responsible to coordinate the requests between clients and servers. Regarding reliability, TP monitors manage the transactions with the two phase commit protocol (2PC), which is commonly used by relational database management systems (DBMS) to provide fault tolerance. Once "prepare to commit phase" have finished, the TP monitor asks the TM to commit the transactions and make it final to all servers ("commit 6 MAREA 2. Design and Optimization of a Distributed Communications Middleware phase"). If at least one of them indicates that it cannot be executed, then the TP monitor asks all nodes to do a rollback in order to do not apply any change. TP monitors provide reliability by means of support for replications and load balancing for the different server components. One of the drawbacks of TM is the overhead introduced to manage and guarantee the transactions. This type of middleware also does not provide automatic marshalling and unmarshalling capabilities. As mentioned before, TM is typically used in DBMS where the transactions have to be synchronized and controlled over multiple databases. BEA’s Tuxedo, IBM’s CICS, and Transarc’s Encina are some typical implementations of this type of middleware. 1.1.3.4 Object Oriented Middleware (OOM) Object oriented middleware (OOM) is an evolution of PM which extends its features adding object oriented capabilities. OOM supports distributed object requests based on a client/server model. This type of middleware provides a local representation of remote objects, and hides the communication between remote objects and its local representation. The main idea is that all the objects can be accessed and invoked remotely anywhere in the network. OOM supports both synchronous and asynchronous communication. There are many examples of OOMs like CCM (CORBA Component Model), SUN’s Enterprise Java Beans, Java/RMI (Remote Method Invocation), Microsoft’s DCOM (Distributed Component Object Model), CORBA (Common Object Request Broker Architecture), etc. OOM presents good heterogeneity. For instance, CORBA supports different programming languages in the server and client. Java/RMI resolves the heterogeneity issue using Java Virtual Machine (JVM). One of the drawbacks of this type of middleware is scalability. Only some specific implementations like Enterprise Java Beans and CORBA support replication and load balancing respectively. OOM has high runtime and network communication overhead introduced by the support of features like service discovery. The bottlenecks appeared in the OOM technologies were due to many factors which includes excessive data copying, less compact encoding and complex encoding rules [3]. For other part, this type of middleware simplifies and provides a rapid integration of programming tasks for distributed and heterogeneous environments. This type of middleware is starting to be integrated with MOM. MAREA 7 1.1.4 Adaptive and Reflective Techniques Most of the middleware used until today present a lack of flexibility and adaptability to different application environments and areas. Adaptive and reflective techniques have been noted as a key emerging paradigm for the development of dynamic next-generation middleware platforms [1]. This type of techniques improves configurability and provides dynamic adaptation to middleware systems and technologies. Adaptive middleware is software whose functional behavior can be modified dynamically to optimize for a change in environmental conditions or requirements [4]. Reflectivemiddlewareapplies reflection technique at middlewarelevel. Reflectionprovides the ability for a program to observe or change its own code as well as all aspects of its programming language during runtime. Reflexive and adaptive techniques are both useful separately, but become very powerful tool together. Reflective capabilities trigger adaptive capabilities by allowing system inspection in case a behavior adaptation is needed. During this work new reflective and adaptive capabilities will be added to MAREA middleware architecture. 1.1.5 Conclusions This section introduces some basic concepts about distributed systems and middleware. Middleware provides mechanisms and tools that simplify the development of distributed applications. The distributed transparency provided by this type of software reduces the complexity of handling widely distributed systems. The idea of this section is not to overwhelm the reader with explanations but only to provide as much information as is necessary to understand basic concepts about middleware. Some of these concepts are mentioned in the subsequent chapters with the aim of describe implementation details about MAREA 2. 1.2 Description MiddlewareArchitecture for Remote Embedded Applications(MAREA) is a middlewaredesigned by ICARUS group specifically designed to fulfill Unmanned Aircraft Systems (UAS) communications and their application to the design of complex distributed UAS avionics [5]. The ICARUS group is composed by researchers of the Technical University of Catalonia - Barcelona Tech (UPC) mainly from the Computer Architecture Department of the Castelldefels School of Telecommunications and Aerospace Engineering (EETAC). The basic aim of this research group, formed in 2005, is develop technologies to automate air traffic management (ATM) with low cost UAS in civil airspace as well as more intelligent platforms that allow the deployment of civil applications for unmanned aircraft. 8 MAREA 2. Design and Optimization of a Distributed Communications Middleware MAREA proposes a modular architecture based on services (SOA). These type of architectures ensures extensibility, flexibility and interoperability across heterogeneous environments. MAREA is a mixed MOM/OOM which implements a Data Distribution System communication model based on the publish-subscribe pattern. This middlewarealso offers additional features like RPC and file-based data transfer. 1.3 Communication Primitives MAREA provides to the developers a different range of possibilities to interact and communicate the services between them through the following communication primitives: •Variables: This type of communication primitive is designed to share periodic and shortdeterministicinformationbetweendifferentservicesfollowingapublish-subscribe model. In this type of primitives the data is sent in a best effort way, through UDP transport, taking in account the periodic behavior of the publisher and the limited lifetime of the information. An appropriate use case of this type of primitive is the telemetry provided by a GPS navigation device. •Events: Similar to the variables, events are also used to share periodic and short information between services following a publish-subscribe model. As opposite of variables, the information in events is guaranteed to be delivered to all the subscribed services through TCP transport. This type of primitive is designed to share information about important and unpredictable facts. An example of this type of communication primitive can be any alarm used to inform about a critical system failure. •Remote invocation: MAREA also offers an alternative Data Distribution System communication model to the publish-subscribe pattern like remote invocation. In this type of primitive the communication is established only between two services following the request/reply pattern using a client/server model. In remote invocation, as opposite of variables and events, the relation between the two services is punctual and it only lasts the execution time of the remote call. This type of primitive allows the use of multiple parameter and a single result value. Remote invocation is useful in these situations where a one-off action has to be taken. •File-based data transfer: This type of communication primitive is used to transfer continuous information in an efficient way. In file-based data transfer the information is sent in chunks in a non reliable way through UDP Transport. File-based data transfer also provides a reliable control mechanism in order the consumer could notify the missing packets to the provider. This type of communication primitive is useful to share any kind of information like images and configuration files. MAREA 9 1.4 System Architecture MAREA describes an architecture based on reusable services that can be distributed over a network of low-cost computing devices. As shown in figure 1.1, two distributed services are running and interacting in two different MAREA instances on the top of the middleware core. MAREA core has been designed according to the following two layer system architecture: service container and network layer. Network Layer Service Container Service 1 Network Network Layer Service Container Service 2 MAREA Core Services Figure 1.1: A high level view of MAREA core layers The network layer, which is explained with more detail in chapter 2, is on charge of translate MAREA protocol messages in streams of bytes and send them through the network. This layer can also undertake the inverse operation: receive a stream of bytes through the network and translate them into MAREA protocol messages. One of the purposes of the network layer is to provide flexibility allowing the use of different transports and encodings depending on the scenario characteristics (e.g. hardware and software limitations). Network layer is a highly reusable component that could potentially send and receive any type of of object over the network. MAREA services are managed and executed by the service container. This component allows message passing between services making communication between both local and remote services transparent, manages the communication protocols and primitives, etc. Service container decouples the services from the core hiding implementation details of some aspects like message management, service location and message delivery. Notice that, only one single service container is executed in each node of the distributed network. The service container delegates the responsibility of managing and processing MAREA protocol messages to a sublayer called protocol layer (chapter 4). This component also manages the exchange of messages in order to discover and use the communication primitives of different services. 16 MAREA 2. Design and Optimization of a Distributed Communications Middleware Network Layer Service Container Battery EC-UPC/IP1/bat1/ Battery Network MAREA Core Battery EC-UPC/IP1/bat2/ Battery Battery */EC-UPC/*/bat1/ Battery BatteryManager EC-UPC/IP1/bat2/ BatteryManager Services Figure 1.2: Services interacting with a query proxy in a multiple battery management scenario According to the implementation of the BatteryManager depicted in listing 1.4 the BatteryManager located in node 2 will consume the communicationprimitives from all the services which match with the query */EC-UPC/*/bat1/Battery (see LocateService attribute in IBattery object). Before following with the example, is necessary to define proxies. Proxy objects are services that implement the same interface (IDU) as the represented service and control the access to the represented service. Proxies just act as redirectors an do not add any extra specific functionality for themselves. As shown in figure 1.2, the service container will generate a proxy object that represent the set of services selected by the query (service in light blue color). In this case the represented service corresponds to the result of the query */EC UPC/*/bat1/Battery, which corresponds to both Battery services. Another kind of proxy used by MAREA middleware is remote proxy. This type of service provides a local representative for a service that reside in a different service container or node than the current one. The functioning of both, remote and query proxies, is detailed in section 3.3.1. 1.7 Conclusions MAREA 2 proposes an architecture based on services that ensures extensibility, flexibility and interoperability across heterogeneous environments. MAREA is a mixed MOM/OOM that provides four MAREA different types of communication primitives to interact and communicate the different services: variable, event, remote produce call and file-based data transfer. MAREA 17 MAREA core has been designed according to the following two layer system architecture: service container and network layer. Service container is responsiblefor managing and executing services following the deployment unit approach. Network and protocol layers are on charge of offering network access and remote message delivery capabilities respectively. The new implemented naming service allows to find, share and access services and its communication primitives hiding the network complexity. 18 MAREA 2. Design and Optimization of a Distributed Communications Middleware NETWORK LAYER 19 CHAPTER 2. NETWORK LAYER The first part of this chapter presents a general overview of the new network system architecture. The following subsections describe each network component with more detail, and present results to the measurements of some performance parameters that represent most critical capabilities and characteristics of the network system architecture. Some of these the key performance parameters are: simultaneous connections, round-trip time (RTT) and memory allocation. The chapters comes to and end with a comparison between the current and previous design of the network layer. 2.1 Architecture The network layer is the lowest level layer on the MAREA stack. This layer it is on charge of providing an optimized, modular and reusable usage of the network capabilities. MAREA network system architecture consist of two main sublayers: encoder and transport. Coder Layer Receive (NetworkMessage,Lane) Serialize (NetworkMessage) Deserialize (NetworkMessage) Send (NetworkMessage) Router UDP TCP Connection Transport Layer Connection Connection Network Message Pool FIFO Figure 2.1: MAREA 2 network layer architecture One one hand, the encoder layer is responsible for coding MAREA protocol messages into byte sequences. This component also undertakes the inverse operation of decode byte sequences into MAREA protocol messages. 20 MAREA 2. Design and Optimization of a Distributed Communications Middleware On the other hand, the transport layer is on charge of send and receive data (byte sequences) from the underlying network through transports (UDP, TCP). The Router is primarily responsible for selecting encoders and transports dynamically by using network lanes (section 2.2.1). This component plays a very important role in offering adaptive capabilities to the network layer and in building a highly reconfigurable system. The idea of the proposed design, in order to improve the performance in terms of speed, is to minimize the amount of time used by the garbage collector to create and destroy object instances. The architecture component called NetworkMessage Pool (section 2.3) achieves this improvement using a memory pool mechanism. In order to provide uniformity and simplicity to the design of each sublayer, the communication between them is done using a common interface. Each of the sublayers send and receive a NetworkMessage entity (figure 2.2) to the upper and lower layers. This common data structure, which contains all the necessary information used by the encoding and transport layers, is modified as it travels downward or upward the architecture. Figure 2.2: NetworkMessage entity Next is detailed the flow of NetworkMessage entities through the entire network system architecture depending on the network lane used. The interaction with different network components (encoder, transport and NetworkMessage Pool) is also explained. •Output lane: Every time a MAREA protocol message is received from the service container a NetworkMessage is dequeued from the NetworkMessage Pool. Then, the MAREA protocol message is stored in the field Object of the NetworkMessage entity. The network output lane is consequently executed and the NetworkMessage entity automatically starts to flow downward the network architecture. First, the encoder sublayer serializes the MAREA protocol message, contained in the field Object of the NetworkMessage entity, and stores the resulting byte stream in the Buffer field of the same NetworkMessage. At the same time, the total length of the serialized data is assigned to the field Offset. Second, the transport layer sends the set of bytes specified by the Offset field from the buffer of the NetworkMessage entity through the network. It is important to note that the network lane takes reference directly to the TCP connection or UDP NETWORK LAYER 21 transport depending on the type of protocol required. Finally, the NetworkMessage is enqueued in the NetworkMessage Pool. •Input lane: Every time a byte stream data representation of a MAREA protocol message is received in the transport layer from the network, a NetworkMessage is dequeued from the NetworkMessage Pool. Then, the byte stream is stored in the field Buffer of the NetworkMessage entity. At the same time, the total length of the received data is assigned to the field Offset. The network input lane is consequently executed and the NetworkMessage entity automatically starts to flow upward the network architecture. First, the encoder sublayer deserializes the set of bytes of specified by the Offset field from the buffer into a MAREA protocol message, which is consequently stored in Object field of the NetworkMessage entity. Second, the network layer forwards the MAREA protocol message stored in the field Object of the NetworkMessage to the service container. Finally, the NetworkMessage entity is enqueued in the NetworkMessage Pool. In relation to the transport layer, when a message is received the fields StatusCode and TransportAddress are set in order to inform the upper layers about the reception status and the source of the incoming MAREA protocol message. The field Id of the NetworkMessage entity is a byte code identifier used by the encoder layer to encode and decode MAREA protocol messages. This identifier is also reused by the service container in order to process the MAREA protocol message according to the type of message and the protocol (discovery, publish-subscribe, rpc) which belongs. 2.2 Router MAREA is able to use different encoders and transports in order to build a modular and configurable network architecture. The Router has the ability to select the elements (encoders and transports), of the layered network architecture, at execution time depending on needs and the state of the network. The main aim of this element is to create and manage the network lanes. 2.2.1 Network Lanes A network lane can be defined as a set of references to the bindings establish between the different network architecture elements (encoder and transports) used at a particular moment. Lanes have been implemented as linked lists of delegates or multicast delegates. A delegate isan object that allowsthe programmer to encapsulatea reference to amethod. A delegate is similar to a function pointer in C or C++ but is object-oriented, type-safe, and secure [7]. 22 MAREA 2. Design and Optimization of a Distributed Communications Middleware The following characteristics of multicast delegates have been taken in account to create network lanes: •The invocation list of multicast delegates is called synchronously and orderly. •If an exception occurs in a delegate, the remaining delegates of the list are not invoked. According to the figure 2.2 the Router use two different network lanes to send and receive data, output and input lanes respectively. Lanes Invocation List Output lane MareaCoder.Serialize(NetworkMessage m) Transport.Send(NetworkMessage m) Input lane Coder.Deserialize(NetworkMessage m) Container.Receive(NetworkMessage m) Table 2.1: Output and input network lanes A way to trap link loss or disconnection exceptions is required in order to notify the upper layers that an error has occurred. There exist two different solutions to this issue: •Use the method GetInvocationList to get each individual delegate from the multicast delegate and invoke each delegate within the try block of an exception handler. This solution is very powerful but it has counterparts like the use of system resources and execution time due to the handling of exceptions. •Use a field in the NetworkMessage entity as a status code (figure 2.2). The second alternative has been implemented in order to accomplish with the objective of improve the performance. 2.3 NetworkMessage Pool The non deterministic process of garbage collection is executed .NET virtual machine in order to maintain the memory clean. This process can introduce non deterministic pauses into the execution of a program which are not correlated with the algorithm being processed. One solution in order to reduce garbage collection interruptions is use a memory pooling mechanism. The main idea of this type of mechanisms is to providea managed set of functions in order to allocate and deallocate memory. Pooling mechanisms keep references to object instances that are beyond destruction, allowing it to be reused when needed. With NETWORK LAYER 23 this technique no objects (NetworkMessage entities) are released to be garbage collected until the middleware is shut down. The proposed design to implement a memory pool mechanism is to use a FIFO queuing discipline for NetworkMessage entities. In this common queue disciple, the elements are added to the tail and removed from the head using a pair of object references to the tail and the queue. The final purpose of the NetworkMessage Pool is to reduce the amount of work that has to be done by the garbage collector and consequently minimize the time used by its own execution. 2.3.1 Results A memory allocation profiling test has been executed in order to evaluate the behavior of the NetworkMessage Pool. The figure 2.3 and the table 2.2 present the total bytes allocated by MAREA 1 and the MAREA 1 network backport (section 2.6) during an echo request/response test with two MAREA instances. Each instance runs a different service which sends or responds to the message. This results correspond to the total bytes allocated by the requester. The test has been executed 10000 times to send variables (UDP) and events (TCP) primitives with a total payload of 1000 bytes and frequency of 100 Hz. MAREA 1 network backport has been tested in two different modes: reusing network lanes and creating every time on demand. MAREA 1 MAREA 1 Backport [Lane Reuse] MAREA 1 Backport [Lane] 100 102 104 106 108 1010 1012 Memory Allocation [byte] UDP TCP Figure 2.3: Number of bytes allocated by MAREA 1, MAREA 1 network backport in an echo test (requester side): 1000 Bytes, 10000 times, 100 Hz 24 MAREA 2. Design and Optimization of a Distributed Communications Middleware Bytes Allocated Middleware UDP TCP MAREA 1 15222996197 15159086765 MAREA 1 Backport [Lane Reuse] 74992540 8092972 MAREA 1 Backport [Lane] 98911888 23693442 Table 2.2: Number of bytes allocated by MAREA 1, MAREA 1 network backport in an echo test (requester side): 1000 Bytes, 10000 times, 100 Hz 2.4 Encoding Layer The encoding layer is on charge of serializing and deserializing MAREA messages. This layer provides an abstraction layer such all logic above contained in the upper-layers does not need to know the particulars about how messages are serialized and deserialized. Serialization is the act of taking an in-memory object or object graph (set of objects that reference each other) and flattening it into a stream of bytes [8]. The reverse operation is deserialization which takes a data stream and regenerates into an in-memory object or object graph. 2.4.1 Previous Work The initial version of MAREA has been implemented with the idea of providing several encoding layer implementations (XML serialization, binary serialization and MAREA coder) in order to allow adaptability and interoperability between the devices and the network. The first two implementations of the encoding layer use .NET Framework serialization mechanisms such as binary serialization through BinaryFormatter class, and human-readable XML serialization through XMLSerializer class. Feature BinaryFormatter XMLSerializer Level of automation ***** **** Type coupling Tight Loose Version Tolerance *** ***** Can serialize nonpublic fields Yes No Preserves the object reference Yes No Suitability for interoperable messaging ** *** Flexibility in reading/writing XML files - **** Compact output **** ** Performance **** * To *** Table 2.3: Serialization engine comparison between .NET BinaryFormatter and XMLSerializer [8] NETWORK LAYER 25 The BinaryFormatter is easy to use and automatic, but it is not such flexible as XMLSerializer. On the other hand, XMLSerializer is slower and less powerful because it is not able to restore shared object references. Furthermore, XML serialization does not convert private fields, indexers, methods, or read-only properties (except read-only collections). In order to do this, is mandatory to use the BinaryFormatter class. One of the main drawbacks of these two mechanisms is the performance overhead. Serializing a message with BinaryFormatter is expensive because of the metadata present. This is more noticeable in XMLSerializer because the overhead introduced by the XML tags is bigger. Another disadvantage of these two mechanisms is the interoperability between the different virtual machine representations of .NET Frameworks. For instance, XML serialization is not available on the Micro Framework and binary serialization works different in .NET Framework and .NET Compact Framework. The last implementation of the encoding layer, which is called MAREA coder, has been designed in order to solve these two drawbacks controlling the serialization and deserialization of the different types. This technique, allows the programmer to have more control over the serialization and deserialization processes and ensures serialization compatibility. The first implementation of this encoder was made using introspection to serialize and deserialize each message dynamically. The results of this first approach were not satisfactory in terms of speed because introspection it is a slow process. Custom serialization solves this issue by using specific methods or routines to serialize and deserialize specific MAREA messages. MAREAcoderhasbetterperformanceinterms of speedandserializeddata sizethan .NET BinaryFormatter and XMLSerializer implementations. MAREA coder is not dependent of the .NET Framework, so the interoperability between different virtual representations of the .NET Frameworks is not a problem like in .NET BinaryFormatter and XMLSerializer implementations. 2.4.2 Drawbacks MAREA coder has some drawbacks inherited from custom serialization like complexity, especially in those cases like tree of objects or object graphs that might contain cycles. In these cases the code could be really hard to study. Another point that has to be taken in account of this approach is development speed. Custom serialization does take time for testing, developing and maintenance. For instance, if some messages are added or modified in the protocol layer, the corresponding methods to serialize and deserialize these messages must be added or modified too in order the encoder layer continues to work properly. MAREA coder has been designed with two serialize and deserialize entry point methods that implement a large switch statement to get the type of object that has to be serialized/deserialized. A large switch statement means the method is large, hard to read and 32 MAREA 2. Design and Optimization of a Distributed Communications Middleware UDP Transports RTT (ms) Asynchronous Synchronous 25 0.1265 0.1092 50 0.1885 0.1395 75 0.2368 0.2097 100 0.2742 0.2354 150 0.3617 0.38550 175 0.4409 0.5431 250 0.5952 0.7966 Table 2.7: Mean round-trip times for an echo test using isolated synchronous and synchronous UDP transports: 1000 Bytes, 10000 times, 100 Hz According to the results, the round-trip time tends to grow exponentially with the number of simultaneous connections. The round-trip becomes lower in asynchronous mode, in comparison to synchronous mode, from around 150 simultaneous transports. 2.6 MAREA 1 Network Backport The whole new MAREA 2 network architecture has been backported to MAREA 1 in order to ensure its proper functioning. Backporting is the action of taking a certain software modification (patch) and applying it to an older version of the software than it was initially created for [10]. This software backport also provides a fair and realistic way to compare the performance between the old and new network architecture. 2.6.1 Results MAREA 1 and MAREA 1 network backport have been compared using a round-trip test. The figure 2.7 presents the round trip time distribution during an echo request/response test with two MAREA instances. Each instance runs a different service which sends or responds to the message. The test has been executed 10000 times to send variables (UDP) with a total payload of 1000 bytes and frequency of 100 Hz. The mean round-trip time is 0.2399 and 0.433 ms for MAREA 1 network backport and MAREA 1. The standard deviation is 0.1756 and 0.3813 ms respectively. The same test has been done using events (TCP). The mean round-trip time is 0.3501 and 0.6564 ms for MAREA 1 network backport and MAREA 1.The standard deviation is 0.2206 and 0.4061 ms respectively. NETWORK LAYER 33 Figure 2.7: Round-trip time distribution for an echo test of MAREA 1 and MAREA 1 network backport using variable primitives: 1000 Bytes, 10000 packets, 100 Hz 2.7 Comparison with MAREA 1 Network Architecture MAREA 1 network system architecture consist of three main sublayers: encoder, transport and Lane Manager. Coder Layer UDP TCP Connections Gateway Lane Manager Transport Multiplexor Transport Layer Figure 2.8: MAREA 1 network layer architecture 34 MAREA 2. Design and Optimization of a Distributed Communications Middleware One of the main differences between the old and new MAREA network designs is the top entry point of the architecture which is called Lane Manager. This component is the responsible to control which transports and encoders are used by the middleware at a particular moment. In contrast, the Router is the responsible for performing this task in the new architecture. As opposite of the new architecture, network lanes (subsection 2.2.1) are not a set of references to the bindings establish between the different network architecture elements. Instead of this, there are object references (copies) to these elements. The main idea of taking out the Lane Manager in the new architecture is to simplify the design and improve the speed by removing unnecessary repeated searches to the lane, which in normal conditions is always the same. Every time a message is sent in MAREA 1 network architecture, a search has to be done into dictionary in order to get the default lane. Figure 2.9: Lane Manager class diagram The new design does not use a dictionary in order to avoid slowdowns in the performance. Instead of this, the network layer use only one output and input lane for the outgoing and incoming data respectively. These lanes are reused while the network continues to work properly. The service container, which is the immediate upper layer, is the responsible for keeping and providing the hint to the network layer every time. Only if a network change happens, new lanes are demanded to the Router. This solution is more complex but reduces the acquisition time of the lanes. In the transport sublayer, lanes also take profit of its new approach by taking reference directly to the TCP connection which is required in a particular moment instead of searching it in a dictionary. Another big difference between the two architectures is the sublayer architecture design. For one hand, the MAREA 2 network sublayers (figure 2.1) are uniform. In MAREA 1 the network sublayers (figure 2.8) are inconsistent because every single sublayer presents different interfaces in order to communicate with to the upper and lower sublayers. On the other hand, in the old architecture every sublayer is dependent of the upper and lower one because it has to keep a reference in order to communicate with them. The new sublayer architecture is more independent and flexible, because the responsibility of manage the bindings between the different sublayers is delegated to the network lanes instead of the sublayers by itself. Another difference, that has been mentioned before, is the pooling mechanism used in the new architecture which is explained in section 2.3. NETWORK LAYER 35 In MAREA 1, the Gateway module is on charge of interconnect networks that use different types or protocols and architectures. Its main goal is the translation of the source network protocol to the destination network protocol, and vice versa, in order to allow the communication between them. In MAREA 2, this task is accomplished by the Router. 2.8 Conclusions Network layer provides an optimized, modular and reusable usage of the network capabilities. MAREA network system architecture consist of two main sublayers: encoder and transport. The Router component together with the network lane approach builds a highly reconfigurable and adaptive architecture. Most of the optimizationwork isfocused in components of thenetwork system architecture: encoder and transports, which are the most delay-sensitive layers of the whole architecture. The global performance of network sublayers has been improved implementing the following new features: •A serialization code generation utility called MAREAGen which generates classes automatically with methods to serialize and deserialize MAREA messages in order to take advantage of the custom serialization benefits. •A memory pooling mechanism to minimize garbage collection interruptions. •New transport implementations with asynchronous sockets which are more scalable than traditional synchronous sockets. 36 MAREA 2. Design and Optimization of a Distributed Communications Middleware SERVICE CONTAINER 37 CHAPTER 3. SERVICE CONTAINER The first part of this chapter sketches out the new version of the service container. Then, the Service Manager and remote services are presented. The last sections present some services used by the middleware, and make a brief comparison between the previous and new version of the service container. 3.1 Architecture Service container, which is the highest level layer on the MAREA stack, consist of two main components: protocol layer and Service Manager. Protocol layer controls the exchange of MAREA protocol messages in order to discover and implement the communication primitives of remote services. This layer implements three different protocols: discovery, publish-subscribe and remote procedure call, which are detailed in chapter 4. Network Layer Service Container Service Manager Services Running Receive (NetworkMessage) Proccess (Message) Send (TransportAddress, Message) Query Manager Loader IDU SDU Protocol Proxies Services D I S C O V E R Y P U B / S U B R P C Service 1 Service n Figure 3.1: A high level view of service container Service Manager is on charge of control the startup and shutdown the services at any moment during MAREA execution. To perform this task the service manger provides the 38 MAREA 2. Design and Optimization of a Distributed Communications Middleware following set of service collections: •Running: Contains the local services that have been started and are actually running in the service container. •Proxies: This type of services are proxy objects that could represent a remote service or set of services selected by a query. •Services: This collection contains references to all the services (running and proxies). Services operate following the Inversion Of Control paradigm. The term Inversion of Control (IoC) is a computer programming technique wherein the flow of the control of an application is inverted. Rather than a caller deciding how to use an object, in this technique, the object called decides when and how to answer the caller, so the caller is not in charge of controlling the main flow of the application [11]. Services do not provide a main() method that starts the ball rolling and procedurally calls methods to send and receive. Instead, service container is responsible for instantiating running and controlling the entire life cycle of services. Services only allow to specify different configuration aspects like what to receive, from whom, and how to process it. Another of the functions regarding service management is service loading. This task is accomplished dynamically at the beginning of the middleware’s execution. All MAREA services are loaded in such a way that information about the service description (IDU) and implementation (SDU) is cached. The Query Manager provides methods to manage and search services that match with queries. This component also provides some functionalities for the creation and retrieval of query proxies used to implement the naming service. 3.1.1 Relationship Between Service Container And Network Layer As shown in figure 3.1, service container provides to different methods to receive and send the incoming and outgoing protocol messages. First, a NetworkMessage entity is passed as a parameter in Receivemethod. As explained in section 2.1, the NetworkMessage contains the protocol message inside the field called Object. In this way, every time a NetworkMessage is received, the service container process the protocol message inside of it, and passes it subsequently to the specific protocol through the corresponding Process method. In addition, during the protocol message serialization stage, the encoder layer is responsible for setting the field Id of the NetworkMessage entity according on the protocol message type. This identifier is also reused by the service container in order to address the received protocol message to the specific Process protocol method. Notice that, service container Receive function is the last method inside the network input lane, which is a multicast delegate (table 2.1). SERVICE CONTAINER 39 On the other hand, Send method is called from the protocol layer to transparently pass MAREA protocol messages from the protocol layer to the network layer. 3.1.2 Relationship Between Service Container And Protocol Layer Asexplainedin section2.4.3, MAREA protocolmessageidentifiers areassignedby MAREAGen with values from 0 to 63 in order to reuse them in the service container (table 2.4). The service container contains an array of delegates used to process each MAREA protocol message according to its identifier. The position inside of the array represents the protocol message identifier, while the object stored inside is the delegate. This delegate basically points to the Process method used to handle the specific incoming MAREA messages according to the protocol type (discovery, publish-subscribe or remote procedure call protocol). Service container provides two different methods to add and remove the delegates from the array. These methods are automatically called inside the start and stop methods of the specific protocol. The final aim of this approach is to extend the use of delegates in the service container in order to reduce the coupling and improve maintainability in the upper layers. This new design also reduces the degree of complexity of adding more protocols in the service container. 3.2 Service Manager The Service Manager does basically two different functions that are going to be described in the following subsections: service loading and service start up and shutdown. 3.2.1 Service Loading Service Manager is responsible for caching information about the interface (IDU) and implementation (SDU) of services during the system startup. This information is necessary to manage services and the communication primitives during the middleware execution. Service Manager searches for services in assemblies in order to carry out the service loading. One of the tools used to manage assemblies, types and namespaces in MAREA is the assemblies manager. This component, which is also used used by MAREAGen, has been implemented as a cache with the idea to improve the performance. 40 MAREA 2. Design and Optimization of a Distributed Communications Middleware 3.2.2 Service Start Up And Shutdown Once the service loading has been done, the Service Manager reads an XML which includes the list of services that will be automatically started by the middleware. However, any MAREA service could be started or stopped subsequently while the interface and implementation of the service exist in the SDU and IDU collections. Thefollowingsubsectionexplainsoneof the steps accomplishedduringtheservicestartup: the creation of communication primitives. 3.2.2.1 Communication primitives management Service Manager is responsible for creating and getting variable and event primitives from services. As shown in listings 1.2 and 1.1 this type of primitives are implemented using generics. Generics make possible to design classes and methods that defer the specification of one or more types until the class or method is declared and instantiated by the programmer. To create generic type instances at run time is necessary to use reflection. On the one hand, reflection provides extensibility to applications by allowing them to see theirown innerworkings. But onthe other hand, reflection isa slowand expensiveprocess. The proposed solution to solve this problem is to implement our own cache on top of the one that exists in the .NET Framework (Pobar, 2005)[12]. The attributes and metadata of communication primitives is stored together with the specific implementation of the service (SDU). The idea is to minimize the reflection operations and execute them at the service loading phase, during the middleware startup process. Variables and events provide a common interface to control and manage primitives. For one hand, this interface implements methods to subscribe and unsubscribe the primitive to specific methods in order to receive or not new values of the communication primitive using a callback pattern. Notice that these two operations are executed when services are started and stopped (listing 1.4) Subscribe and unsubscribe functionalities are used to add or remove a method from the invocation list of a multicast delegate respectively. This specific method has to accomplish with the following signature: a string which represent the address of the primitive and an object, of the same type as the generic type of the primitive, which carries new primitive values (see AmperageChanged and LowBatteryChanged BatteryManager methods in listing 1.4). For the other hand, this interface provides a Notify method to inform to all the subscribed services about the new data. When the Notify method is called all the methods inside the delegate’s invocation list are fired (see Run Battery method in listing 1.3). The figure 3.2 shows the interaction between the communication primitives of one BatteryManager and three Battery services. The BatteryManager is consuming the Amperage variable and the Low Battery event from all the Batteries of the current subsystem. Notice SERVICE CONTAINER 41 that the Battery service with the address EC-UP/IP2/Bat2/Battery is a remote proxy which represent a remote service (node field info is different from the other services). Service Manager Proxies IDU SDU IBatteryManager EC-UP/IP1/bat1/Battery EC-UP/IP1/bat2/BatteryEC-UP/IP2/bat2/Battery AmperageChanged GlobalBatteryWarning Amperage LowBattery Amperage LowBattery Amperage LowBattery Notify Notify Notify IBattery BatteryManager Battery Running EC-UP/IP1/man1/ BatteryManager Communication Primitive Event-Variable Service Service Container Figure 3.2: Interaction between communication primitives and services from Service Manager’s point of view in a battery management system 3.3 Proxy And Remote Consumer Services The following section describes some of the services involved in the communication between remote consumers and proxy services. Figure 3.3 shows the scenario of a battery management system with four service containers. In this example, blue color is used for representing services that are running in service containers, while light blue an grey colors are used for representing remote producer proxies (section 3.3.1.1) and remote consumers (section 3.3.1.2) respectively. Each container is running one service: one Battery and three BatteryManager. Proxies and remote consumers services are used to share communication primitives between services that are actually running in different service containers. The following subsections describe both proxy and remote consumer services. 48 MAREA 2. Design and Optimization of a Distributed Communications Middleware 3.6 Conclusions The service container consist of two main components: protocol layer and Service Manager. Service Manager is on charge of service loading and service start up and shutdown. Protocol layer controls the exchange of MAREA protocol messages in order to discover and subscribe the communication primitives of remote services. The implementation and internal details of the protocol layer are presented in the next chapter. MAREA provides to end users a console an a GUI service to manage and monitor the service container. The service container implements services following the service deployment approach and interacts with them following the Inversion Of Control paradigm. Proxies and remote consumers services are used to share communication primitives between services that are running in different service containers. PROTOCOL LAYER 49 CHAPTER 4. PROTOCOL LAYER This chapter introduces the different protocols used by the middleware to dynamically discover remote services (those which reside in a different service containers) and consume their communication primitives. The different type of messages and some the most important features of each protocol are presented in the following subsections. Some improvements made in all the whole protocol architecture are introduced at the end of this chapter. 4.1 Discovery Protocol The main purpose of this protocol is to discover and advertise services using a dynamic and non-centralized mechanism. Discovery protocol behaves actively to request services and also passively to listen service announcements by exchanging different types of discovery messages (red messages in figure 4.1). BatteryManager (Container A) Battery (Container B) BatteryManager (Container C) PUBLISH (@Control B) PUBLISH (@Control B) SUBSCRIBE (@Control A, @Data A) DISCOVER PUBLISH (@Control B) SUBSCRIBE-ACK (@Data A) SUBSCRIBE (@Control C, @Data C) SUBSCRIBE-ACK (@Data C) DATA DATA UNSUBSCRIBE (@Control A, @Data A) DATA DATA UNPUBLISH (@Control B) Figure 4.1: Discovery and subscription protocol message exchange bettwen a Battery and two BatteryManager services into different service containers As shown in figure 4.1, the discovery protocol is divided in two phases: the discovery and advertisement of services (discover and publish messages) and the publish service termination (unpublish message). In contrast to MAREA 1, this new implementation discovers and announces services instead of the communication primitives. This reduces considerably the message traffic in 50 MAREA 2. Design and Optimization of a Distributed Communications Middleware the network during the initial discovery stage. 4.1.1 Messages This subsection makes a brief description of the different types of messages used in discovery protocol. 4.1.1.1 Discover This type of message is used to request services that are actually running in other containers. Discover message could request a single or group of services depending on the type of address is used (single or query, subsections 1.5.1.1 and 1.5.1.2 respectively). This type of message is broadcasted using the UDP protocol. Considering a Battery is running in container B, as shown in figure 4.1, if a BatteryManager is started afterwards in container C, a discover broadcast message is sent requesting a service or a group of services according to the address contained in the LocateService attribute of the object Battery inside the BatteryManger SDU (listing 1.4). In case of the requested service address is a query, all the containers will respond to the discover message with a publish message for each of the running services that actually match with the given query. Otherwise, if the requested service address is not a query, a single publish message will be sent by the container which actually owns the requested service. The discovery protocol retransmits discover message periodically, for each service, as long as there is no service which publishes one or more communication primitives that are consumed by the given service. 4.1.1.2 Publish Publish message is used to advertise a service that is actually running in a container. This type of message, which contains the control transport address of the container that offers the service, is broadcasted using the UDP protocol. Publish messages are used in one of these two possible scenarios: •To advertise a specific service that has been requested through a discover message (see discover and publish message exchange between container B and C in figure 4.1). •When a service is started (see first broadcast publish message in container B of the figure 4.1). PROTOCOL LAYER 51 4.1.1.3 Unpublish This type of message is used to notify that a service has been stopped. This implies that all its communication primitives have stopped publishing information. Similarly to publish and discover messages, unpublish message is also broadcasted using the UDP protocol. 4.2 Publish-Subscribe Protocol Publish-subscribe protocol is on charge of manage the subscriptions and data transfer of the variable and events primitives. Unlike discovery protocol, publish-subscribe protocol works at communication primitive level instead of service level. As shown in figure 4.1, the publish-subscribe protocol (blue messages in figure 4.1) is divided in three phases: the subscription (subscribe and subscribeACK messages), the primitive data transfer (data messages) and the unsubscription (unsubscribe message). Subscription and unsubscription messages are sent in a reliable way through TCP protocol. On the other hand, the protocol used to transport data messages depends on the type of primitive, which is TCP and UDP for events and variables respectively (section 1.3). 4.2.1 Messages This subsection makes a brief description of the different types of messages used in publish-subscribe protocol. 4.2.1.1 Subscribe Subscribe message isused to specify the primitivewhich wantsa serviceto be subscribed. This type of message, besides the address of the primitive, specifies the control address of the service container which requires the subscription and a single or a set of data address to receive the incoming primitive. 4.2.1.2 SubscribeACK SubscribeACK message is sent as a response of remote subscription request (subscribe message). The main purpose of this message is to negotiate the address used in the future data transfer of the primitive. This message is used to confirm the transport data address, if more than one have been offered in the request (subscribe message). 52 MAREA 2. Design and Optimization of a Distributed Communications Middleware 4.2.1.3 Unsubscribe This type of message is used to end the subscription of a primitive. The aim of unsubscribe messages is to inform to the publisher, the service that holds the primitive, that a service subscribed to the primitive is not longer a consumer. As shown in figure 4.1, unsubscribe messages specify the control and data transport address of the cosumer. 4.2.1.4 Data This message is used to send data to those services that have been subscribed to the primitive. Data messages are sent systematically due to the nature of the variables and eventsprimitives(section1.3), which are bothused toshare periodic information. This type of message contains three different fields: address, type and data value of the primitive. 4.3 Remote Procedure Call Protocol Remote procedure call protocol implements a point-to-point synchronous communication model using the remote invocation primitive. As explained in section 1.3, remote invocation follows a request/reply pattern using a client/server model. One of the features added in the new version of this protocol is the support of session tokens as a way to avoid reply attacks. To allow the client to assign a certain result to a previous request, the client assigns a token to each request. The server always returns this token together with the result so that the client can easily associate a result with the corresponding previous request. 4.3.1 Messages The remote procedure call (RPC) paradigm is implemented through the exchange of call and reply function messages. Both type of messages are sent reliably by using the TCP protocol. In the figure 4.2, a Battery and BatteryManager are running in different service containers. The BatteryManager service offers the method Recharge according to its IDU definition. BatteryManager (Container A) Battery (Container B) CALL FUNCTION (Name, Id, Params[], Reply @, Token) Recharge(bool) RETURN FUNCTION (Id, Result, Token) Figure 4.2: Remote procedure protocol message exchange bettwen a Battery and a BatteryManager services into different service containers PROTOCOL LAYER 53 The BatteryManager (container A) starts the communication by sending a call function message to make the remote invocation call of the method Recharge of the service Battery (container B). Once the execution of this method has been done, a reply function message is returned with the results of the procedure’s execution. 4.3.1.1 Call Function The call function message contains the following fields: •Name: Contains the name of the remote method. •Identifier: Identifies the procedure with a random number. •Parameters: Contains an array with the parameters of the call. As an inherited limitation of using the .NET Func and Action parameterized delegates, the maximum number of parameters is sixteen. •Reply address: Contains the address of the procedure’s caller. •Token: A session token that the caller will transmit as part of the future response. 4.3.1.2 Reply Function The reply function message contains the result of the procedure’s execution together with the identifier and the token of the call function message. This message is returned to the remote call procedure requester according to the field reply address of the call function message. 4.4 Improvements One of the problems with MAREA 1 is that all the protocols and its functionalities are included inside the service container. The goal of the new implementation of the protocol layer is to decouple the different protocols into smaller independent modules. This approach makes protocols easier to modify by decomposing them around smaller design decisions. The different protocols have been implemented in independent classes with two different methods for each type of message of the specific protocol. The idea is to use one them to process the incoming MAREA messages from the service container and the other to build and send the MAREA messages to the service container. Each protocol offers a start and stop method in order to make the whole protocol layer highly configurable. The different protocols could be used by calling these methods, depending on the scenario and requirements of the system. The implementation of these two methods basically adds or removes the different delegates from the array located in service container. 54 MAREA 2. Design and Optimization of a Distributed Communications Middleware 4.5 Conclusions In contrast to MAREA 1, MAREA 2 implements a new component called protocol layer. Protocol layer is on charge of offering remote message delivery capabilities according to the communication protocol used for each communication primitive. Protocol layer allows to easily plug additional protocols for implementing other primitives (e.g. file-based data transfer) and other communication features. The protocol layer implements three different protocols: discovery, publish-subscribe and RCP. Firstly, discovery protocol, as its name suggests, aims to discover and an announce services. Secondly, publish-subscribe protocol is on charge of manage the subscriptions and data transfer of the variable and events primitives. Finally, RPC protocol defines a point-to-pointsynchronous communicationmodelbasedonarequest/reply pattern in order to implement remote invocation protocol. CONCLUSIONS 55 CHAPTER 5. CONCLUSIONS Thischapterprovidesa finalanalysisofthemasterthesis. Thefirst sectionspresentresults and the different conclusions: the project conclusions and the personal conclusions. This chapter comes to an end with some of the future lines of work and the environmental impact. 5.1 Results Although MAREA was not designed for hard real-time applications, one of the objectives of this master thesis is to optimize, evaluate and compare the performance between MAREA 1 and MAREA2 middlewares. For this evaluation, the performance analysis consists in communicate two services that are deployed in two different service containers. The communication between them is accomplished following an echo request/response exchange-pattern. The requester service starts a timer and sends a message with a timestamp to the replier service. The replier service simply returns the message to the requester. Then, the requester calculates the round-trip time with the actual time and the timestamp of the message. 5.1.1 Round-Trip Time The test has been executed 10000 times to send variables (UDP transport) and events (TCP transport) with a total payload of 1000 bytes and frequency of 100 Hz. The chosen frequency is 100 Hz, because in avionics the typical requirements are in the 20-100 Hz range. The test has been executed with the following three versions of MAREA middleware: MAREA 1, MAREA 1 network backport and MAREA2. In relation to variables, the mean round-trip time is 0.8313, 0.2797 and 0.3587 ms for MAREA 1, MAREA 1 network backport and MAREA 2 respectively. In events, the mean round-trip time is 0.9422, 0.3922 and 0.6481 ms for MAREA 1, MAREA 1 network backport and MAREA 2 respectively. The table 5.1 shows the standard deviation of the RTT results. In both variables and events the mean RTT has been reduced a 56.93% and 31.21% respectively. Middleware Variable-RTT (ms) Event-RTT (ms) Mean STD Mean STD MAREA 1 0.8313 0.6594 0.9422 0.5453 MAREA 1 Backport 0.2797 0.2992 0.3922 0.5221 MAREA 2 0.3587 0.4081 0.6481 0.5737 Table 5.1: Mean round-trip times for an echo test using MAREA 1, MAREA 1 network backport, MAREA 2: 1000 Bytes, 10000 times, 100 Hz 56 MAREA 2. Design and Optimization of a Distributed Communications Middleware Variable [UDP] Event [TCP] 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 RTT [ms] MAREA 1 MAREA 1 Backport MAREA 2 Figure 5.1: Mean round-trip times for an echo test using MAREA 1, MAREA 1 network backport, MAREA 2: 1000 Bytes, 10000 times, 100 Hz 5.1.2 Memory allocation The number of bytes allocated by the different versions of the middleware has been also measured during the execution of the same test in the requester side. Notice that the y axis of the figure 5.2 is in logarithmic scale. In relation to variables, the total number of bytes allocated are 15834797234, 85587107 and 139311411 bytes for MAREA 1, MAREA 1 network backport and MAREA 2 respectively. In events, the total number of bytes allocated are 14805029654 , 54609735 and 122285449 bytes for MAREA 1, MAREA 1 network backport and MAREA 2 respectively. In both variables and events the total number of bytes has been reduced a 91.2% and 99.17% respectively. Middleware Memory Allocation (bytes) Variable Event MAREA 1 15834797234 14805029654 MAREA 1 Backport 85587107 54609735 MAREA 2 139311411 122285449 Table 5.2: Number of bytes allocated by MAREA 1, MAREA 1 network backport, MAREA 2 in an echo test (requester side): 1000 Bytes, 10000 times, 100 Hz CONCLUSIONS 57 Variable [UDP] Event [TCP] 100 102 104 106 108 1010 1012 Memory Allocation [byte] MAREA 1 MAREA 1 Backport MAREA 2 Figure 5.2: Number of bytes allocated by MAREA 1, MAREA 1 network backport, MAREA 2 in an echo test (requester side): 1000 Bytes, 10000 times, 100 Hz In MAREA 2 the number total of bytes allocated and the mean RTT has been significantly reduced from MAREA 1, for both variables and events primitives. The number total of bytes and the mean RTT results for MAREA 1 network backport are slightly lowercompared toMAREA 2. One plausibleexplanation couldbe that in MAREA2, the service container should manage one additional proxy service to support the queries. This additional service may introduce some delay due to the propagation of variable and event primitives. The other reason is that no specific performance optimization work has been done in the service container. In this sense, there is still room for performance improvement in the service container. 5.2 Project Conclusions The specified objectives at the start of the project have been successfully reached. A new version of the middleware MAREA 1, has been designed and implemented. The enhancements provided by MAREA 2 design is a more modular, flexible and reusable architecture. The new design offers a starting point for providing reflective and adaptive capabilities, and builds a highly reconfigurable system. The use of delegates both in the service container and network layer reduces the coupling and improves maintainability of the different sublayers. Based on the first feedback received from developers, at least regarding the network layer, MAREA 2 programming complexity has been reduced com- TOOLS 1 APPENDIX A. TOOLS A.1 Package management system A package management system, is a collection of software tools to automate the process of installing, upgrading, configuring, and removing software packages. It typically maintains a database of software dependencies and version information to prevent software mismatches and missing prerequisites. A.1.1 NuGet NuGet is a free, open source developer focused package management system for the .NET platform intent on simplifying the process of incorporating third party libraries into a .NET application during development. The NuGet tools provide the ability to create, publish and consume packages. Each packages consists in a nupkg (NuGet package) file which contains packaged source code or libraries that can be used for any developing program components. A.1.2 Package Management Manage package references in projects becomes very simple with the NuGet extension, which is included by default in Visual Studio 2012. The packages can be added, updated and removed in projects using the Manage NuGet Packages dialog box. Figure A.1: Manage NuGet Package option in right click menu project Figure A.2: MAREA service package management in Manage NuGet Packages dialog box A.1.3 Package Creation The first step to create a NuGet package is to configure a nuspec (NuGet specification) file. Nuspec files are manifests in XML format that specify all the settings and the package dependencies. The metadataof the nuspec file contains thefollowingfields: id, version, authors (collection of author elements), description, language, tags (collection), licenceUrl (Uri), projectURL (Uri), iconUrl (Uri), requrireLicenceAcceptance(boolean)and a set of dependencies. Each dependency element has the following attributes: id (the ID of a package that this package depends on) and version (the required version of the dependency package. The figure A.3 shows an example of nuspec file. Notice that the nuspec file is in the root folder of the project. Figure A.3: Nuget specification (nuspec) file from a MAREA service project The following command creates a package(nupkg file) from a given project and a nuspec file: nuget pack "ProjectName".csproj A.1.4 Package Publication Once the package has been created, it needs to be pushed to a NuGet server A.1.6.3. Depending on the NuGet server configuration, an access key could be required to publish and delete packages. The following command pushes a package to a Nuget server: nuget push "PackageName".nupkg -s "NugetServerURL" "AccessKey" A.1.5 Package Managment Automation An external Visual Studio tool has been created in order to facilitate to developers the creation and publication of his own packages. With this tool the developers are able to create and push services just by selecting the project and clicking the option MAREA 2 package source->Create and Push service on the Tools menu. Figure A.4: Visual Studio Create and Publish package external tool This tool has been created specifying the following command and arguments: •Command:C:/WindowsMicrosoft.NET/Framework64/v4.0.30319/MSBuild.exe •Arguments:/t:Build,Package,Publish/p:Configuration=Debug;NuGetServer=http://localhost:7073; NuGetKey=3237c377-4253-4b48-91ec-7b5457df28d5$(ProjectDir)$(ProjectFileName) Figure A.5: Create and Publish package external tool details A.1.6 Third Party Packages This subsection includes the different third party libraries that have been used to develop MAREA. All of them have been obtained and installed through the NuGet package management system. A.1.6.1 Log4net Log4net, a port of the popular Java library log4j, is an open source library that allows .NET applications to log output to a variety of sources (e.g., console, files or SMTP). The information is logged via one or more loggers which provide a the following five logging levels: debug, information, warnings, errors, fatal. A.1.6.2 NUnit NUnit, a port from JUnit, is a unit-testing framework for all .NET languages. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit does not support Visual Studio integration. Instead of this it provides an external program compiled either as a console app or a GUI. This program is able to run and execute the unit tests from an assembly. Figure A.6: MAREA unit tests executed by NUnit GUI application This framework has been especially used to test encoder layer, naming and service management functionalities. The listing A.1 shows an example of a unit test to serialize and deserialize a double with two diferent pair of parameters. Listing A.1: MAREA encoder layer unit test: serialization and deserialization of a double private byte [ ] seralizedData =null ; private long start ,serializeTicks,deserializeTicks; private long clock_freq =PerformanceTimer.Clock_freq () ; [SetUp ] public void RunAfterAnyTest () { serializeTicks = 0; deserializeTicks = 0; } [TestCase(0.100000234523, 0) , NUnit .Framework.Description ("Coder( double , System . Double ) " ) ] [TestCase (double.MaxValue ,0) ] public void TestDoubleM2(double oDouble ,double rDouble) {for (int i= 0; i<CoderTestsConstants .CODIFICATIONS;i++) { start =PerformanceTimer.Ticks () ; seralizedData =AdaptedMareaCoder .Send(oDouble) ; serializeTicks += PerformanceTimer.TicksDifference(start) ; start =PerformanceTimer.Ticks () ; rDouble = ( double)AdaptedMareaCoder .Receive(seralizedData) ; deserializeTicks += PerformanceTimer.TicksDifference(start) ; } Console .WriteLine(CoderTestsConstants .MAREA2) ; Results results =ResultsManager .GetResults(serializeTicks,deserializeTicks,clock_freq, CoderTestsConstants .CODIFICATIONS,seralizedData.Length ,rDouble.GetType () .FullName) ; i f (oDouble == rDouble) {Assert .True(true ) ; Console .WriteLine(CoderTestsConstants .OK_STATE) ; Console .WriteLine(results .ToString () ) ; } else {Console .WriteLine(CoderTestsConstants .KO_STATE) ; Assert .True(false ) ; } } A.1.6.3 Nuget Server According to the proposed design MAREA services should be pushed as packages in an own NuGet server (like the one hosted at nuget.org). The steps to configure a NuGet server are presented bellow. •Create a ASP.NET Empty Web Application: Go to the File | New | Project menu option which will bring up the new project dialog and select ASP.NET Empty Web Application. •Install the NuGet.Server Package: Make Right click on the created ASP.NET Empty Web Application and select the option Manage NuGet Packages (figure A.1). Search and install the package Nuget.Serverin the Manage NuGet Packages dialog box (figure A.2). With this last step the NuGet.Server package has just converted the ASP.NET Empty Web Application into a site that is ready to serve up the package feed. To start the NuGet server build an run the ASP.NET Web Application project. The configuration of the NuGet server can be easily modified through Web.config file. The most important parameters are: •packagesPath: Specifies a custom (absolute or virtual) path for packages folder. •requireApiKey: Determines if an access key is required to push/delete packages from the server. •apiKey: Sets the value of the key to allow people to push/delete packages from the server. Figure A.7: NuGet server application settings from Web.config file The next step is configure the new package source in Visual Studio by clicking in the button Settings of the dial box Manage Nuget Packages. To add the new source add a name and specify the URL of the NuGet server like is shown the figure A.9. Note that the URL is http://domain/nuget/ and depends on how the site has been deployed. Figure A.8: Visual Studio package source configuration The URL http://domain/nuget/Packages lists the name and description of the packages that have been uploaded to the server. Figure A.9: View of OData over ATOM feed of MAREA packages A.1.6.4 Thorn Thorn is a command line utility accelerator for .NET applications. Thorn essentially provides a lightweight routing and dispatch layer to code. Such an environment encourages a style of development which relies on application-aware utilities, and also serves as a good springboard for experimentation, and ensures that one-offs that work out are already built in a repeatable, deployable, reusable fashion. A.1.6.5 StringTemplate StringTemplate is a java template engine (with ports for C#, Python) for generating source code, web pages, emails, or any other formatted text output. StringTemplate is particularly good at code generators, multiple site skins, and internationalization / localization. StringTemplate also powers ANTLR. A template engine is simply a code generator that emits text using templates, which are really just "documents with holes" in them where you can stick values called attributes. An attribute is either a program object such as a string or VarSymbol object, a template instance, or sequence of attributes including other sequences. Template engines are domain-specific languages for generating structured text. StringTemplate breaks up your template into chunks of text and attribute expressions, which are by default enclosed in angle brackets <attribute-expression> (but you can use whatever single character start and stop delimiters you want). StringTemplate ignores everything outside of attribute expressions, treating it as just text to spit out. A.2 Source control tools Source control is defined as the management of changes to documents, computer programs, large web sites, and other collections of information. A.2.1 Git Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. The major difference between Git and any other version control system (like Subversion) is the way Git thinks about its data. Conceptually, most other systems store information as a list of file-based changes. These systems think of the information they keep as a set of files and the changes made to each file over time. Git does not think of or store its data this way. Instead, Git thinks of its data more like a set of snapshots of a mini filesystem. Every time the user commits, or saves the state of a project in Git, it basically takes a picture of what all the files look like at that moment and stores a reference to that snapshot. Git has three main states that your files can reside in: committed, modified, and staged. Committed means that the data is safely stored in users local database. Modified means that the programmer has changed the file but have not committed it his database yet. Staged means user has marked a modified file in its current version to go into his next commit snapshot. The basic Git workflow goes something like this: •The user modifies files in his working directory. •The user stages the files, adding snapshots of them to his staging area. •The user does a commit, which takes the files as they are in the staging area and stores that snapshot permanently to his Git directory. A.2.2 GitHub GitHub is a web-based hosting service for software development projects that use the Git revision control system. GitHub is one of largest open source community which offers both