Integration of service robots in the smart home

Universidad de Sevilla Departamento de Ingenier ´ ıa de Sistemas y Autom´ atica Doctoral Thesis Integration of Service Robots in the Smart Home Author: Javier Ram´ırez de la Pinta Supervisors: Dr. Eduardo Fern´andez Camacho Dr. Jos´e Mar´ıa Maestre Torreblanca It is no measure of health to be well adjusted to a profoundly sick society. Jiddu Krishnamurti No es saludable estar bien adaptado a una sociedad profundamente enferma. Jiddu Krishnamurti Acknowledgements In first place, I would like to express my sincere gratitude to professors Eduardo Fern´andez Camacho and Jos´e M. Maestre Torreblanca for the continuous support and guidance of my Ph.D study and research. During these years they have been very supportive and have taught me how to become a true researcher. There have been many other researchers that have helped me with this work. I would like to thank all my colleagues in Seville. Working in the Systems and Automation Department and with people over there has been a great privilege. In addition, I would like to thank my family, particularly my parents for their patience during my study life and I hope they will be proud about this thesis. Last but not the least, I would like to thank my wife, Montse, for all the years she has been waiting me while I had been studying. Javier Ram´ırez de la Pinta Seville, January 2016 Contents 1 Introduction 1 1.1 A Brief History of Interoperability . . . . . . . . . . . . . . . 2 1.2 Objective of the Thesis . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Thesisoutline........................... 5 2 Interoperability Systems 7 2.1 Interoperability evolution . . . . . . . . . . . . . . . . . . . . 10 2.2 Interoperability technologies . . . . . . . . . . . . . . . . . . . 12 2.3 CORBA.............................. 13 2.3.1 Components ....................... 14 2.3.2 Services .......................... 15 2.3.3 Application Examples . . . . . . . . . . . . . . . . . . 17 2.4 Jini................................. 17 2.4.1 General Features . . . . . . . . . . . . . . . . . . . . . 17 2.4.2 Specific Features . . . . . . . . . . . . . . . . . . . . . 19 2.4.3 Organization of the Jini Architecture . . . . . . . . . . 22 2.4.4 Application Examples . . . . . . . . . . . . . . . . . . 24 2.5 RMI ................................ 24 2.5.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . 25 2.5.2 Components ....................... 25 2.5.3 Application Examples . . . . . . . . . . . . . . . . . . 26 2.6 OSGi................................ 27 2.6.1 General Features . . . . . . . . . . . . . . . . . . . . . 27 2.6.2 Specific Features . . . . . . . . . . . . . . . . . . . . . 27 2.6.3 Organization . . . . . . . . . . . . . . . . . . . . . . . 30 2.6.4 Application Examples . . . . . . . . . . . . . . . . . . 34 2.7 UPnP ............................... 34 2.7.1 General Features . . . . . . . . . . . . . . . . . . . . . 35 2.7.2 Specific Features . . . . . . . . . . . . . . . . . . . . . 36 2.7.3 Protocols ......................... 36 i ii CONTENTS 2.7.4 Components of a UPnP Network . . . . . . . . . . . . 39 2.7.5 UPnP Operation . . . . . . . . . . . . . . . . . . . . . 42 2.7.6 Application Examples . . . . . . . . . . . . . . . . . . 56 2.8 DLNA............................... 57 2.8.1 General Features . . . . . . . . . . . . . . . . . . . . . 58 2.8.2 Specific Features . . . . . . . . . . . . . . . . . . . . . 64 2.8.3 Application Examples . . . . . . . . . . . . . . . . . . 68 2.9 WebServices ........................... 70 2.9.1 Components ....................... 71 2.9.2 Application Examples . . . . . . . . . . . . . . . . . . 73 2.10 Semantic Web Services . . . . . . . . . . . . . . . . . . . . . . 73 2.10.1 Required Functionalities . . . . . . . . . . . . . . . . . 74 2.10.2 Main Technologies . . . . . . . . . . . . . . . . . . . . 74 2.11 Military Standards . . . . . . . . . . . . . . . . . . . . . . . . 75 2.11.1JAUS ........................... 75 2.11.2 Other Military Standards . . . . . . . . . . . . . . . . 80 2.12 Other Technologies . . . . . . . . . . . . . . . . . . . . . . . . 82 2.12.1Salutation......................... 82 2.12.2 Service Location Protocol . . . . . . . . . . . . . . . . 82 2.12.3 Ad hoc Developments . . . . . . . . . . . . . . . . . . 83 2.12.4URBI ........................... 83 2.12.5 DH Compliant . . . . . . . . . . . . . . . . . . . . . . 84 2.12.6ROS............................ 84 2.12.7OROCOS......................... 86 2.12.8OpenJAUS ........................ 86 3 Integration of Service Robots in the Smart Home by means of the Universal Plug and Play Protocol 89 3.1 Smart Homes and Robots . . . . . . . . . . . . . . . . . . . . 91 3.1.1 Roomba Robot . . . . . . . . . . . . . . . . . . . . . . 96 3.1.2 RovioRobot ....................... 99 3.2 UPnP Virtual Devices Development . . . . . . . . . . . . . . 102 3.2.1 Step 1: Creation of the Service Description . . . . . . 103 3.2.2 Step 2: Creation of the UPnP Stack . . . . . . . . . . 104 3.2.3 Step 3: Implementation of the Actions . . . . . . . . . 105 3.2.4 Step 4: Operational Testing and Validation of the Device106 3.3 UPnP Gateways for Robots . . . . . . . . . . . . . . . . . . . 107 3.3.1 Roomba Gateway . . . . . . . . . . . . . . . . . . . . . 107 3.3.2 Rovio Gateway . . . . . . . . . . . . . . . . . . . . . . 112 3.4 Experiments............................ 117 3.4.1 Robots in a UPnP Network . . . . . . . . . . . . . . . 118 CONTENTS iii 3.4.2 Roombatests....................... 118 3.4.3 Roviotests ........................ 120 3.4.4 Rovio Robot Tracking a Ball . . . . . . . . . . . . . . 122 3.4.5 Rovio Robot Tracking a Roomba Robot . . . . . . . . 123 3.4.6 Garbage Detection . . . . . . . . . . . . . . . . . . . . 124 3.5 Conclusion ............................ 125 4 Collaborative Tasks between Robots based on the Digital Home Compliant Protocol over UPnP 127 4.1 The DH Compliant protocol . . . . . . . . . . . . . . . . . . . 128 4.2 DHC-Groups operation . . . . . . . . . . . . . . . . . . . . . 130 4.3 The Roomba vacuum cleaner: from Roomba UPnP to Roomba DHC................................ 134 4.4 Experiments............................ 135 4.4.1 Implementation . . . . . . . . . . . . . . . . . . . . . . 135 4.4.2 Experimental results . . . . . . . . . . . . . . . . . . . 137 4.5 Conclusion ............................ 141 5 Cloud Robotics 143 5.1 From Cloud Computing to Cloud Robotics . . . . . . . . . . 144 5.2 Integration of Rovio in a home network through Vera . . . . . 146 5.2.1 Rovio ........................... 146 5.2.2 Vera............................ 147 5.2.3 Creation of a basic Rovio interface for Vera . . . . . . 147 5.3 Cloud capabilities . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.3.1 Rovio, home and cloud interaction . . . . . . . . . . . 151 5.3.2 User interaction . . . . . . . . . . . . . . . . . . . . . 153 5.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 155 5.4.1 Teddy bear search experiment . . . . . . . . . . . . . . 156 5.4.2 New object for search service experiment . . . . . . . 156 5.4.3 Z-wave + Rovio integration experiment . . . . . . . . 156 5.5 Conclusion ............................ 157 6 Conclusions 159 6.1 Conclusions............................ 160 6.2 Futureresearch.......................... 161 6.3 Publications derived from this thesis . . . . . . . . . . . . . . 162 A Resumen en castellano 165 A.1 Sistemas de Interoperabilidad . . . . . . . . . . . . . . . . . . 166 A.2 Objetivos de la Tesis . . . . . . . . . . . . . . . . . . . . . . . 168 6Introduction its inhabitants. Nowadays, there are robots that works more or less independently in the home, a good example is the vacuum cleaner robots which are widely spread and accepted in homes around the world. In this chapter, the integration of two robots, the Roomba vaccum cleaner robot and the Rovio surveillance robot, in a UPnP network is presented. In addition, different scenarios developed using these robots are detailed. •Chapter 4: Collaborative Tasks between Robots based on the Digital Home Compliant Protocol over UPnP. This chapter presents a novel interoperability protocol: Digital Home Compliant (DHC) which is focused on the solution of the interoperability problem between domotic and robotic devices. In particular, the DHC-Groups module has been implemented for Roomba, which allows a robot group (hive) to perform collaborative tasks. Different experiments are shown in order to test the DHC-Groups module and to show its possibilities. •Chapter 5: Cloud Robotics. Basic concepts of cloud computing and cloud robotics are presented in this chapter. A cloud development is also detailed, as well as the scenarios carried out with it. •Chapter 6: Conclusions. The thesis ends with a chapter that analyzes the most relevant contributions and, additionally, points out future research lines in the field of interoperability in the smart home. Chapter 2 Interoperability Systems Interoperability deals with the integration of heterogeneous devices in the same network regardless of their architecture, operating system, programming language or their location in the network. The problem of the lack of interoperability arises when there are different devices that comply with a certain system but these devices are not compatible with other technologies in the smart home, so it is not possible to make all devices and technologies work together. This fact is not new. Actually it is one of the biggest handicaps of the smart home market. Real interoperability is necessary to make possible a future in which homes detect automatically user needs and satisfy them. The lack of interoperability causes that different systems and devices can not interoperate with each other and hinders the take-off of the current smart home. For example, in the home automation market many technologies have emerged but none of them worried about providing interoperability mechanisms with previous technologies. Many factors that concurred explain the situation: new technologies usually come with new possibilities that did not exist before, compatibility implies higher development costs and a risk of clients buying devices to other manufacturers, very strong lack of standardization in the sector... The list of causes is almost endless and helps to identify mistakes that should not be repeated in the future. Nevertheless, it does not help too much to solve the interoperability problem that we have to face nowadays. In particular, it is possible to identify the following areas that work more or less independently and that have to be integrated in the future smart home: •Home automation: it arrived at homes 30 years ago with the arrival of X-10, which despite of its “antiquity” is still one of the most used systems. For this reason it has become a de facto standard. There are other systems that have enjoyed enough success to establish themselves 7 8Interoperability as solid options in their market segments. This is the case, for example, of KNX and LonWorks, both of them supported by international standards. Apart from these systems there is a constellation of proprietary systems with good future perspectives such as Z-Wave or Ingenium. •Access and Surveillance systems. •Digital entertainment: this is the field in which interoperability is closer to become a reality. The success of UPnP for this kind of applications has greatly simplified the access to multimedia content shared among the computers of a network. In this sense, new protocols take advantage of this as DLNA. •Assistive Computing and Healthcare. •Energy management. •Advance telecommunication services: probably it is easier to understand what we mean with advance service with an example: IP- Domo, which is a Spanish home automation system with many capabilities in this sense, allow to deviate the incoming calls from the entryphone to the mobile phone when the user is not at home. The future smart home must be able to support advanced services like this one, that demand a strong degree of interoperability. •Service robots: they are a reality in many homes thanks to robots like the vacuum cleaning robots. Right now the integration of robots in the smart home is far away from becoming a standard. The most advanced robots from the interoperability point of view include some port to receive external commands or a server with a page to receive commands. •Electrical appliances and current consumer electronics: it is not possible to walk to the future without using the bridge of the present. Nowadays homes are filled with many useful devices that are closed systems. In the case of the appliances, almost always it is possible to use a relay to control the plug that feeds the devices. This allows a small grade of integration. There are other devices such as air conditioning systems or DVD players, to name a few, that open a small door for a more advanced interoperability through a wise use of infrared. However, the operation of all these devices, systems or robots does not really require them to “think” as they are simply programmed to perform Interoperability 9 a series of repetitive tasks. If anything interferes with the pre-programmed task, they would malfunction since none of them is able to sense the interference and “think out” a solution. As service robots are in greater proximity to humans, the technology involves more safety concerns over the humanmachine interaction. Therefore, it remains a great challenge today for us to build smart homes and intelligent robots that can “think” like we do. To achieve such a goal, scientists and engineers have been trying hard to capture the essence of human intelligence in our homes and robots to make them intelligent to function well in the real world. This is a challenging and ambitious task since the robot or home intelligence must cope with various noises, uncertainty and dynamic changes in the real world. Like human beings, smart homes and intelligent robots should be able to sense their environments, reason and make decisions and respond to tasks and unexpected events quickly. In general, intelligent robots and smart homes are broad, interdisciplinary subjects that involve many different technologies such as sensor integration, data fusion, wireless sensor networks, map building, embedded computing, navigation, planning and artificial intelligence. For each individual smart home or robot, the “thinking” process takes place at many different levels. At its lowest level,“thinking” needs to be fast to respond quickly to unexpected events. At higher levels, “thinking” enables the homes and robots to handle a dynamic and uncertain world. By contrast, “thinking” should exhibit an adaptation and learning capability at its highest level. Moreover, a close interaction among smart homes and robots should be realized for achieving the common goal cooperatively. In all emerging markets the compatibility between systems plays a crucial role for the success of the market itself. The digital home market is like the dog that chases its own tail in this sense: as there is no ideal system between the existing ones, the market is guided by the offer and not by the demand, but how is there going to be a strong demand for devices and services when there is no simple way of combining them? It is necessary a solid commitment towards interoperability from all the actors of the smart home industry if they want to speed up the penetration of advanced services to the end users. In this chapter we present some background for the research addressed in this thesis and review several systems that can be found in this field. Section 2.1 presents an introduction to the evolution of the interoperability systems and the approaches to solve the interoperability problems. Next, section 2.2 provides an introduction about interoperability systems that will be detailed in subsequent sections. Specific systems that try to solve the interoperability problem are deeply analyzed in sections 2.3, 2.4, 2.5, 2.6, 10 Interoperability 2.7, 2.8, 2.9, 2.10, 2.11 and 2.12 which detail CORBA, Jini, RMI, OSGi, UPnP, DLNA, Web Services, Semantic Web Services, some military standards (JAUS, 4D/RCS and NANO STANAG 4586), and other interesting technologies (Salutation, Service Location Protocol, Ad hoc Developments, URBI, DH Compliant, ROS, OROCOS and OpenJAUS) which have not had enough success in the digital home market. 2.1 Interoperability evolution Nowadays there are several initiatives and different standards available whose objective is to interconnect all consumer electronic devices in a digital home. However, none of them seems to be the definitive interoperability system. The lack of interoperability between the proposed systems and the lack of commitment from vendors to make devices that comply with such systems hinder the take-off of the smart home in new buildings. Over the past few decades there has been an exponential growth in service robots and smart home technologies, which has led to the development of exciting new products in our daily lives. Service robots can be used to provide domestic aid for the elderly and disabled, serving various functions ranging from cleaning to entertainment. Service robots are divided by functions, such as personal robots, field robots, security robots, healthcare robots, medical robots, rehabilitation robots and entertainment robots. A smart home appears “intelligent” because its embedded computers can monitor so many aspects of the daily lives of householders. For example, the refrigerator may be able to monitor its contents, suggest healthy alternatives and order groceries. Also, the smart home system may be able to clean the house and water the plants. The lack of interoperability becomes more evident with the inclusion of such service robots in homes around the world. In this sense, it would be interesting to control the robots remotely and coordinate all the robots to perform complex tasks and reduce time to complete them. For this reason, the goal of achieving full interoperability between consumer electronic devices and service robots becomes more and more important. All efforts are focused on achieving a common and accepted standard which allows to interconnect all devices in a digital home (Figure 2.1). Another interesting approach is to use an existing interoperability system (intermediate standard) to interoperate with the rest of the systems, in such a way that everything is interconnected through this intermediate system (Figure 2.2). The continuous evolution of computers together with lowest prices in consumer electronics has allowed a wide deployment of the digital home. The 2.1. INTEROPERABILITY EVOLUTION 11 Figure 2.1: Interconnection through a common standard Figure 2.2: Interconnection through an intermediate standard 12 Interoperability next revolution in the smart home is expected to come from the world of robotics. At present, the use of robotics is limited to industrial areas, although service robots that assist us in routine tasks such as cleaning the house, mowing the lawn or even preparing meals are becoming common. In recent years, the number of robotic standards has increased, and this progress has encouraged the integration of service robots in the smart home and the emergence of different communication protocols. Nevertheless, different problems have to be solved before service robots become as popular as computers. In particular, interoperability between the different systems that may exist in future homes is an ongoing issue. 2.2 Interoperability technologies When looking through history from distributed systems to interoperability, it is important to recognize a need for cooperation and expansion of networks that are already in use. The beginning is the concept of middleware. Middleware is connectivity software that offers a group of services that make possible the running of distributed applications over heterogeneous platforms. The idea of middleware, as an abstract layer of software, is to encapsulate all the available resources on a network, which can comprise all kinds of devices (from embedded processors to super processors, laptops, PDAs and mobile phones) and interconnect them in a transparent way. In other words, give an API (Application Programming Interface) to the programmers for the use of distributed applications. There are some works related to the design and implementation of middleware generic distributed systems. For example [8], which represents an approximation to the design of a configurable system, based on the concept of reflection. The usefulness of this component’s engineering is also important when giving a system the ability to configure. These concepts are also commented on by [9]. These authors also talk about the link to the application layer by using this component’s technology. They suggest the development of a model (OpenORB, based on the model CORBA) independent of the platform and the language of programming. They also define meta-structures and meta-data to give intelligence to the protocol so it can apply reason to its own interpretations, so the system’s (re)configuration will be easier. The idea that the reader must have in mind during this chapter is interoperability. Interoperability is the key component to solving the smart home jigsaw puzzle. Thus, in this chapter, we will place special emphasis on the interoperability aspects of the different standards. In addition, different 2.3. CORBA 13 research projects on the interoperability and control of robotic systems and unmanned vehicles will be surveyed. Behind all these standards and projects, there are stories of success and failure, and many valuable lessons about the complex world of interoperability. At this point, it is difficult to know if any of these alternatives will prevail and become a consolidated standard for the integration of robots in the digital home. However, what we know for sure is that any succeeding standard will have learnt from all that will be presented here. In this chapter, different standards that could be used for the integration of mobile robots and unmanned vehicles in the digital home will be detailed. These standards have been proposed within different contexts. On the one hand, standards have been developed for military purposes such as JAUS or 4D/RCS, which is logical given that the control of autonomous vehicles has many potential applications in this field. On the other hand, standards have been developed in a computer science context, where interoperability between the different agents that may interact in a networked environment is a major problem. This chapter is mainly based on the tw chapters published in the Springer’s book “Service Robotics within the Digital Home” [10][11]. Some of the most-used middleware initiatives to provide interoperability in the smart home are analysed. The protocols or existing systems used to communicate among heterogeneous platforms will be described. 2.3 CORBA CORBA (Common Object Request Broker Architecture) is a standard that provides a platform for the development of distributed systems. CORBA is defined by the Object Management Group (OMG), which defines APIs, communication protocols and all necessary items to ensure interoperability between different applications running on different platforms. CORBA uses an IDL to specify the interfaces through their functionality. This is a way to indicate how CORBA data types must be used in implementations of client and server. All this means that CORBA is a kind of middleware (platform of distributed services, independent of the operating system) that guarantees success in the transit of data across different platforms and applications. It is applied in RTS and is efficient enough for any kind of problem. The main features of this standard are: •It is a distributed object standard. 14 Interoperability •It specifies the architecture the system should have, is flexible and heterogeneous. •Interoperability. •Scalability. •Transparency, facilitating client-object communication [12]. •Naming service. •It sets a minimum object model. •Each object implements an interface. - The definition of interfaces is made through the IDL, making it independent of the programming language. - The reuse in software is achieved through interface inheritance. - Multiple inheritance. - The details of an object’s implementation cannot be accessed. 2.3.1 Components •The Object Request Broker (ORB) is the CORBA object manager and is part of its core. It allows for the invocation of static and dynamic objects [12]. It can operate without the services and facilities provided by CORBA. It handles the invocation and search for remote objects using dynamic methods for the invocation. It is responsible for giving back the object attributes of the object accessed through the IDL of the object. Locally, it also collects information on the objects to pass to other ORBs and handles local computer security (Figure 2.3). •IDL, Language for defining interfaces. Since it is a declarative language and not a programming language, it defines interfaces independent of the implementations of objects. •Dynamic Invocation Interface (DII). Generic Stub. Client side. •Dynamic Skeleton Interface (DSI). Generic skeleton. Server Side. •Both DII and DSI are based on the interface repository, which is a CORBA object that contains information on the object’s interfaces and their types. It allows applications to access this information in a static or a dynamic way. The main advantage is the support given to the dynamic calls. 2.3. CORBA 15 Figure 2.3: CORBA architecture •The implementation repository is required when the objects are persistent. Most general purpose ORBs provide a repository of implementations that supports indirect connections for persistent references. This characteristic solves the problem of direct connections for persistent references. It has also a bad point; it slightly reduces the good working of the first invocation from client to server. It also offers various modes for the automatic activation of server objects [13]. •The object adapter is the bridge between the ORB and CORBA object implementations. This allows it to make requests to an object without knowing its interface, since the object adapter adapts the object’s interface to that expected from the object making the request. •Communication protocols between ORBs. CORBA is based on the protocols GIOP (General Inter-ORB Protocol) and the standard protocol IIOP (Internet Inter-ORB Protocol). GIOP specifies the types of messages and the format to transport requests between ORBs. IIOP specifies the way TCP/IP is implemented over GIOP. Thanks to these protocols, ORB can be integrated even if it comes from different developers. 2.3.2 Services There is a large set of standard services offered by CORBA [14]. These services are added to the ORB interface to complete it; however, they are optional. The most important include: •Concurrency Service. Mediates concurrent access to an object such that theconsistency of the object is not compromised when accessed by 22 Interoperability Table 2.1: RMI components Infrastructure Programming model Services Basic Java JVM API Java JNDI RMI JavaBeans Enterprise Beans Java security - JTS Java + Jini Discovery/Join Leasing Printing Distributed security Transaction Transaction manager Lookup Events JavaSpaces services gories or without any of them. By contrast, the main feature of Jini is that it is a system built with a particular infrastructure and described programming models, based on the concept of service. The separation of the segments in the architecture means that only a slight change is needed in the inherited code to be used in a Jini system. However, the power of a Jini system is only available for services built using the integrated model from the beginning. A Jini system can be considered as an extension of the network’s infrastructure, programming model and services that made Java technologies popular in the case of a single machine. These categories, along with the components for the Java application environment, are shown in the Table 2.1. 2.4.3 Organization of the Jini Architecture Infrastructure The infrastructure defines the basic core of this technology. It includes: •A distributed system for security, integrated in the RMI, which extends the security model from Java to the world of the distributed systems. •The discovery and join protocols, service protocols that allow other services (hardware or software) to discover and announce the services offered to the other members of the group. •The lookup service, which is used as a backup for the services. The entries in the lookup service of objects are written in Java, and they can be downloaded as a part of a search operation and work as local proxies for the service that sets the code in the lookup service. 2.4. JINI 23 Programming Model The infrastructure enables the programming model and makes use of it at the same time. The contracts made in the lookup service have a limited lifetime. This fact allows the lookup service to precisely check the set of available services at a specific moment. When the services binds to or separates from the lookup service, the events are notified about it, and the objects that have already shown an interest in receiving this information are updated about these new or defunct services. The programming model is based on the ability to move the code, supported by the infrastructure. Both the infrastructure and the services that use it are calculation entities that live in the physical environment of the Jini system. However, services also constitute a set of interfaces that define the communication protocols used by services and the infrastructure to communicate between them. These interfaces together form the distributed extension of the standard model in Java programming, which constitutes the Jini programming model. Among the interfaces that make up the Jini model are: •The leasing interface, which defines a way to allocate and release resources through a model based on the renovation of their lifetime. •The event and notification interfaces, which are extensions of the event model used by JavaBeans components for distributed environments. This feature allows event-based communication between services enabled by the Jini technology. •Operation interfaces, which enable entities to cooperate so that all changes occur in the group or none take place. Services The technology infrastructure and the Jini programming model are designed to enable the services to offer themselves and to be found on the network. These services make use of the infrastructure to call and discover each other and announce their presence to other services and users. The services appear as objects written in Java, perhaps made up of other objects. A service has an interface that defines the operations that may be requested of it. Some of these interfaces are intended to be used by programs, whereas others are intended to be administered by the client to enable the service to interact with a user. 24 Interoperability The kind of service determines the interfaces of which it is composed and defines the set of methods used to access the service. A service can be implemented only by other services. Some of the Jini services are: •A printing service, which can print from Java applications. •A service of JavaSpaces, which can be used for simple communication and storing groups of objects written in Java. •A transaction administrator, which allows groups of objects to participate in the Jini transaction protocol defined by the model of programming. 2.4.4 Application Examples There have been various initiatives to implement Jini as a form of communication between devices. In this regard, we underline the Ronin Agent Framework [23], an environment based on Jini’s distributed agents. This implementation attempted to improve the initial protocol, making it independent of the domain (so external devices could communicate with the local network), among other advances. [24] and [25] described the implementation of a SOA (architecture oriented to services [26]), ICENI, using the Jini platform, among others. This environment is based on an independent specification of SOA. The integration of different service platforms is not easy. [27] integrated Jini with UPnP, but it is important to note that these two protocols are incompatible by themselves. This new platform allows UPnP services to use Jini devices and vice versa by making just a few configuration changes. However, these authors do not answer several of the questions referring to the limitations encountered when working with these two architectures together. In addition, for devices that do not have enough capacity to run a Java Virtual Machine (JVM), Jini offers the possibility of using a surrogate host. This is just another device capable of supporting a JVM, which works as a bridge between the original device and the Jini network architecture. 2.5 RMI RMI (Remote Method Invocation) emerged from the need to communicate among different objects, and it is implemented on different machines as happens on distributed systems. Therefore, this technology is a remote invocation of Java objects. The initial version of Java RMI required a JVM in both the origin and destination machines [28]. 2.5. RMI 25 After the RMI-IIOP was developed, it was added to the RMI, providing it with the best features of CORBA. RMI is pure Java and since it does not support other languages, CORBA emerged. The adaptation to a distributed system has not prevented the continued development of RMI as a secure system. The main characteristics of RMI are: •Simple, easy to write and easy to maintain. •Transparency, because the distribution of objects and parameters passing is transparent to the programmer. •Pass an object by value (as parameters of methods). •The definition of interfaces is done directly in Java. •Implementation in Java. •Independence of the communication protocol. •Separation between interface-client and implementation-server. •Naming service. 2.5.1 Architecture RMI is a layer architecture made of a stub/skeleton layer, a remote reference layer and a transport layer. The programmer only interacts with the application layer. The RMI system manages the three previous layers (Figure 2.6), which could be replaced by others with the same function without altering the rest. 2.5.2 Components RMI allows the programming of CORBA servers and applications via the RMI API. It is possible to work entirely in the Java programming language using the Java Remote Method Protocol as a transport or to work with any other CORBA implementation using IIOP Java RMI over IIOP. RMI-IIOP is designed for developers who program in Java and want to use the RMI interfaces using IIOP as the transport layer. The RMI-IIOP interoperability with CORBA objects implemented in other languages is available only if all the remote interfaces have been previously defined as Java RMI interfaces [29]. 26 Interoperability Figure 2.6: RMI architecture 2.5.3 Application Examples At the University of Bielefeld, Germany, one research group has integrated memory-based software for the development of autonomous robots. This is an approach to an architecture of autonomous mobile robots operating in human environments. It replaced the use of data on a closed chain based on the long- and short-term memory. RMI was used for the exchange of critical information, such as the module that controls the hardware. RMI also allows the system to estimate when the configuration has been completed. The system can then send information on the result of the configuration [30]. [31] focuses on task-level programming and monitoring robots in their daily operations. It is not a framework limited to robots and it could be used in other distributed environments. During its development, the authors took advantage of technologies available in Java, such as Jini, RMI and Java Native Interface. [16] supported Java RMI over Bluetooth, GPRS and WLAN technologies. As a conclusion of this, the good work of Java RMI was tested in heterogeneous wireless environments, allowing parallel and distributed control. In a study by researchers at the Information and Communications University in Korea, RMI is used to access external ontologies in the development of a selfexpandable software. This kind of software is useful for intelligent robots for two reasons. First, they study their environments and then they decide their appropriate behavior based on what they have learnt about their surroundings. 2.6. OSGI 27 DEVS/RMI is a distributed, self-adaptive and reconfigurable simulation environment for engineering studies. It is based on the standard implementation of DEVS, in which Java RMI supports the synchronization of local and remote objects. It is designed for the intensive testing of programs, and this is the reason for it supporting dynamic models [32]. 2.6 OSGi OSGi (Open Services Gateway Initiative [33]) is an independent corporation that brings together about 40 companies in an alliance responsible for defining and promoting open specifications for the delivery of managed services in network environments. It is based on the modularity of the Java environment, trying to abstract the implementation of components (bundles) using services to communicate. One of its main aims was to resolve certain development and deployment conflicts, such as class conflict and the explicit dependencies [34]. 2.6.1 General Features OSGi is based on a layer model (Figure 2.7) that includes, among others, the bundles or packages (components developed as jar files), services (which provide communication between bundles through Java objects), modules and security layer. The features of OSGi could be a good alternative for the development of complex systems because of its versatility and cross-platform feature (only a JVM resident on each node of the network would be required for the running). Nevertheless, OSGi has several problems (some delimited, some not) because of its poor basis of compatibility and poor management of dependencies. 2.6.2 Specific Features Framework of OSGi The OSGi framework is a module system for Java that implements a complete and dynamic model of components, which does not exist in independent environments of JVMs. The applications and components (which come in packets or bundles) can be installed, started, stopped, updated and uninstalled remotely without rebooting. The management of Java packages and classes is carefully specified. Lifecycle management is performed through APIs that make possible the remote download of management policies. The registry 28 Interoperability Figure 2.7: Layer model of an OSGi system allows service bundles to detect if services have been added or deleted and acts accordingly. Originally, it was focused on service gateways, but the scope has since widened. OSGi specifications are now used in applications ranging from cell phones to the Eclipse development environment (open source). Other application areas include automotive, automation in industry and buildings, PDAs, grid computing, entertainment (such as iPronto), fleet management and application servers. Figure 2.7 shows the hierarchical structure of an OSGi system. Specification Process OSGi specification has been developed by its members in an open process that is available for the public free of charge under the OSGi specification license. The OSGi Alliance has a performance program that is open to its members. In September 2008, the list of certified OSGi implementations contained five entries. Architecture Any framework that implements the OSGi standard provides an environment for the modularization of applications in small packages. Each package is a 2.6. OSGI 29 Figure 2.8: Lifetime of an OSGi bundle collection of well-coupled and dynamically loadable classes, jar files and a configuration that explicitly state their external dependencies (if any). The framework is conceptually divided into the following areas (see Figure 2.8): •Bundles: These are jar components with extra headers in a detailed manifest file. •Services: The service layer connects bundles dynamically, offering a model of publication, search and link to plain old Java objects. •Register services: The API of some management services (ServiceRegistration, ServiceTracker and ServiceReference). •Lifetime: The API for the management of lifetime (install, start, stop, update and uninstall) bundles. •Module: The layer that defines the encapsulation and declaration of dependencies (how a bundle can import and export code). •Security: The layer that deals with security issues, limiting the functionality of the bundles to predefined capabilities. •Running environment: This defines what methods and classes are available on a specific platform. Since they are susceptible to change, there is no fixed list of running environments. The Java community is creating new versions and editions of Java constantly. 30 Interoperability 1. Bundles A bundle is a set of Java classes and additional resources accompanied by a detailed manifest file (MANIFEST.MF) of all its contents as well as the additional services required to provide the included group of Java classes more complex behavior until the point of abstraction, where the whole is treated as one component. An example of a MANIFEST.MF file, typical of OSGi headers, is shown below: Bundle-Name: Hello World Bundle-Symbolic Name: org.wikipedia.helloworld Bundle-Description: A Hello World bundle Bundle-ManifestVersion: 2 Bundle-Version: 1.0.0 Bundle-Activator: org.wikipedia.Activator Export-Package: org.wikipedia.helloworld;version=“1.0.0” Import-Package: org.osgi.framework;version=“1.3.0” 2. Lifecycle A lifecycle layer adds bundles that can be installed, started, stopped, updated and uninstalled dynamically. The bundles trust in the module layer for class loading but they add an API to manage the runtime modules. The lifecycle layer provides mechanisms that are not usually part of an application. Some extensible dependency mechanisms are used to ensure the correct working of the environment. Lifecycle operations are fully protected by the security architecture (Table 2.2). 3. Services The OSGi Alliance has specified many services, all of them by a Java interface. The bundles can implement this interface and register it with the service registry. Service clients can find it in the service registry or detect it when it appears or disappears (Tables 2.3-2.5). 2.6.3 Organization The OSGi Alliance was founded by Ericsson, IBM, Motorola, Sun Microsystems and others in March 1999 (before becoming a nonprofit corporation called Connected Alliance). Among its members (as of May 2007) are more than 35 companies from different business fields, such as IONA Technologies, Ericsson, Deutsche Telekom, IBM, Makewave - before it was Gatespace Telematics - Motorola, 2.6. OSGI 31 Table 2.2: Description of the lifetime of an OSGi bundle State of the bundle Description Installed The packet has been successfully installed Resolved Every Java class that needs the bundle is available. This state indicates that the packet is ready to be started or stopped Starting The package is being started; it will call the method BundleActivator.start, and this one has not finished yet. When the bundle has an activation policy, it will remain in the initial state until it is activated, according to this policy Active The package has been successfully activated and it is running. Its starting method, Bundle Activator, has been called and it has returned Stopping The packet is being stopped. The method Bundle- Activator.stop has been called but the stop method has not returned yet Uninstalled The packet has been uninstalled. It cannot be changed to a different state Nokia, NTT, Oracle, ProSyst, Red Hat, Samsung Electronics, Siemens, Spring- Source and Telef´onica. The alliance has a board that establishes the governance of the organization. OSGi officers have different roles and responsibilities to support the alliance. The technical work is carried out in the expert groups (EGs) organized by the board of directors, and the non-technical work is carried out in various working groups and committees. The technical work in EGs includes development specifications, reference implementations and compliance testing. These EGs have made four versions of OSGi specifications (as at 2007). There are EGs dedicated to business areas, mobile phones, vehicles and central platforms. The Expert Group Company is the latest EG and handles applications regarding the company/server. In November 2007, the Residential Expert Group began working on specifications to remotely administer residential gateways orc homes. 38 Interoperability link to the location of the document that contains its description, with its properties and the services it offers. In addition to the SSDP properties, it provides the device with methods for disconnection notification and updates the device’s information using timeouts. A SSDP packet is just an HTTP request with the statement “NOTIFY” (to announce) or with “M-SEARCH” (to find a service), leaving the HTTP body empty, and keeping UPnP-specific attributes in its head. GENA The General Event Notification Architecture (GENA) allows sending and receiving notifications using HTTP over TCP/IP and HTTPMU over UDP/IP. UDP multicast is useful because it allows a single report to be distributed to a potentially large group of receivers using a single request. GENA defines the terms of the subscriber and the publisher of the notifications, which enable the event’s mechanism used by UPnP to warn of changes in the state of services. When a subscription to a service takes place, it sends event messages updating the changes in the status of the device. These event messages are in XML format. Apart from this, GENA is also used to create presence messages, which are sent using the SSDP protocol. SOAP The Simple Object Access Protocol (SOAP) provides a standard mechanism for packaging messages. It defines how two objects in different processes can communicate exchanging XML data. Thus, UPnP makes use of XML and HTTP to run remote procedure calls (RPCs), sending control messages to devices and getting the results or the errors in each case. Each control request is a SOAP message that contains the action invoked and all the necessary parameters. The response is another message of the same type with the state or the result of the action requested to the device. Although many protocols are created to simplify the communication between applications (RPC from Sum, DCE from Microsoft, RMI from Java and ORPC from CORBA), the SOAP has received more attention because of the great support received from the industry. It has been accepted by almost all large companies. Consequently, it is becoming the standard for communication based on RPC over the Internet. Some of its advantages are: •It is not associated with any language. •It is not strongly associated with any transport. •It is not tied to any distributed object infrastructure. 2.7. UPNP 39 Figure 2.10: UML (Unified Modelling Language) class diagram of a UPnP device [41] •It makes the most of the existing standards in the industry. •It enables interoperability among multiple environments. XML Extensible Markup Language (XML) plays an important role in the exchange of data. It is similar to HTML, but its main function is to describe data and not to display them as is the case of HTML. XML is a format that allows reading data through different applications. Specifically, it can structure, store and exchange information ([43]). It is used in UPnP for device and service descriptions, control messages and events. 2.7.4 Components of a UPnP Network A UPnP network defines various types of components, such as control points, devices and services. These are detailed below. Devices UPnP devices are logical containers for a service or set of services, and sometimes for other devices (embedded devices). Embedded devices can be discovered and used independent of the main container. Each UPnP device can offer any number of services. By itself, a device merely provides a self-description 40 Interoperability Figure 2.11: UML class diagram of a UPnP service [41] of its information, such as developer, model name and serial number. Device services are those that provide real functionality (Figure 2.10). There are different categories of UPnP devices, standardized according to the set of services provided by each device. This information (along with properties such as those mentioned above) is saved in an XML document that must be kept in the device until it needs to be sent. Services Each service in a UPnP device can contain any number of actions. An action has a name, a set of input parameters and a return value (optional). Each argument has a name, a value and an address. The address can be input or output depending on whether the argument is given to the action or returned by the action. A service has a service identifier (URI) that identifies it from all the others; there cannot be two services with the same identifier. It can keep the 2.7. UPNP 41 Figure 2.12: Control point invoking a service action [41] variables that represent the current state of the service. These state variables have a name, type, default value, current value and a range of permissible values. If a variable sets an event to indicate a state, then it is an event notification variable. A service is a state table, a control server and an event notification server. The state table contains the variables updated when there is any change in service status. The control server receives action requests and performs them, updates the state table and returns the result. The event notification server publishes updates of changes in the state of service (Figure 2.11). Control Points A control point is a network entity that invokes the functionality of a device. It is capable of discovering and controlling other devices. In terms of client/server in a UPnP network, the control point will be the client and the server role will be played by the device. Once the control point finds the device, it is capable of: •Getting the description of the device and a list of related services. •Getting the descriptions of the services in which it is interested. •Invoking actions to control the service. •Subscribing itself to the service’s events (Figure 2.12). When the status of the service changes, the event notification server sends an event to the control point. In short, a control point finds the devices, invokes the related actions to their services and signs up for event notifications. By contrast, a device responds to the actions invoked by the control point and sends the events when the variables change state. 42 Interoperability Figure 2.13: UPnP protocols stack [44] 2.7.5 UPnP Operation To describe the UPnP way of working, we will show the development in six basic steps or stages: Addressing, Discovery, Description, Control, Event Notification and Presentation. The routing stage can be considered step zero. The representation of the protocol stacks used in each one of the following steps is shown below (Figure 2.13). Addressing Since all UPnP communications are based on the Internet Protocol (IP), a device must obtain an IP address before it can join to a network that supports UPnP. The first step, also known as the zero phase, is based on this; an address for the control points and devices connected to the network must be obtained. All the reasoning presented in this phase is valid for both devices and control points. Addressing is the process by which a device automatically gets an IP address. It allows a device to join to the network and be prepared for communication with other devices and control points. The routing protocols implemented in the UPnP devices enables them to join dynamically to an IP network and to get an address without being configured by the user. UPnP devices can use the DHCP, UDP-based, to retrieve an IP address from a DHCP server. To do this, both devices and control points must have a DHCP client. Being connected to the network, the first thing to do is to find a DHCP server that provides them an address. If this server already 2.7. UPNP 43 exists on the network, they must use the address they have been assigned. If the network does not have a DHCP server, automatic IP addressing (Auto-IP) must be used to get the IP address. Through this mechanism, the device takes a random address within the 169.254/16 range to minimize potential collisions with other devices. This range was established by the ICANN (Internet Corporation for Assigned Names and Numbers) and the IANA (Internet Assigned Numbers Authority) for IP self-configuration in private networks. Once assigned an IP address using Auto-IP, it must be checked that this address is not used by any other device on the network using the ARP (Address Resolution Protocol). Each device must periodically verify the existence of a DHCP server on the network to manage the process of addressing. In this case, the automatically assigned IP is ruled out and so they start with the dynamic addresses assignment using the DHCP server. First, a device or control point tries to contact a DHCP server to obtain an IP address. If it is unable to locate the server, the device uses Auto-IP, which allows devices to select addresses without having a server to assign it to them. It may be necessary to resolve the assignment to IP addresses because the devices can implement protocol layers higher than UPnP. To obtain this functionality, devices must incorporate a DNS client and support the DNS dynamic registration. Discovery The discovery phase defines how a device announces its presence and how control points discover it. A UPnP device is like a mini web server that can be detected and monitored by a control point. The discovery process allows control points to find devices and services of interest and obtain information about them. The devices use the SSDP to announce their services to control points. These last ones use the SSDP to search for devices. In the tower of protocols below, you can see certain color codes that match the parts of each message defined below. These color codes are useful until the operation’s description of the UPnP technology is finished (Figure 2.14). Once a device has acquired an IP address, the SSDP announces its services to all the control points of the network. Similarly, when you add a control point to the network, the SSDP searches for relevant devices on the network. They will answer if there is an agreement with the data of the search message. The message exchanged in both cases is a discovery message that contains essential details about the device or its services, such as the type of device, identifier and a pointer to more detailed information (Figure 2.15). It must be kept in mind that when a control point or device initializes and connects to the network, it must wait a random time between 300 and 44 Interoperability Figure 2.14: Protocol stack for discovery Figure 2.15: Recovery of the descriptions of a service and a device [41] 2.7. UPNP 45 3,000 ms before sending any message of discovery. These ranges are set to avoid problems when many devices connect to the network at the same time (300 ms) and to minimize delays in the recovery of a network (3,000 ms) [45]. A URL of the XML document describing the device is included in the discovery and the advertisement responses of every device. This URL provides the necessary information to the control points to retrieve the descriptions of the devices and their services. All services contained in a device have three URLs that provide the necessary information to allow the control points to communicate with them: •The URL of control is where the control point sends requests to control the service. UPnP device manufacturers specify one for each device. •The URL of subscriptions to events is where control points send requests to subscribe to events. There is a URL for this kind of service in each device. If a service does not have event variables, and therefore no notification of events, the element URL of subscriptions to events must appear, but it will be empty. •The URL of description indicates the location of the control points from which the service description document will be retrieved. The service description document is returned by an HTTP GET request. A control point has two possibilities to search for devices. It can pick up a notification message sent by a device or it can request the response of the device using a discovery message sent by the control point itself (Figure 2.16). The devices must refresh their advertisement messages periodically because they have a limited lifespan. For this reason, they are not obliged to cancel those sent previously (announcing their capabilities) when they disconnect from the network. There are two ways of discovering devices: 1. Advertisement Once a device joins to the network, it announces its embedded devices and its services to control points through NOTIFY messages defined by GENA. These are multicast messages that use the SSDP. These messages are sent to the address and port (239.255.255.250:1900). This default value is indicated by ICANN/IANA to use it with the SSDP. The control points are supposed to listen to arriving messages in this port, knowing this way the capabilities that are available on the network. These messages do not require a response. One important fact 46 Interoperability Figure 2.16: Discovery step [46] about advertisement messages is the time of validity, which indicates the period in which the device is available. After finishing this period without sending an advertisement message, the device will stop being available on the network. During the advertisement process and considering that a root device has dembedded devices and provides kdifferent types of services, a total of 3+2d+kadvertisement messages are sent to the network. This can be deduced, assuming they are different devices, by interpreting the number of messages that should be sent by a device: •A message for each type of service with NT=type of service. •A message for each type of device (root or embedded) with NT=device type. •A message for each device (root or embedded) with NT=UUID of the device. •A message regarding the root device with NT=upnp:rootdevice. 2. Search This procedure is activated when a control point requires a type of device or a specific service. This is when the control point sends a multicast message with the address and port specified above, i.e., 2.7. UPNP 47 Figure 2.17: Protocol stack for description 239.255.255.250:1900. In this case, unlike in the method of advertisement, it will require answers from the devices that fit with the specifications defined by the control points. A control point must send multiple M-SEARCH messages since the messages are sent over the UDP (which does not guarantee delivery). A control point will receive multiple messages, but some will be duplicates. To filter these replies, the control point uses the USN header, which provides a unique identifier to look for answers. Description The description enables a device to list all the features it can provide. The descriptions of the devices and their services are stored in XML documents. The device summarizes its services and capabilities in a standard format. A device description document contains device information (such as developer, make, model and serial number), a list of the services provided by the device and a list of its embedded devices. A service description document contains detailed information about the device’s service, the actions that the service provides, the parameters and values returned by the service (Figure 2.17). The answers to the search messages received by a control point contain 54 Interoperability tees message delivery to the subscriber. When the subscription expires, the subscription identifier becomes invalid and the service stops sending events to the corresponding control point. If this control point attempts to send any message (renewal or cancellation, but not the subscription), the service is rejected because the ID is no longer valid. The control point will send a subscription message to the URL of the service to receive its events. This message uses the SUBSCRIBE method defined by GENA and its syntax is: SUBSCRIBE event route of the service HTTP/1.1 HOST: host:port CALLBACK: <delivery URL>NT:upnp:event TIMEOUT: request for the lifetime, in seconds A blank line must be added at the end of the last header. When this message is received, the service establishes a list of subscribers with the following information for each of them: SID, URL for the event messages delivery, event counter and length of subscription. If the subscription is accepted, the service sends a message with the identifier of the subscription and validity period. This message has the following syntax. It is important not to forget about the final blank line: HTTP/1.1 200 OK DATE: request time SERVER: OS/version UPnP/1.0 product/version SID: uuid:subscription UUID TIMEOUT: lifetime of the subscription, in seconds The first event notification message must be sent after the message above. It contains the names of the variables and their current values in XML. In addition, each time that one of these variables, which are set as event variables, changes the service must send an event message to all subscribed control points. These event notification messages are labeled with a different key for each subscriber to detect errors. In every control point, in the initial event message, this key is set at zero and increases with each subsequent notification message. This way, if the subscriber receives a notification with an incorrect key, it will reply to the service with an error message. All subscriptions must be renewed periodically for the control points to go on receiving notifications. To keep a subscription active, the control point must send a renewal message before the subscription expires. The renewal message is sent to the same URL as the original subscription message, but this time it does not include the URL for event message delivery. Instead, the renewal message includes the subscription identifier received in the initial message, which confirmed the subscription. We can see this message format 2.7. UPNP 55 Figure 2.25: Protocol stack for presentation below and, as already mentioned, it must include the blank line: SUBSCRIBE: service route HTTP/1.1 HOST: host:service port SID: uuid:susbcription UUID TIMEOUT: request for the time of subscription, in seconds The answer to this message is exactly the same as in the subscription message case. When a control point does not want to get any more events from a service, it can call off its subscription by sending a cancellation message: UNSUBSCRIBE: service route HTTP/1.1 HOST: host:service port SID: uuid:subscription UUID The answer to this message is, as in the case above, an HTTP confirmation. If the control point abruptly disconnects from the network without sending the message to cancel the subscription, the service will keep on sending it notifications until the subscription time expires. Presentation In a UPnP network, a control point can monitor a device or check its status through the presentation of an HTML page. A home page can be loaded by the control point in a browser and this allows users to view and control the device. The protocol stack required for this is shown in Figure 2.25. Home pages are not necessary; if a device has no home page, it can still be controlled through standard control messages. If the device allows a home 56 Interoperability Figure 2.26: Presentation step [46] page, its description document contains the URL for the presentation page on the label <presentationURL>. This label must always be present. If the device has no home page, the label will be empty. In the presentation phase, the control point sends an HTTP request using the GET method to the presentation URL (available in the device description) and the device then returns to the home page. After loading the page in the browser, the control point can monitor the device or check its variables. The diagram in Figure 2.26 shows this. The presentation message for requests includes the field ACCEPT-LANGUAGE, and the language of the presentation page will be defined by the Content- Language field, which is defined in the device. An additional component of a UPnP network is the application layer. The capabilities of a device are defined by itself and the service models that provide the framework for the network components (description, control and events). A device manufacturer can develop these models by itself or work with other manufacturers inside the UPnP Forum to prepare the standard for the device and the service models. Currently, the working committee for UPnP has developed definitions for standard models. 2.7.6 Application Examples Some modern projects work with UPnP, such as [47], which compares it with other distributed systems such as CORBA. [48] used UPnP to build a middleware layer for a home network. It is important to notice that, for this project, we have used device emulators (TV, fridge, etc.). This will be important for future implementations of the final solution. 2.8. DLNA 57 [49] demonstrated UPnP protocol integration in a system consisting of a robot to manipulate objects. This paper described the design of the system systematically, the data used to compile the XML document of services and the definition of actions and control variables. In [5], there were conclusions to develop a flexible and low cost home automation, which has been implemented using UPnP. This architecture is also used for sensor networks. [50] studied the design of a network with an interface between wireless sensor networks and UPnP via TCP/IP. This application makes possible the communication between control points and sensors and provides the use of web technologies for the control interface. By contrast, [51] discussed the few resources that present the sensing devices using a UPnP agent (BOSS, bridge of the sensors) that acts as an interface between the PC and the elements not supported by UPnP. Currently, you can find different solutions in the market for developing UPnP systems, highlighting initiatives such as CyberLink for Java [52], a Java implementation that automatically controls all the internal aspects of the protocol and allows the programmer to focus on the business layers and the tool’s interface. 2.8 DLNA The DLNA (Digital Living Network Alliance) is an international and collaborative organization of companies involved in consumer electronics, industrial computers and mobile devices. DLNA is a standard that allows different devices from the same network connected together to share information easily and without complicated configurations [53]. This system works with both wireless and Ethernet networks and even with the power supply. The DLNA has established a set of standards for the platforms and infrastructure software to be completely compatible. It focuses on the interoperability among mobile devices associated with multimedia images, digital audio and digital video. Thus, assuming that all available devices on the network support this technology, a copy of the content and the network can be accessed from any device. In other words, we can listen to music from the files stored on our computer, watch movies stored on the digital video recorder on our computer or see photos of our camera on the TV. Figure 2.27 shows a possible scenario using this technology. The objectives proposed by this technology are listed below: •Digital music should be easily captured, stored and accessed from anywhere in the house. Digital photos should be managed, viewed and 58 Interoperability Figure 2.27: Interoperability between devices using DLNA printed very easily. •It must be possible to read content anywhere and enjoy it while traveling by car or walking down the street (there are already projects to synchronize information). •It must be possible to save the distributed content to be able to see it as many times as we want. 2.8.1 General Features The digital home is an electronic network made up of PC and mobile devices that cooperate transparently. The aim of DLNA is to become a home network for all its global customers. This objective integrates the interoperability of the three digital islands within the home: the Internet, broadband electronic network and island of mobile devices (Figure 2.28). The DLNA network must have at least a server and a client to work. The main objective of DMS (Digital Media Servers) is to provide multimedia content to DMP (Digital Media Players), which act as clients. These devices include camcorders, digital cameras, game consoles and mobile phones, but they need to be certified, that is, they must have integrated the electronics and configure to the DLNA standard. DLNA makes use of a part of the technology developed for UPnP that allows the discovery of other devices on the local network. DLNA is based on UPnP and IETF (Internet Engineering Task Force) technologies [54]. The DLNA standard is based on standards established in the industry and developed by groups such as IETF, World Wide Web Consortium (W3C), Motion Picture Experts Group (MPEG) and the UPnP Forum. Interoperability between devices is transparently performed by providing a particular service to 2.8. DLNA 59 Figure 2.28: Objective of DLNA: digital islands at home the user. This includes the ability of the devices to communicate with each other and exchange useful information. The interoperability guidelines require that all devices must support connectivity via Ethernet, Wi-Fi or Bluetooth. It uses TCP/IP for all network connections and works with HTML and the SOAP for transport and media management. The required formats to support images, audio and video are also defined. They are JPEG, LPCM (Linear Pulse Code Modulation) and MPEG2, respectively. DLNA is based on a specification created by the working groups of the UPnP Forum. This specification is the UPnP AV (Audio and Video UPnP), and it has been the greatest success for these working groups, at least in terms of digital content (Figure 2.29). DLNA Model for Devices The model for devices used by DLNA comes from the UPnP Forum and consists of devices, services and control points. The devices are network entities that provide services. These services are the basic control units and they perform actions to keep a state through its variables. The control points are network entities used to discover and control other devices on the network. A group of multiple devices can be controlled by a control point. In the UPnP standard, interoperability was first between the control point and a single device. However, with the evolution of the specification of UPnP AV (and DLNA as well) the basic model of devices was improved. For this reason, although interoperability between the control point and device still 60 Interoperability Figure 2.29: DLNA interoperability model 2.8. DLNA 61 Figure 2.30: Categories and kinds of DLNA devices works, it has been extended to other devices so that they can interact with each other by exchanging digital content using different communication protocols (Figure 2.30). There are 12 kinds of DLNA devices in three different categories. The category Home Network Device (HND) consists of five classes of devices that share the same use on the network system, with the same media formats and connectivity requirements. •DMS. These are devices that can originate, acquire, record and store media on the model of interoperability in the digital home. There are DMS that help to protect the content saved. These devices, in case a customer is not able to handle a particular format, must be able to convert the file into another format before sending. Some examples of these devices include digital video recorders, computers, home cinema with hard disk drives (such as music servers), devices to capture video and images and multimedia mobile phones. We can see the protocols and services of DMS in Figure 2.31. •DMP. These devices select and play the digital media stored on the network and include TV monitors, home cinema, PDAs, multimedia mobile phones, consoles and digital media adapters. •Digital Media Renderer (DMR). Devices that reproduce the content 62 Interoperability Figure 2.31: Protocols and services of DMS received from DMS or their mobile counterparts after being configured by another device on the HND, such as a Digital Media Controller (DMC, see below). DMC and mobile DMC devices will be explained in subsequent studies. Examples of such devices include televisions, audio/video receivers and remote speakers for music. The services and protocols of a DMR are show in Figure 2.32. •DMC. This device has the ability to find content exposed by DMS and adapt it to the rendering capabilities of a DMR, establishing the connections between them. It can also send instructions to another device, such as telling a server to play a particular video on a TV or sending a photo to a printer. A possible example of a DMC could be a learning remote control or a multifunctional device such as a multimedia mobile phone. •Digital Media Printer (DMPr). These devices provide printing services to the home network. Some examples are a network printer or an application running on a PC with a USB-connected printer. The category Mobile Handheld Device (MHD) consists of five classes of devices that use the same model as in the HND category, but have different requirements for media formats and network connectivity. This category includes the following kinds of devices and features: 2.8. DLNA 63 Figure 2.32: Protocols and services of DMR •Mobile DMS. Wireless devices that provide and distribute content to a mobile DMP, DMR or DMPr. Examples of these devices are mobile phones and music players. •Mobile DMP. These devices are able to find and play the content offered by DMS or mobile DMS and play it in a local environment. An example of this kind of device may be a media tablet, which is a portable player with Wi-Fi connectivity that can be used as an Internet browser. •Mobile Digital Media Controller. A device that finds content offered by a mobile DMS and adapts it to the capabilities of a DMR, establishing connections between the server (DMS) and renderer (DMR). A PDA and an intelligent remote control are examples of such devices. •Mobile Digital Media Uploader. These wireless devices send (load) a mobile DMS or DMS with an upload functionality. A digital camera and a phone with an integrated camera are examples of such devices. •Mobile Digital Media Downloader. This finds and downloads the content exposed by DMS or mobile DMS and reproduces it after downloading. An example is a portable music player. MHDs interact with stationary devices in the DLNA digital home and allow a wide variety of uses. Some examples include: 70 Interoperability Attempts have been made to expand the DLNA domain further so a device will be able to connect to any network. For example, [57] implemented the DLNA proxy server to service any virtual network [58]. 2.9 Web Services WS (Web Services) is a technology that allows websites to use distributed applications and offers features such as access to the information and functionalities of any platform. At first, they were created to meet the need to standardize communication between different platforms and programming languages because earlier attempts such as CORBA had little success. In the case of CORBA, this was because there are certain limitations for more complex applications that require a security control or transaction management. WS provide a standard means of interoperating between different software applications, running on a variety of platforms and frameworks. WS are functions or procedures that can be accessed via the web. Regardless of the programming language of the service and its platform, they enable the exchange of data and provide services between different applications. Such a degree of interoperability is only possible using open protocols. WS are mainly used with HTTP because this is widely used and is rarely blocked by firewalls. WS are a set of protocols and standards used to exchange data between applications, and they are used on important websites for tasks such as e-commerce, web browsers and computer services by companies such as Google, eBay or Amazon. The W3C is responsible for managing the specifications. The main features of WS technology and its advantages and disadvantages are listed below: •It is supported by any platform and any programming language. •It is a W3C standard. •It provides functionality to websites. •It uses HTTP to transport data. •It uses standard elements for each of its components (SOAP, UDDI, Web Services Definition Language (WSDL) and XML). - One of the main advantages of WS is that they allow applications to communicate efficiently, regardless of the platforms used, offering greater interoperability. WS use standards and text-based protocols, which allows a better understanding and easier access to the data exchanged. They also use HTTP to allow the information to pass through 2.9. WEB SERVICES 71 Figure 2.36: WS communication architecture firewalls without major complications. This fact together with the use of XML promotes interoperability. - However, WS are much less efficient than are CORBA or RMI because they make use of formats based on text, such as XML, which are not the best options to process tasks. Nevertheless, new WS standards may define more optimized protocols. Also they are not as developed as standards such as CORBA. Both HTTP and XML have a high run-time cost compared with other distributed applications approaches. Skipping the firewall security can also be seen as a drawback. 2.9.1 Components WS use text-based standards and protocols, and this involves the components listed below. Figure 2.36 shows the diagram of the interactions between the entities and flows of the incoming and outgoing data of each component. 1. WSDL It is desirable that WS have information on the operations and data types involved. For this reason, WSDL is used. This is a standard adopted by the W3C that defines the public interface of WS. It is structured as follows: •Ports (<portType>): these describe the operations provided by WS. Its function is similar to an object-oriented class. 72 Interoperability •Messages (<message>): these define the data involved in an operation, where each message can have one or more parts. It is considered one of the parameters used in object-oriented programming. •Types (<types>): these define the data types involved in WS, using XML Schema, an XML language that accurately describes the structures and constraints of the XML file. It has been in the W3C since 2001. •Links (<binding>): these describe the message formats and the protocols for each one of the ports. •Operations (<operations>): these can be one-way, request-response (makes a request and waits for a response), request-response (receives a request and makes a response) or notice. •Services: these define a set of web service ports. 2. UDDI In order to register and publish WSDL we use Universal Description, Discovery and Integration (UDDI). This is a standard developed for the publication and registration of WS. Its way of working is similar to a database and has two different parts: •Registration of business: - White Pages (Overview) - Yellow Pages (categories of services) - Green Pages (business rules) •Registration of services 3. SOAP In addition, there was a need to define the way of exchanging data between different processes on different machines. For this task, we use the SOAP, which defines the format of the messages to send. It is independent of the transport protocol. The elements of a SOAP message are [59]: •Encapsulation of the message. •Description of the data coding. •Body, which contains the specific message of the application. 2.10. SEMANTIC WEB SERVICES 73 Figure 2.37: The emergence of SWS 2.9.2 Application Examples Websites ask WS for a series of functions. They are currently used in almost all websites and they provide most logic to the website. Another possible application of WS is for the control of robots. WS are used to control robots from anywhere in the world via the Internet through a user interface, which will provide the services offered by the robot as well as its status [60]. 2.10 Semantic Web Services SWS (Semantic Web Services) were derived from the combination of WS with the emergence of the semantic web (Figure 2.37). Tim Berners-Lee created the semantic web states that the “Semantic Web is not a separate web but an extension of the current one, in which information is given well-defined meaning, better enabling computers and people to work in cooperation.” WS meet the requirement of a specified syntax; however, they have a lack of semantics so they cannot resolve ambiguities. This is solved by using SWS, optimizing this way the reuse of WS and creating smarter websites, resulting in the concept of Web 3.0. This simplifies the sharing and integration of web resources. To represent knowledge, ontologies that structure information, resources or services based on the meaning of words emerge. This allows computers to interpret and process this information to work automatically. The languages of high-level ontologies are backed by a formal logic, which makes sure that the ontology can be interpreted by the machines. This means that the computer and its software can interpret the semantics of the model without direct human intervention. The ontological software rises to the level of human conceptual knowledge; humans do not have to descend to the machine’s levels [59]. SWS are an important line of the semantic web, which aim to describe not only information but also WS’s functionality ontologies and procedures: its 74 Interoperability inputs, outputs, conditions for implementation, effects produced or steps followed. These machine-processable descriptions will automate the discovery, composition and implementation of services, as well as the communication among them. The semantic web has emerged to provide the syntactic web with semantic intelligence and has the following main features: •Automatic data interpretation. •Ontologies as data models. •Discovery, selection and automatic service composition. •Service implementation through the web. 2.10.1 Required Functionalities •Publication of service descriptions. •Services discovery. •Service selection. •Composition of services. •Resolution of problems caused. •Implementation of automated services. •Monitoring of implementation. •Compensation. •Substitution of services for similar ones. •Verification of implementation. 2.10.2 Main Technologies •Web Ontology Language (OWL-S). This is an ontology based on OWL, which is a markup language for publishing and sharing data using ontologies. It was created by [61], which is part of the US Department of Defense, where they automate tasks such as the discovery, invocation and composition of WS. 2.11. MILITARY STANDARDS 75 •Web Service Modeling Ontology (WSMO). This is a conceptual model for the relevant aspects of SWS and it belongs to the European Semantic Systems Initiative. The WSMO working group includes the technology of Web Service. Modeling Language, which formalizes the WS that model the ontology [62]. Its main components are: - Goals. These are the customer’s aims when they access the web service. - Ontologies. A formal description of the semantics used by all components. - Mediator. These are connectors that provide interoperability among different ontologies. - WS. These can include the functional and usage descriptions of WS. - OWL-S has a weak point in the architecture because it is undefined. It also has little development in comparison with WSMO. Its difficulty is also higher and less intuitive than WSMO is. However, its groundings of use are well developed. However, WSMO is not mature in key areas of use. It has a robust and flexible architecture for the consumer in contrast to OWL-S. It has defined important aspects such as languages and mediation. There are also plans to automate the creation of WS based on WSMO to semi-automate this process, thereby saving money, time and resources; the same as in the IRS III project. 2.11 Military Standards 2.11.1 JAUS The JAUS (Joint Architecture for Unmanned Systems) standard was developed for the US Defense Department [63] by the JAUS Work Group, which is composed of research groups from the government (US Army ARMDEC), industry (SSC San Diego, WINTEC Inc., iRobot) and academia (Virginia Tech, University of Florida). JAUS was defined as an open and scalable standard that would meet the needs related to the communication of unmanned systems regardless of the platform used. The development of JAUS has tried to meet the following six goals [64]: 1. Independence of the vehicle’s platform. 2. Isolation of the mission. 3. Hardware independence. 76 Interoperability 4. Independence from the technology. 5. Independence from the operation. 6. Independence from the connection used. The JAUS architecture is composed of three levels: •Level 1 - Inter subsystem: The purpose of this level is to support interoperability between subsystems. It is responsible for specifying requirements between the subsystems (Robot to Robot, Robot to Controller, Controller to Controller). •Level 2 - Inter nodal: The purpose of this level is to support the interoperability between nodes. To this end, it specifies requirements between the subsystems (interoperability between data loads or between the on-board control and data loads). •Level 3 - Inter components: The purpose of this level is to provide a reusable software source. It specifies requirements for each component (component by component, such as sensors and motors). In 2004, a process of transition from the JAUS Work Group to the Society of Automotive Engineers [65] started. This developed the standard through the AS-4 (Technical Committee on Unmanned Systems) [66]. The following norms migrated from JAUS to a framework based on the following services: •JAUS Transport Standard, AS5669 [67]. This is in charge of defining the creation of packets with the destination and source addresses and TCP and IP headers and links. •JAUS Core Service Set, AS5710 [68]. This is responsible for providing the means for the software entities in an unmanned system to communicate and coordinate among their activities. •JAUS Mobility Service Set, AS6009 [69]. This is in charge of making the migration from the first drivers to the new development platform of the AS-4. Today, the main application of JAUS is focused on the use of unmanned civilian and military vehicles. 2.11. MILITARY STANDARDS 77 Application of Military Unmanned Vehicles A major center for development of military unmanned vehicles exists at the SPAWAR Systems Center (SSC) in San Diego (California). There, a JAUS work team focuses on the development of surveillance systems, such as MDARS (Mobile Detection Assessment Response System), which are used in autonomous vehicles for military bases with restricted access. The US Defense Department uses MDARS to meet security and surveillance needs in hostile environments for humans. In this way, it provides an integrated solution, where unit patrol vehicles are controlled just by a single control operator. Moreover, SSC has developed a distributed processing system called Multiple Resource Host Architecture [70] which, along with MDARS, was tested by the JAUS work team in December 2003 to demonstrate the level of interoperability between control operator units (COUs) and unmanned systems [71]. In this experiment, COUs were equipped with a screen capable of displaying the statuses of each patrol vehicle, and thereby they controlled each one of the unmanned systems [72]. These experiments show how the JAUS architecture provides interoperability for the remote control of unmanned systems while fulfilling the objectives mentioned in the general characteristics section. Application of Civil Unmanned Vehicles In 2004, Virginia Tech launched a project to implement simultaneously the JAUS standard in the following seven unmanned vehicles: 1. MATILDA This was the first interoperable vehicle designed by Virginia Tech in 2002. It was designed as an evaluation, development and demonstration platform of the JAUS standard. It had to fulfill some functional requirements: •It had to be teleoperable through a COU. •It had to be capable of driving autonomously via GPS commanded by a COU. •It had to interact with other subsystems of JAUS (either vehicle or COU). •It had to accept JAUS workloads from other devices. •It had to allow an easy modification and/or addition of intelligent software. 78 Interoperability Figure 2.38: JAUS topology •It had to ease the demonstration, evaluation and testing of the JAUS standard. 2. JOHNNY-5 This was developed in 2004 to participate in the AUVSI Intelligent Ground Vehicle Competition in 2005. Owing to its robustness and its capability to navigate via GPS, it quickly replaced MATILDA. The main problems of this model were the failures in the camera interface and the starting force on the wheels. 3. CADILLAC SRX Grant Gothing and Jesse Hurdus, researchers from Virginia Tech, managed to implement the JAUS standard on the Cadillac SRX, creating the first luxury unmanned vehicle in the world [73]. The challenge of this model depended on development of a JAUS-based vehicle able to use potential field methods [74] for navigation. The result was the creation of a software topology, based on operational subsystems, nodes and components (Figure 2.38). However, when they launched this vehicle in the Blind Driver competition [75] they detected some issues that could be improved [76]. For example, every driver had to know the turn angle of the vehicle and, according to the control messages of the JAUS specification, only one controller per component was allowed. 4. GEMINI Gemini was developed as an extension of Johnny-5. The idea was to create an articulated robot with four wheels. It won the JAUS Award at the AUVSI Intelligent Ground Vehicle Competition in 2006 because 2.11. MILITARY STANDARDS 79 of its refined design, the long life of its batteries (5 h), its innovative mobility and the ability to deal with bigger workloads under the JAUS architecture. 5. HELIUM RED (Unmanned Ground Vehicle; UGV) and THE RMAX (UAV) HeLiUm RED (HElicopter LIfted UnManned Reconnaissance and Exploration Drone) redefines the traditional notion of collaboration between UAVs and UGVs (RMAX-HELIUM THE RED). This small unmanned vehicle is light enough to be carried by the VT Yamaha RMAX UAV. Initially, the JAUS standard was implemented to simplify communication with the vehicle; however, vehicles are usually treated as subsystems of the JAUS architecture, but in the project HELIUM RED, the UGV operates as a single node. 6. ROCKY This is another example of the vehicles used by Virginia Tech in the DARPA Grand Challenge. The JAUS implementation in Rocky has taken place in two stages: •Teleoperability: Through the primitive driver, they could make sure that the vehicle was teleoperated making use of the COU, but nowadays with the use of Global Position/Speed Sensors, the COU, speed and position can be kept on track and transmitted through a connection service. •Portability of the basic code from Cadillac SRX directly to Rocky. This feature can be seen as a demonstration of the reusability existing when developing autonomous vehicles under the JAUS architecture. Owing to these achievements, Virginia Tech established, as functional requirements, that their prototypes had to be interoperable with other JAUS subsystems (applied to both COUs and vehicles). Throughout this research, they realized the need to integrate some specifications in the JAUS Service Specification standard that would make use of messages in charge of waiting for a response that will allow the COU and the vehicles to make behavior decisions for a better interaction between them. With respect to the development of unmanned vehicles, the company TORC started the ByWire XGV Project [77]. This project is being 86 Interoperability Figure 2.41: OROCOS libraries [109] ment; finally, these images are processed by a computer vision node to find particular objects in the environment. 2.12.7 OROCOS Orocos (Open Robot Control Software) is an Open Source C++ software framework for building real-time component-based applications in automation and robotics [108]. It is a general-purpose and modular framework for robot and machine control. The Orocos project supports 4 C++ libraries (Figure 2.41): •Orocos Real-Time Toolkit (RTT) provides the infrastructure and the functionalities to build robotics applications in C++. •Orocos Components Library (OCL) provides some ready to use control components (Component management and Components for control and hardware access). •Orocos Kinematics and Dynamics Library (KDL) is a C++ library which allows to calculate kinematic chains in real-time. •Orocos Bayesian Filtering Library (BFL) provides an application independent framework for algorithms implementation such as Kalman Filters or Particle Filters. 2.12.8 OpenJAUS OpenJAUS [110] is a free Open Source middleware implementation for Unmanned Systems based on the Joint Architecture for Unmanned Systems 2.12. OTHER TECHNOLOGIES 87 Figure 2.42: jROS design [110] (JAUS). OpenJAUS emerged as the libraries and applications developed at the University of Florida in the 2005 DARPA Grand Challenge. OpenJAUS has created jROS, a software library to integrate JAUS and ROS solutions, which is required to use a JAUS interface for a ROS-based system or vice versa. jROS is distributed as a set of ROS Packages which consist of custom ROS Messages for each of the JAUS Service Sets (i.e. Core, Mobility, Manipulators, etc). Figure 2.42 shows how OpenJAUS uses code generation to automatically create the jROS bridge. 88 Interoperability Chapter 3 Integration of Service Robots in the Smart Home by means of the Universal Plug and Play Protocol The digital home concept has recently emerged. It is characterized by a network of sensors and actuators which provides services in terms of comfort, security and energy management. Moreover, the future points to the paradigm of ambient intelligence whereby home should automatically adapt to the needs of its inhabitants. The emergence of home automation systems was launched in the late 1970s with the development of a system based on X-10 Power Line Carriers. From that moment up to the present, the number of home automation systems has increased significantly: KNX, Lonworks, Zwave, Delta Dore, Control4, Cardio, ... In this sense, it is remarkable the situation in Spain, where the variety of home automation systems is even larger because of its real estate bubble in the beginning of this century. Because of the large investments made in this sector new systems emerged, such as IPDomo [111] or Ingenium [101], which were able to survive despite the subsequent real estate market drop. Not only the large number of home automation systems should be considered, but also the wide variety of consumer electronic devices that can be integrated within the digital home network. Different interoperability standards have been proposed in [6], [112] and [113] in order to solve the problems resulting from the integration of such heterogeneous systems. In this sense the Universal Plug and Play (UPnP) [114] is the standard that has achieved 89 90 Integration of Robots in the Smart Home by means of UPnP the greatest success, although with some nuances. Nowadays, multiple heterogeneous electronic devices coexist at homes. However, most of these systems are independent and only ad hoc integrations between them are possible. Given that smart home applications are based on the integration in a same network of a set of sensors and actuators, it is obvious that this lack of interoperability hinders the development of more advanced services for the home inhabitants and it is a major issue in the road towards the ambient intelligence paradigm [115]. This problem is not new, though there have been several attempts to standardize the way in which heterogeneous systems can be integrated [116]. The first important attempt to solve this problem in a systematic way was CORBA (Common Object Request Broker Architecture) in 1991. CORBA is a distributed object oriented architecture. It allows to integrate distributed objects regardless of the hardware architecture, operating system and programming language [1]. In 1998 Sun proposed a different standard, Jini, with the goal of setting the basis for the definition, publication and searching of services in a network [20]. A similar alternative was given by UPnP, which was proposed by Microsoft [41] in 1998. UPnP has been very successful in the field of multimedia interoperability. For this reason, there are other interoperability standards based on UPnP. This is the case for example of DLNA (Digital Living Networks Alliance), whose main focus is multimedia interoperability [57]. Another remarkable standard is OSGi (Open Services Gateway Initiative), which is focused on domestic applications. OSGi defines its own architecture, but it is designed to work together with other protocols such as UPnP or Jini [117]. Finally, some authors [118] bet on web services as the solution for the interoperability problem. Despite the absence of interoperability, the relevance of technology in our lives has grown significantly during the last decades. In particular, the evolution of the role played by robotics is remarkable. Despite the fact that the use of robots has traditionally been associated to military and industrial applications, it is common to find them now even in domestic environments. The purpose of UPnP is the integration of new devices within the home network transparently to the user as many devices that operate on a computer. To that end, the devices in a network exchange information about the services and their capabilities that may provide to the network. In addition, these devices must expose their services to be invoked remotely. The most successful application field of UPnP is the interoperability between multimedia devices. This fact is confirmed in different studies such as [119] or [120]. Nowadays, the UPnP features related to audio and video streaming are widely used by many devices. However, UPnP is not so employed in other 3.1. SMART HOMES AND ROBOTS 91 different fields. Nevertheless, the success of UPnP is greater than other interoperability standards that have aslo been proposed, such as OSGi and Jini [116]. This chapter presents a methodology for creating UPnP virtual devices. This methodology can be used to ensure real systems interoperability by means of UPnP gateways working on a computer. In other words, it allows the creation of hybrid UPnP devices that are capable of translating UPnP orders into actions performed by real systems. In this chapter we explain how to design UPnP devices via software to integrate robots in the smart home. The problem of integrating service robots in the smart home is also addressed in this chapter, and the integration of two service robots by creating UPnP gateways is detailed. The stages for the creation of UPnP devices have been detailed in the Paraninfo’s book “Home Automation for Engineers” (Domotica para ingenieros) [121] written in spanish language. This chapter contains information extracted from the paper “Robots in the smart home: a project towards interoperability” [44] published in the international journal IJAHUC in which the Roomba robot and its integration in the smart home is deeply studied. In this chapter, the Rovio robot and its inclusion in the digital home through UPnP is also analyzed. This research has been published in the international journal RAS with the title “Integration of service robots in the smart home by means of UPnP: A surveillance robot case study” [122]. We have developed UPnP applications for each robot and different experiments have been performed to illustrate the possibilities derived from the integration of service robots with home automation technologies. The outline of the chapter is as follows: In section 3.1 Roomba vacuum cleaner and Rovio surveillance robots are presented. Section 3.2 details how to create virtual devices based on the UPnP architecture. The next section explains the UPnP application developed in order to integrate the robots in the digital home. Some experiments related with the robots were carried out in the next section. Finally, the conclusions are presented in section 3.5. 3.1 Smart Homes and Robots Nowadays, the use of service and industrial robots is spreading worldwide. In general, robots reduce costs and improve the precision in the tasks they are assigned to do. Moreover, they substitute human beings in many complicated tasks. The take-off of robotics is motivated mainly by the reduction of production costs. Service robots are still at a very early stage. Until now, the definition for 92 Integration of Robots in the Smart Home by means of UPnP service robot was not well defined. The International Federation of Robotics (IFR) [123] had defined the term service robot as “a robot which operates semi or fully autonomously to perform services useful to the well-being of humans and equipment, excluding manufacturing operations.” However, it seems that the IFR has specified the definition: “A service robot is a robot that performs useful tasks for humans or equipment excluding industrial automation application.” Other entities as Hisparob [124] defined service robotics as: “...nowadays industrial robotics extension in which new robots are being developed to answer the industry needs not directly linked with the productive tasks...” The most basic way to interpret these definitions is that any robot that is not used in manufacturing or production operations is a service robot. For the purposes of this chapter, this simple definition is enough. Service robots can be classified into different types. We present a classification inspired on the one provided by the IFR: •Professional service robots This group includes robots (Figure 3.1) capable of collaborating in tasks performed by humans. These robots are designed to enhance the capacities of people in their jobs. Examples of this type of service robots are: Communication Robot, developed by AEON MALL [125] in collaboration with Tmsuk Co. [126]; An-9RR created by the Japanese company ALSOK [127]; da Vinci Surgical System designed by the company INTUITIVE SURGICAL [128], among others. 3.1. SMART HOMES AND ROBOTS 93 Figure 3.1: Professional service robots •Personal service robots Personal service robots (Figure 3.2)have become a reality in many houses. They can solve daily tasks at home. They improve the quality of life of their users. The Roomba is included in this category. Besides the Roomba, the following can be included: Asimo, presented by Honda [129]; Aibo, developed by Sony [99]; Scitos G5 created by MetraLabs Robotics [130]; Home Assistant Robot is a joint research between Toyota [131] and the University of Tokyo, among many others. •Defence robots These robots (Figure 3.3) are prepared for dangerous situations such as deactivation of explosives or intervention in contaminated areas. Within this group, we can cite the following robots: Pointman, designed by Applied Research Associates (ARA) [104] or mSecurit, developed by MoviRobotics [103]. Service robots will bring a robotic revolution to homes, streets, schools, hospitals, etc. One company, which has taken some of the steps towards penetrating service robots into the home, is iRobot Corporation [132], the creators of the service robot Roomba. Roomba is an autonomous robotic vacuum cleaner that has sold more than 10 millions of units worldwide. 94 Integration of Robots in the Smart Home by means of UPnP Figure 3.2: Personal service robots Figure 3.3: Defense robots 3.1. SMART HOMES AND ROBOTS 95 In this chapter, we are interested in how service robots can be integrated in the digital home through middleware. The problem is complex because of the large number of systems available in the home automation market. The absence of real interoperability between the systems makes the development of new services based on the combination of heterogeneous systems more cumbersome. Given that at the present time there is no ideal choice between all the technologies that can be installed in the home, the best choice to develop new advanced services comes from mixing different systems and technologies in the same installation [133][134]. Given that one of the most important objectives of smart homes is the development of new services that make the life of its inhabitants better, service robots should be easily integrable in a smart home environment. The variety of robotic and smart home information systems’ companies represents the main problem for the interoperability approach. The absence of minimum requirements makes it difficult to assume the existence of systems or services in the home. The consequence is that difficulties arise for service developers. At this point, it is needed to emphasise that the interoperability among systems deviates far from the simple exchange of information. The intercommunication among the information systems that support the new services must satisfy a large number of needs. The use of mark-up standards, consult services and some web services provide the functionality and interoperability required but there are also some other important issues, such as the ability of interaction and parallel implementation tasks, to name a few. UPnP has been successfully used for home automation, see for example [5], and it has been considered as a middleware platform to integrate robots [135]. In addition, UPnP is an important element of other home networking standards such as the Digital Living Network Alliance [136] (formerly the Digital Home Working Group) and Intel’s Networked Media Products Requirements (NMPR) specifications. Despite its relative success, UPnP still has some drawbacks that should be improved to be the ideal middleware platform, specifically in the case of robot integration. Here, we show a list with some of the features that have to be improved. The list is based on works [137][116], and it has been complemented with other deficiencies that we have found in the standard during our research. •Lack of priority mechanism for messages. •Lack of a deterministic time of response for commands. •Lack of a mechanism for synchronous request and responses. •Unclear definition for elements in the network that assume functions of 102 Integration of Robots in the Smart Home by means of UPnP •Speaker for 2-way audio •3 omni-directional wheels •Headlight (LED illumination) The Rovio API [155] is based on the HTTP protocol. In particular, Rovio works by means of CGI commands within HTTP GET requests. CGI stands for Common Gateway Interface and is a World Wide Web technology which allows a web-based client to send and require some data from an application which is executed in a web server. In this manner, Rovio’s web server passes user’s request to the CGI application program which controls the robot. Note that this implies that the only requirement to create a Rovio control application is to create a TCP/IP binding with its web server in order to send the HTTP requests and receive the corresponding responses. Therefore the communication with Rovio does not depend on any particular platform, programming language or operating system. There are nine categories of CGI commands which can be sent to the Rovio robot: movement control, camera control, user management, time, network, MAC address, HTTP server, mail and other commands. For example, the command HTTP://192.168.1.2/rev.cgi?Cmd=nav&action=1 corresponds to a HTTP request to the IP address 192.168.1.2 whose argument is a CGI command to order Rovio to go to its home location in front of its charging dock. More details about other commands of the Rovio API can be found in [155]. 3.2 UPnP Virtual Devices Development In chapter 2 we explained the operation of UPnP. The steps to follow in order to develop UPnP virtual devices are detailed below. Note the emphasis on the virtual nature of the device, i.e. they are not physical devices. Therefore, it is possible to define a virtual UPnP device as that which is only implemented at a software level and is executed on a general purpose computer as a PC. We have classified UPnP devices in three groups: physical, pure virtual and mixed virtual. These types of devices differ in how they are implemented. •Physical: These devices are a physical implementation, i.e., the device is a printed circuit board (PCB) or a real system that provides a UPnP interface and must be fed properly. Specifically, a physical UPnP device is the hardware and software that provide the UPnP functionality. Within this group we can include the IPDomo cards for home automation. 3.2. UPNP VIRTUAL DEVICES DEVELOPMENT 103 •Virtual: This group includes devices implemented at a software level. An example of a virtual device may be a weather station. •Mixed: They are characteristic virtual devices that translate orders from the UPnP network to another physical system, so they are at an intermediate point between the two previous groups. They are gateways that provide a UPnP interface and another interface to the corresponding system. On the one hand, it collects data from the UPnP network, interprets and translates the information into commands known by the system at the other side which actually perform the action; on the other hand, it collects data from the agent and provides them to the UPnP network. In this group we can include the UPnP gateways for the Roomba or Rovio robots. As it can be deduced from these devices, both the virtual and the mixed devices require a computer program to provide support for the connection in a UPnP network. The development of virtual devices is also useful as the basis for adapting non-UPnP devices within a UPnP network. To that end, it is necessary to program a virtual device that works as a gateway between the equipment you want to provide UPnP functionality and the UPnP network. In general, the development of virtual devices would be very complex if we had to write all the code associated with the UPnP protocol. This difficulty may be avoided using any of the existing development kits for UPnP such as Allegro [156], Cybergarage [157], GUPnP [158] or the Atinav [159]. In particular, this chapter explains how to use the Intel SDK for UPnP [160]. This development kit is a suite of programming tools for UPnP which can be downloaded for free on the Net. The kit generates a code template that implements the basic features of UPnP, so developers may focus on developing the code associated with the functionality of the device. The template may be generated in different programming languages (C, C++, C#). From all the programming languages, we have chosen C# for the development of the Roomba and Rovio UPnP gateways. The design of a UPnP device can be summarized in four main steps or stages which are described below. 3.2.1 Step 1: Creation of the Service Description The first step is to create a description file for each of the services the device will provide. This file contains information related to the actions and the state variables that characterize the service (Figure 3.7). In this step we will use the Service Author program which is one of the applications included in the Intel tools. This program simplifies the definition of variables, actions 104 Integration of Robots in the Smart Home by means of UPnP Figure 3.7: Creation of the device service description and arguments that form a given service. Firstly, it is necessary to define the state variables required. In particular, it is necessary to specify the format of each variable and if it is evented type or not. When the value of an evented type variable changes, it generates event notifications to the control points subscribed to this variable. This is the basis of subscription and event notification in UPnP. In order to create the service description file it is also necessary to declare the actions. Each action may optionally receive input arguments or generate output results. To that end, some state variables previously defined must be employed. In particular, the type of each variable must be specified: input, output or return. Thus, if the state variables are declared as input variables, they will be arguments or values sent to the device when the user invokes the actions. On the contrary, if they are declared as output or return variables, they will be values returned by the device and stored in such variables during the execution of the action; or as the result of the action. Once the actions and the state variables are declared, all this information is saved in an XML file. Therefore, a device will have as many service description XML files as services offered. Therefore, a device has so many XML service description files as services it provides. Note that the actions and the state variables must be created separately for each service. 3.2.2 Step 2: Creation of the UPnP Stack The second step employs the service description files created in the previous step to generate the device UPnP stack (Figure 3.8). The stack contains all the UPnP protocols as it is illustrated in Figure 2.13, i.e., it contains all the code responsible for the registration and announcement of the device and its services, and manages the subscriptions and requests whith the devices in a UPnP network. The creation of the stack requires the use of the Device 3.2. UPNP VIRTUAL DEVICES DEVELOPMENT 105 Figure 3.8: Creation of the device stack Builder program which is another tool from Intel. This program supports the creation of UPnP stacks for both control points and devices. In case of developing UPnP devices is necessary to import XML service description files from the previous step. It is possible to customize certain parameters that characterize the device or control point in the network. For example, it is possible to edit the Friendly Name of the device that is the name by which the device will be identified in the network; or the Service Name and the Service ID attributes of each service, i.e., the name and identifier of each service. After importing the service description files and editing the attributes required, the developer must generate the code in the chosen programming language for the development of the application. This will create all the code files required for the compilation of the device or the control point. Note that at this point it is available the source code required to compile and build a fully functional device at a network level. 3.2.3 Step 3: Implementation of the Actions Despite the code required to create a UPnP device is available after the previous step, it is essential to type the code that implements the functionality of each action defined in the first step. Otherwise, the device does not respond to any requests made by other entities in the network since it has no code to do it. Therefore, it is necessary to use an editor and a compiler for the 106 Integration of Robots in the Smart Home by means of UPnP Figure 3.9: Operational testing full development of the device. The UPnP gateways detailed in this chapter have been developed using the Visual C# 2010 Express [161] program. Obviously, the code in this step is custom designed for each application. Also, note that it is not necessary to implement only the functionality of the UPnP actions. In other words, the program can be as complex as desired and, even, a very complex program may provide only a simple UPnP action. Finally, the resulting code is compiled and an executable file is obtained. From this moment on, any UPnP control point can control the virtual device. 3.2.4 Step 4: Operational Testing and Validation of the Device Finally, you should check the correct operation of the device developed. This procedure could be considered as another step in the design process of UPnP devices since the device may provide a non-expected functionality or cause errors during execution. To that end, the application is run and is tested with any control point. The Device Spy program is an Intel tool that allows to invoke actions and to subscribe to events. This verifies the proper operation of our UPnP device (Figure 3.9). If you detect any deficiency in the operation of the device you should check the code implemented during the third step. However, if there is a deficit in the services provided by the device, you should change some details from the first and second steps, and add the appropriate code associated with the third step. 3.3. UPNP GATEWAYS FOR ROBOTS 107 Figure 3.10: UPnP application for Roomba robot 3.3 UPnP Gateways for Robots 3.3.1 Roomba Gateway The first gateway developed in this thesis was the UPnP Roomba bridge. It is an application (Figure 3.10) that offers Roomba’s services and state in a UPnP network. Specifically, this application is a UPnP virtual device that works as a bridge intercommunicating the home automation network and the Roomba (Figure 3.11). Roomba’s service provides the actions programmed and the necessaries variables to know Roomba’s state. Users may invoke actions and Roomba will execute them transparently. Implementation The first problem to be solved is the physical communication. The virtual device establishes the communication with Roomba using a Bluetooth modem, called RooTooth (Figure 3.12). RooTooth is connected into Roomba’s external serial port. Thus, what the virtual device does is to collect the actions from the UPnP control points and translate them into Roomba ROI commands that are transmitted via Bluetooth (Figure 3.13). Once that communication between the Roomba and the PC is solved, we can focus on the creation of the UPnP device. This procedure was exposed previously and during the step of the implementation of the actions, most 108 Integration of Robots in the Smart Home by means of UPnP Figure 3.11: Roomba UPnP gateway Figure 3.12: RooTooth 3.3. UPNP GATEWAYS FOR ROBOTS 109 Figure 3.13: Roomba UPnP-Bluetooth bridge stack hand-written code is dedicated to managing the communication with the RooTooth and sending via the serial link the command opcodes that will be obeyed by Roomba. A description of all the opcodes can be found on the iRobot SCI document [147]. After this step, an executable file is obtained. From this moment on, any UPnP control point can control the virtual device. The virtual device works like any standard UPnP device, i.e., it follows the operational steps shown in Section 2.7: •It gets an IP address from a DHCP server or the AUTO-IP mechanism (addressing step). •It announces its services and control points discover them (discovery step). Next, the device sends information about itself and its services to the control point (description step). •Upon discovery, the control points are able to control Roomba transparently; they have a list with the actions users may invoke and a list with the variables they may check (control step). •It is even possible to include evented variables (eventing step). However, it was preferred not to implement them to reduce the UPnP traffic in the network. Therefore, to have an updated list of the variables, our virtual device polls the Roomba periodically and saves the data into internal variables. Then, when users invoke the proper action, the bridge will update and return the value of the UPnP device variables. 110 Integration of Robots in the Smart Home by means of UPnP Actions and State Variables Once the application is running, it can be detected from a UPnP control point. The variables and services offered to the network were chosen carefully as the ROI allows the control of an overwhelming number of options. To keep the application as simple as possible, the device offers the same basic actions that can be controlled with the Roomba buttons complemented with some extra features for the digital home. In Figure 3.14, it can be seen a schematic representation of the virtual device. As can be seen in the figure, the following variables were implemented: •Port (string): Stores the virtual serial port used to connect with the Roomba. •Macro commands (string): Stores a path to be followed by the Roomba. The format is composed of instructions to go forward (U), backward (B), turn to right (R) or turn to left (L) followed by the time in milliseconds to go in the specified direction. For example, if the string is “U700L200”, the Roomba will go forward for 700 ms and then it will spin leftwards for 200 ms. •Dirt (unsigned 8-bit integer): Stores the dirt level detected by the capacitive sensor. •State (string): Stores the Roomba internal mode. •Battery charge (unsigned 16-bit integer): Stores the charge of Roomba’s battery in milliamperes-hours (mAh). •Distance (signed 16-bit integer): Accumulates the relative distance travelled by Roomba in millimetres (mm) since the device was executed. •Angle (signed 16-bit integer): Provides the relative angle that Roomba has rotated. To reduce the network traffic, these variables are not evented. It is needed to invoke an action so that the application updates the published value of the variables. Otherwise, the Roomba virtual device would be constantly sending event messages because the value of the sensors is continuously changing. An option to lower the network traffic with evented variables would be to implement a threshold in the change of value of the variables before proceeding to their update. The actions we have implemented are: 3.3. UPNP GATEWAYS FOR ROBOTS 111 Figure 3.14: Roomba UPnP virtual bridge scheme 118 Integration of Robots in the Smart Home by means of UPnP 3.4.1 Robots in a UPnP Network The first experiment to test the UPnP gateways was to control the robots through the control point included in the Intel UPnP Tools. In [162] a case of use with the Rovio robot is shown. It consists of invoking the Connect action, then the robot state is obtained by invoking the Get Status action. Afterwards, a movement sequence is performed by means of the Macro Movement action, then the robot speed is changed by invoking the Set Speed action, and finally the same movement sequence is executed. It worked as it was expected, the robot offered all the services to the UPnP network and it performed all the actions invoked by the control point. The Roomba robot was also tested with this tool and every action was executed to check the proper operation. In addition, the UPnP devices were tested with a different manufacturer equipment, in particular, in an installation of IPDomo [111], a native home automation system. Once the application was proved to work in lab conditions, it was tested in a real smart home. The IPDomo installation was composed of a PC in which it was installed the IPDomo control point and several multifunction and security cards. Thanks to this equipment, a total amount of 4 lights, 4 blinds and different sensors (presence, gas and fire detectors) were controlled. The main difference between the control points of Intel and IPDomo is that the second allows the programming of macros with all the UPnP devices present in the network. The Roomba application and the necessary Bluetooth stick were installed in the same PC that ran the control point. Once the Roomba software bridge was executed, the UPnP device was recognised immediately and some macros were programmed. For example, cleaning tasks were scheduled in some days if and always if the condition that there was nobody at home in the moment the cleaning was fulfilled. In addition, the Rovio UPnP virtual bridge was executed in the same PC as the IPDomo control point and macros were programmed to test the operation of the device in the installation. The system was configured to invoke the Surveillance action in the case that presence sensors detected something. This is an useful application in order to reinforce the security at home. As we checked from these tests, the UPnP devices work well in a common UPnP network and they perform all the actions perfectly. 3.4.2 Roomba tests Different test were carried out with the Roomba robot: •Trajectory tests: Although it has not been developed the necessary software to calculate the position of the Roomba using the measurements 3.4. EXPERIMENTS 119 Table 3.1: Dirt sensor tests Dirt Measurement Chickpeas 2 Beans 51 Star-shaped noodles 128 Lentils 251 provided by the wheel encoders, it is still possible to control the movement of the vacuum cleaner in open loop. The Roomba has shown a very lineal time-distance relationship when the action Macro Function is invoked. The repeatability in the behaviour of the robot is good enough to guarantee that Macro Function can be used to command Roomba to go to a room. •Dirt sensor test: The dirt sensor provides a value between 0 and 255 as a function of the dirt density. This information is useful to determine the type and degree of dirtiness in the house. The tests consisted of spreading over a small area typical cooking ingredients on the floor and the results are shown in Table 3.1. The conclusion that can be extracted from the experiments with this sensor is that it registers higher values as the flow of individual elements grows. It is also remarkable that if the dirt size is equal or greater to chickpeas then jams may occur in the main brush of the cleaner. •Distance and angle sensors: We also made some experiments related to the use of the distance and angle sensors. Our results showed that there exist linear relations between the distance and angle displacement of the robot and the measurements obtained. The results show a correlation between the speed of the robot and the quality of the measurements. The faster the Roomba goes the worse performance of the measurements was obtained. This application also depends dramatically on the reliability of the communication link between the control point and the Roomba. In case, there are lost packages, the estimated location of the Roomba can be very wrong. We recommend the interested reader to see [142] to get more information about location of the Roomba using only information coming from its own sensors. 120 Integration of Robots in the Smart Home by means of UPnP Figure 3.19: Tool for statistical measures 3.4.3 Rovio tests The tool shown in Figure 3.19 facilitates the measuring task and it allows to study the movements of the Rovio robots. It is possible to adjust the direction of the movement, the speed and the movement duration in order to get the displacement and rotation measured by Rovio’s positioning system. The statistical operation of this tool is determined by the number of iterations of each test. In each iteration Rovio gets some samples of its position and the mean and standard deviation is calculated. The robot accomplishes the movement and the mean and deviation of its position is calculated again. The tool will ask for manual measures if the corresponding tick box is marked. Then, a new test iteration will start. After the last iteration the tool provides the mean and standard deviation of the robot displacement/rotation, and a relation between the manual measures and that provided by the positioning system. In order to get the model of each robot movement, several tests have been developed. All tests have been performed for three different time intervals for each direction, and linear interpolation has been used between samples. The results obtained are illustrated in Figures 3.20, 3.21 and 3.22 and the 3.4. EXPERIMENTS 121 Figure 3.20: Forward Figure 3.21: Backward 122 Integration of Robots in the Smart Home by means of UPnP Figure 3.22: Left rotation Table 3.2: Rovio movements models Direction Speed Equations of motion Forward High y [mm] = 294,46x - 54,3 Medium y [mm] = 266,9x - 37,67 Backward High y [mm] = 277,63x - 58,41 Medium y [mm] = 253,06x - 25,97 Rotation High y [radians] = 3,068x + 0,004 Medium y [radians] = 2,07x - 0,125 models are represented in table 3.2. 3.4.4 Rovio Robot Tracking a Ball In this experiment we have used the Rovio’s webcam to track a ball. We have implemented this tracking application as the base for the tracking of more complex objects as it will be shown in the next subsection. In order to detect the ball, the robot takes periodic pictures using its webcam and processes these images in real time looking for the ball. When the robot finds it, Rovio moves in order to maintain a certain alignment and position with respect to the ball. The ball-tracking application is based on the x-coordinate of the centroid and the size of the ball. Each image taken by Rovio is processed, in order to get the centroid and the radius of the ball, by applying several image filters available in the AForge [163] and OpenCV [154] libraries: •HSL: allows to extract objects from the image with similar features 3.4. EXPERIMENTS 123 in hue, saturation and lightness. It has been used to get the ball by adjusting the saturation parameter. •Median: is used to reduce noise in the image. •Grayscale: allows to convert a color image into a gray scale image. •Gaussian: is similar to the median filter. It is used to blur and smooth the images and it removes details and noise. •Circular Hough Transform: detects the presence of circular shapes on an image. For this application it has been configured to detect one circle by adjusting the filter parameters. It is the last image processing and it provides the radius and the centroid coordinates if the ball is on the image. Once the application detects the ball, it calculates its centroid and it tries to align the robot with the ball by performing rotary and lateral movements. In addition, when Rovio has centered the ball, it tries to keep at a certain distance from the ball using its radius to measure this distance. If the robot loses the ball, Rovio tries to find it on the direction that the ball was moving. On the other hand, if the Rovio is moving and it detects the ball, and it has disappeared in the following image, it will spin the opposite of its last move. Finally, after looking for the ball, if the robot does not find it, the Rovio will move randomly. As we can see in [164], it has been developed an application for Rovio that is able to look for the ball, to align with it and to follow it if you move the ball. 3.4.5 Rovio Robot Tracking a Roomba Robot In this case we have developed an application that allows the Rovio to follow the autonomous vacuum cleaner Roomba. In order to simplify the tracking problem, we added red rectangles around the Roomba. Thus, the image processing is based on that performed to detect the ball. First an HSL filter is applied to get red objects by adjusting the hue and saturation parameters, then, a grayscale filter is applied. Next, the image processing is a bit different and follows these steps: •Segmentation: this process allows to locate objects and boundaries. The contours extracted from the image are filtered with a size filter in order to remove unwanted red small objects. 124 Integration of Robots in the Smart Home by means of UPnP •Sobel edge detector: performs a spatial gradient on an image and emphasizes regions of high spatial frequency. It detects and remarks the edge of the objects previously filtered. •Rectangles validation: this step consists on look for objects that fulfill some geometrical conditions. The object should be composed of four vertices, and thus it should have four sides. In addition, two consecutives sides should form an angle close to ninety degrees. Once the rectangles have detected, the average of the centroids and the average height of the rectangles’ sides are calculated. The mean centroid is used to determine if Rovio has focused Roomba or not, and the direction that Roomba is following. The mean height is useful in order to know if Roomba is too far or too close to Rovio. The Rovio spins on its axis looking for Roomba. When it identifies some of the red rectangles included on Roomba, Rovio moves towards or away from the robot depending on the average height of the sides. If Roomba goes forward the Rovio will follow it. If the Rovio does not find Roomba during the pursuit, it will turn in the direction where it was shifting the mean of the centroid and it will go forward to find it again. The goal of this application is to follow the Roomba in order to know its position through the Rovio. Samples of this experiment are available in [165] and [166]. 3.4.6 Garbage Detection Another interesting application in a domotic environment is the detection of garbage by means of image processing. In this scenario, Rovio is programmed to patrol the house while looking for garbage. In case of garbage detection, an UPnP event is generated and the digital home acts consequently. For example, an autonomous vacuum cleaner robot such as the Roomba [44] could be sent to clean the area. An example of this is available in [167]. This scenario is a good example of the digital home operation based on the interoperability between domotic and robotic devices. Rovio acts as a mobile sensor since it moves and can detect garbage, and the vacuum cleaner robot has the role of an actuator performing a cleaning task. Both devices are coordinated by a control point which manages this operation. It is important to remark that precise detection of garbage in an arbitrary floor is in general a very difficult problem which is beyond the scope of this work. For this reason, we have developed a simplified garbage detection application which allows the Rovio to detect garbage in some particular scenarios. 3.5. CONCLUSION 125 Within a testing framework, we have developed an application that looks for garbage when a control point invokes the Garbage Detection UPnP action or when the user enables it directly through the GUI. In addition, users may adjust three parameters in order to detect garbage: 1. The percentage of the bottom of the image to be analyzed. 2. The threshold to binarize the image which processes and transforms it into a black and white image. 3. The percentage of black pixels after the threshold step that are necessary to consider them as garbage. If Rovio detects garbage it announces it to the UPnP network by changing the value of the boolean evented variable (Garbage). Even, it is possible to invoke the Get Garbage Position UPnP action that returns the position of garbage in the Garbage position variable. 3.5 Conclusion We have presented applications to integrate the service robots, Roomba and Rovio robots, into the smart home. The integration is based in UPnP software bridges that are executed in a computer. One of the program controls the Roomba via Bluetooth, and the other controls the robots using Wifi, and both of them offer UPnP interfaces so that UPnP control points can command the Roomba and Rovio robots. Both robots are deeply analyzed in this chapter as service robots and UPnP is analyzed as a middleware platform for the integration of service robots into the smart home. In addition, we have performed several experiments which have shown that the developed applications can be successfully integrated in a commercial UPnP home automation system. Moreover, we have shown that there are interesting services which arise from the full integration of robots in smart homes. For example, we have presented an experiment in which Rovio acts as a surveillance camera after a presence detector sensor warns the home automation system. Another interesting example is the application in which the Rovio’s bridge uses its garbage detection routine and communicates the control point the coordinates where garbage was detected. After that, the control point sends Roomba robot to clean the dirty area. These are good examples which highlight the benefits of interoperability in smart homes. Despite that total interoperability seems to be far away due to the large number 126 Integration of Robots in the Smart Home by means of UPnP of existing systems and the lack of a common standard, the potential benefits of the integration make the necessary efforts worthwhile. The application presented in this chapter is a contribution to go towards this objective. The goal of this research is to provide a framework for the integration of service robots in the smart home. Although the applications has proven to be useful in a real home automation environment, there are some aspects that have to be improved and new lines of research that have to be explored. The new services that emerge from the integration of robots and smart homes are promising. For example, in this application, we have made possible that the house control system can send the Roomba to any room. Consequently, it is possible to schedule the cleaning of the rooms or, imagining a more technological and integrated scenario, if a camera at home would detect dirty spot the control system could send the Roomba to clean it. Nevertheless, we have to remark that the home automation market is not mature enough to incorporate these ideas now, at least not at a mass market level, but we must not forget that in the long run ambient intelligence is the paradigm to follow, and it is clear that some of the steps needed to go towards this objective depend on the development of applications like the one that we have presented in this chapter. Chapter 4 Collaborative Tasks between Robots based on the Digital Home Compliant Protocol over UPnP The lack of interoperability is one of most important problems in the smart home. As a consequence, the integration of heterogeneous devices into the same system is a very difficult task which usually requires ad hoc solutions. For this reason, different interoperability standards have been proposed during the last two decades. The Digital Home Compliant (DHC) protocol [100] is an attempt to provide interoperability between domotic and robotic devices. This protocol is developed and supported by a consortium comprised of different companies and public institutions. DHC is an UPnP-based protocol aimed at making consumer electronic devices and robots work in a collaborative and effective way, and also to improve their energetic efficiency. Given the novelty of the protocol, in this chapter we provide a brief description of its architecture. Nevertheless, we will focus on one particular component, the DHC-Groups module, which is the DHC core since it allows a group of robots to work together and provides the necessary mechanisms to perform collaborative tasks. Given that this protocol is based on the UPnP architecture, robots working in accordance with UPnP technology are likely to meet the DHC requirements. UPnP has important features which make it a good starting point for developing a new home automation system for device interoperability. In addition, several studies have been developed for the implementation of UPnP 127 134 DHC Protocol of CollaborationKeyword associated with the robot. There are as many CollaborationKeyword as tasks that can be performed. A robot will have a CollaborationKeyword associated with each task it is capable of executing. For example, spot and clean are CollaborationKeyword associated with Roomba or any robot with similar features. 7. If the robot has the corresponding CollaborationKeyword, it will propose itself to perform the task and to lead a working hive. 8. The DHC-Groups module receives all the proposals and calculates the working hive and the leader. Then the state variable Leader has a new value, and an event is generated. In addition, the robots that will collaborate on the task are notified by several event messages. Finally, the DHC-Groups module subscribes to the leader’s EndTask event state variable. 9. When the adapter detects the new value of the variable Leader, it determines whether the leader ID is equal to the robot ID. When the IDs match, the adapter must send the order to the robot to start the task. 4.3 The Roomba vacuum cleaner: from Roomba UPnP to Roomba DHC Roomba is a floor-cleaning, disc-shaped robot developed by iRobot. As it was designed with the retail home market in mind, Roomba’s hardware is composed of standard hardware components. In this work, we have developed an application that offers Roomba’s services and state according to the DHC protocol. This application has been built from the adapter previously developed in [44], the Roomba UPnP virtual device. This adapter acts as a virtual bridge between the UPnP network and Roomba. The software bridge provides services to the network that will be performed by Roomba. To this end, the virtual adapter receives invocations from control points and it translates them into orders recognized by Roomba via Bluetooth. Once the robot interprets the orders, it executes the corresponding actions. DHC Roomba adapters are subscribed to all DHC-Groups variables at startup. When the user sends one task through the GUI, the working hive (size of the robot team) and the kind of task to be performed must be specified. Then, this task is received by the DHC-Groups module and it collects all information about the task using invocations and UPnP event notifications 4.4. EXPERIMENTS 135 messages. The values of the DHC-Groups variables such as TaskID,CollaborationKeyword or WorkingHive change and the DHC Roomba adapters are in turn informed of these changes through event notifications. In this way, the DHC adapters have all the necessary information to perform the task. The adapters receive a notification message with the ID of the selected devices by DHC-Groups in order to perform the task, and they check if their own ID appears in the ID list or not. If it does, the adapter configures the corresponding profiles to perform the task. In addition, they receive another notification in which the task to be performed is indicated (the CollaborationKeyword value) and the adapter invokes an action to be executed by the robot, working in similar fashion to the UPnP adapter. As it can be seen, the DHC adapter operation is based on the exchange of UPnP messages. When DHC-Groups updates its variables, the adapter saves the information, invokes actions to get the information required to perform the task, configures its profiles, takes any necessary action based on information collected and performs the task properly. 4.4 Experiments In this section we describe briefly how the implemented DHC-Groups module was developed. In addition, experimental results are presented. 4.4.1 Implementation We used Visual C# 2010 Express [161] integrated development environment to develop the adapter. Each of the three layers of Figure 4.2 has been implemented as a DHC entity relying on UPnP technology. Thus, we have a GUI device, a DHC device (which includes the DHC-Groups service) and a Roomba adapter device. To create and develop these devices, we used the Developer Tools for UPnP Technology [160]. This is a set of tools which are used in the design, creation and management of devices with UPnP features. Two of them, the Service Author and the Device Builder are used in the DHC project. Since DHC works over UPnP, DHC entities are essentially UPnP devices with some advanced capabilities. These capabilities have to do with the mechanisms that allow several entities to work in a collaborative manner. In addition, in DHC devices a dual behaviour is introduced, which means they can act as UPnP device or they can have control point features according to the situation. The development of DHC entities is based on the steps given to implement UPnP devices: 136 DHC Protocol Figure 4.3: DH Compliant Network 4.4. EXPERIMENTS 137 1. In the first step the XML service description file is created. This file contains the actions included in each service provided by the device. The description file is generated and implemented with the Service Author tool. It allows the developer to define the variables and the actions of the service. Every device must have at least one service description file in order to provide some information about the actions that the device offers to the network. 2. Once all the services included in the device are defined by means of their description files, we used the Device Builder to define the device. This tool allows for specification of the device and properties of services and generates the associated UPnP stack in the .NET framework. 3. In the third step the code related to the service actions must be developed in C# in order to implement the real functionality of the device. The developer must decide the technical complexity of the algorithm to implement each action. A dual behaviour must be implemented in the DHC entity, meaning that it can work either as a UPnP control point invoking actions and subscribing to events or as a UPnP device offering services and executing actions. It can notify of events as well. 4. Finally, it is necessary to check that the designed DHC entity works properly. In case that it does not, it may be necessary to repeat and debug one of the previous steps. 4.4.2 Experimental results In this subsection we present results from a real experiment in order show the DHC-Groups module in action (see [100] for videos of the DHC). In this experiment, the user is provided with a touchscreen that presents the graphical user interface (GUI) with UPnP features. This is the top layer of the scheme illustrated in Figure 4.2. The user must specify the task that the robot team will accomplish. To do this, the user must complete a form where the task parameters are indicated. The most important parameters are the following: •A task identifier. •Identifiers for the floor and the room where the task will be performed. •A collaboration keyword which represents the task type to be accomplished. •The number of robots in the working hive. 138 DHC Protocol •The time when the task should begin. In this case of use we design a task for a team composed of two Roombas. They are required to perform a cleaning task which is specified in the collaboration keyword value. Once the form is filled, the user can launch the task. The information about the task is sent through the DHC network and collected by the DHC-Groups module, which is subscribed to the GUI variables. This module processes the information and establishes a communication by means of events and subscriptions with the DHC Roomba adapters. When these adapters receive the task, they send a request for the coordinates of the room where they are, and how to reach the point where they should go to accomplish the task. An external localization system is used to exchange this information with the robot team using the DHC protocol and to guide the hive to the desired coordinates. In particular, the localization system consists on a zenithal high definition camera and a PC with the OpenCV library that provides the coordinates and orientation of the robots (see [173, 181] for more information). Although it is beyond the scope of this work, the DHC Localization module works as any other DHC entity: it provides an UPnP service that can be invoked whenever a robot needs to know its current coordinates and orientation. This service is connected to the localization system and sends the position information back to the adapter as the return value of the invocation received. Finally, when the robot hive reaches its destination, they start the cleaning task. This process is shown in Figure 4.4 in which the following steps are distinguished: (1) The GUI allows the user to create a new task and send the task parameters to the DHC module. (2) The robot adapters are subscribed to the DHC-Groups module state variables. As the task received from the GUI changes the values of the evented DHC state variables, the DHC adapters are notified. (3) The adapters analyze the evented variables and check the kind of task to be performed. If the robot has enough capabilities to accomplish the task, the adapter will propose the robot’s candidancy to perform the task by invoking a DHC Groups module action. (4) Once the DHC-Groups module has all the candidancies, it will choose the robots that will accomplish the task and the leader of the alliance by updating its state variables. At the end of this step the DHC-Groups module subscribes to the leader’s EndTask state variable. 4.4. EXPERIMENTS 139 Figure 4.4: Roombas scenario. (5) Once the leader adapter has divided the cleaning place and has calculated the coordinates for each subtask, it invokes the CleanAtPoint action of every robot in the hive. This action performs a cleaning task at a certain coordinates. Then, the leader subscribes to the EndTask state variable of every robot. (6) The adapters involved in the task invoke the DHC-Groups module to obtain their robot’s current location. (7) The DHC-Groups module requests the robots’ position from a DHC localization system. In this particular case, a zenital camera has been used for locating the robots. (8) The DHC localization system responds with the current position of each robot. This information is forwarded to each adapter. (9) Each adapter guides its robot to the destination point provided with the CleanAtPoint action in order to start the subtask. This navigation is performed by the adapters, which can update the robot’s position by requesting it from the DHC-Localization module. (10) When the robots accomplish their subtasks, they change the value of their EndTask variables to send the corresponding event to the leader. 140 DHC Protocol Figure 4.5: A cooperative task over the DHC Compliant protocol. Once the leader accomplishes its own subtask and has received the EndTask event from every robot, it sends a similar event to the DHC- Groups module so that it knows that the task has concluded. In Figure 4.5 the behaviour of the DHC Groups entities over time at our scenario can be seen. The sequence of states is: (1) The GUI is launched; (2) the DHC-Groups module starts and it subscribes to the GUI state variables; (3) the DHC-Adapter1 starts and it subscribes to the DHC-Groups state variables; (4) the DHC-Adapter2 starts and it subscribes to the DHC-Groups state variables; (5) the task is launched from the GUI and it notifies by changing its state variable; (6) DHC-Groups notifies the task to the adapters by means of its state variable and it invokes a GUI action to ask for the task information; (7) the adapters present their candidacy by invoking the Groups action; (8) DHC-Groups notifies the leader (DHC-Adapter2 for our particular case) and the task participants (DHC-Adapter1) and it subscribes to the leader EndTask state variable; (9) the leader invokes the CleanAtPoint action of the robots in the hive and it subscribes to the corresponding EndTask state variable; (10) the DHC-Adapter1 and the leader invoke actions to get their coordinates from the DHC-Localization through the DHC-Groups; (11) the DHC-Adapter1 finishes it subtask and it notifies to the leader by changing its EndTask state variable; (12) the leader finishes its subtask and it notifies to the DHC-Groups by changing its EndTask state variable. Finally, we examine the communicational burden of DHC-Groups module, which is shown in Figure 4.6 for the aforementioned experiment. During the three minutes of the experiment, 169 messages were sent, with an average size of 322 bytes. The average bandwidth used by DHC was 305 bytes/s and the total amount of bytes sent in the scenario was 54395. Notice that the largest part of this amount was sent during the launch of each entity. It is 4.5. CONCLUSION 141 Figure 4.6: A cooperative task over the DHC Compliant protocol. important to remark that these amounts are low when compared with the bandwidth available on modern home networks, which shows that the DHC architecture is suitable for current smart homes. According to the results that have been obtained from our tests we can conclude that DHC-Groups module works properly and allows for the scheduling and execution of several tasks performed by a robot team. Additional performance measurements for the Rovio adapter can be found in [182]. 4.5 Conclusion In this chapter an implementation of the DHC protocol in the service robot Roomba is presented. One of the DHC modules (DHC-Groups) is deeply detailed and is implemented on this robot. DHC is based on UPnP technology and has a three-tier stack. Devices meeting DHC requirements are implemented in specific adapters that act as a bridge between DHC and the robot proprietary technology. In the case of Roomba, its adapter translates orders received via UPnP into orders that are recognized by the robot and are transmitted using Bluetooth technology. As it has been seen, DHC relies on UPnP in order to provide additional M2M mechanisms to make different types of devices work together (as seen in Figure 4.3). The standard is composed of different modules which implement key features, such as collaborative tasks, energy management, localization, 142 DHC Protocol security and privacy. The chapter deals with the operation of the DHC- Groups module, which manages the collaboration between different types of robots to perform tasks. The paticular adapter presented in this chapter, DHC Roomba, allows users to clean a house with a set of cooperating Roombas. As a consequence of this first project a new alliance emerged. DH Compliant 2 [183] continued the goal proposed by the previous consortium to develop a universal protocol for communicating service robots, smart appliances and digital homes using Web Services. Chapter 5 Cloud Robotics This chapter is an adapted copy of a paper sent to Sensors journal, which is an open access journal published by MDPI, with the title “Off the shelf cloud robotics for the smart home: empowering a wireless robot through cloud computing” [184] for the special issue Sensors for Home Automation and Security. Nowadays service robots are gradually being introduced at homes. For example, autonomous vacuum cleaners such as Roomba or Navibot have become common devices in many homes. Other popular robots are Rovio, which provides users with telepresence services, and the quadrotor ARDrone, which has become a successful domestic recreational device. All of them have contributed to provide a more realistic view of the current capabilities of service robots, still very far away from the expectations generated by science-fiction and futuristic movies. Besides the proliferation of service robots at homes, there are other remarkable household technologies that are being adopted to improve people’s lives and to optimize energy consumption. This is the case of smart home technologies, which provide users with services regarding automation, security, or multimedia management. Unfortunately, the lack of standardization of the smart home market is a major issue that hinders its development [185, 186]: there are many different technologies that offer smart home services (e.g., the old X-10, Lonworks, KNX, Z-wave,. . . ), but none of them has become a true reference for marketeers and developers. For this reason, different middleware platforms have been proposed to integrate heterogeneous technologies in the same network, e.g.: OSGi [33], UPnP [45, 37, 41, 187], or Jini [20], to name a few. In this chapter, we deal with the integration of service robots in the smart home. In the literature, different works that deal with this interesting 143