Agile Development and Maintenance of Applications based on Web Services

Jornada de Seguimiento de Proyectos, 2005 Programa Nacional de Tecnolog´ıas Inform´aticas Agile Development and Maintenance of Applications based on Web Services (AgilWeb) TIC2003-02737-C02 Miguel Toro Bonilla ∗ University of Sevilla Macario Polo Usaola † Univ. of Castilla–La Mancha Abstract The incredible successfulness of the Internet has paved the way for an industry devoted to developing and running web services. Our main goal in this research project is to provide a framework for the agile development and maintenance of applications that build on them. In this report, we provide an insight into its current status. Keywords: methodologies, web services, automatic provisioning. 1 Introduction The term web service is fairly self-explanatory since it refers to having access to services on the Web. However, there is more than this behind the scenes since the common use of the term also refers to the SOA reference architecture, standard protocols such as HTTP or SOAP, repositories such as UDDI, and data languages based on XML. The promise is that they will help simplify enterprise application integration and will bring new opportunities to outsource services on the Web. According to the Aberdeen Group, the percentage of companies opting for outsourced web services grew to more than 60% in 2004, and it is likely to increase as e-commerce becomes more mainstream and a must for almost every business. As of the time of submitting AgilWeb, however, the process of building and maintaining a web application based on outsourced web services was not a small undertaking since it was a relatively new technology. This motivated us to work on an appropriate framework both from a conceptual and a practical point of view, and to provide software engineers with appropriate tools and infrastructure. In this report, we present an insight into the current development of the project. In Section 2, we summarise our goals and the tasks involved; in Section 3, we report on its current status and our main achievements; in Section 4, we provide some hints on the relevance of our contributions; finally, our main conclusions and the work that remains to be done are summarised in Section 5. ∗Email: [email protected] †Email: [email protected] TIC2003-02737-C02 2 Goals of the project The main goal of AgilWeb was to devise a framework for the development and maintenance of applications based on web services. Their main feature is that time to market is usually very short, changes to the initial requirements are very frequent, and the risk of failure is high. The framework should then build on agile practices so as to keep changes under control, manage knowledge and expertise gained from previous projects to reduce development and maintenance costs, assess risks to help managers decide whether to embark or not on a new project, and provide an appropriate infrastructure for provisioning web services. In summary, the framework relies on the following pillars and the schedule of tasks that is shown in Figure 2: Methodology: Since we thought that the research on processes was rather mature and had reached the industry, we did not want to devise another process, but new guidelines and techniques in the context of web services. The idea was then to devise kind of methodological fragments ready to be integrated in well-established software processes. In summary, the main working areas and goals were the following: 1. Guidelines and notation: devise guidelines for requirements elicitation, analysis, design (USE-3.1), re-engineering (UCLM-2), and refactoring (UCLM-4); design a quality-driven reference architecture (USE-3.2); design an extension to UML to annotate software models with quality constraints (USE-3.3). 2. Knowledge management: turning intuitive knowledge about the applications to be built into formalised software product lines (USE-3.4); developing an explicit system to manage knowledge and expertise (UCLM-6). 3. Risk management: adapt a number of existing techniques to identify risks in our research context and react to them (UCLM-7). Infrastructure: A methodological fragment is quite useless unless one can count on a reference infrastructure to support it. Summing up, the main goals were the following: 1. Provisioning system: design a trading system to outsource web services using quality criteria (USE-2.1, UCLM-5); make it interoperable by means of well-established industrial standards (USE-2.2); design automated testing techniques so that a trader can check automatically whether a web service can be trusted from a functional point of view (UCLM-3). 2. Advanced interaction models: refine previous results on multiparty interaction models to adapt them to a setting based on web services (USE-3.5). 3. Repositories: study current repositories of web services (USE-4.1); devise techniques to integrate human-friendly repositories (USE-4.2). Tools: Last, but not least, engineers must have a set of tools to apply the guidelines, to reuse knowledge or to design complex multiparty interactions on the Web. The goal was to implement them during the third year (USE-5, UCLM-5, UCLM-8, UCLM-9). TIC2003-02737-C02 ID Task Name 2005 20072004 Q2Q4Q3 Q4Q3 Q3 Q1 Q3 1 USE-3.1 2 UCLM-2 3 UCLM-4 4 USE-3.2 5 USE.3-3 6 USE-3.4 7 UCLM-6 8 UCLM-7 9 USE-2.1 10 UCLM-5 11 USE-2.2 12 UCLM-3 13 USE-3.5 14 USE-4.1 15 USE-4.2 16 USE-5 17 UCLM-5 18 UCLM-8 19 UCLM-9 2003 Q2Q4Q1 Q1 2006 Q1 Q2 Q4 Scheduled duration % completion Figure 1: Chronogram of activities. 3 Current development 3.1 Difficulties and deviations AgilWeb was awarded with budget to hire four technicians, but we soon soon realised that it was difficult to find qualified people. In addition, there were unexpected delays in the release of the final draft of WS-Agreement, WS-Resources, and WS-Negotiation. As a result, we began working on tasks USE-2.1 and USE-2.2 late. There was also a significant budget cut regarding the group of UCLM, and this has had an important impact on the development of task UCLM-7. Currently, we are in arrears with them. The project was assigned an FPI student. Unfortunately, he decided to move to the industry. This was not problematic since one of the collaborators from the industry (David Benavides) decided to work on the project full time, and a new junior researcher was invited to join it (Pablo Trinidad). The tasks in which the FPI student was involved have progressed at an adequate pace since them. We have added a new task on studying ontologies since we found that current repositories of web services have sprouted chaotically. It is not surprising then that the terminology they use is inconsistent and makes it difficult to integrate them. This is why we decided to work on the design of a consensus ontology and divided the team in charge of task USE-4.1 into two groups: Jos´e M. Ca˜nete and Francisco Gal´an to work on ontologies, and Jos´e L. Arjona to study current repositories and techniques to extract knowledge from them (USE-4.2). There has been a little deviation regarding task USE-3.1. As of the time of submitting AgilWeb, it was clear that the role of use cases would gain importance, but we thought this was a well-established field. Soon after, we realised that there was a significant lack of techniques to verify use cases, and we then decided to invest a little effort on researching this topic. This deviation led to the development of a new PhD thesis by Beatriz Bern´ardez. Notice that not all of the deviations are negative. Tasks USE-3.5, USE-4.2 finished before it was expected, and the work on tasks UCLM-5, UCLM-8 and UCLM-9 is currently in progress. TIC2003-02737-C02 This is the reason why we think we will be able to compensate for the negative deviations in tasks USE-2.1 and USE-2.2. 3.2 Guidelines and notations USE-3.1: Here, our work focuses on multi-agent systems, since most web services back ends have an implicit degree of autonomy, proactivity and reactivity that allows to view them as agents. The goal was to define a methodological fragment to model, analyse, and design such systems, and the idea was to build on the results on requirements elicitation we developed in previous projects, namely GEOZOCO (TIC2000-1106-C02), and MENHIR (TIC97-0593-C05). We have also developed a number of metric-based heuristics to verify use cases. UCLM-2: We have devised a technique that allows to re-engineer a legacy system so that it can be transformed semi-automatically into a set of web services ready to outsource. Most of the results build on the work that was done in projects MANTIS (TIC1FD97-1608) and MEDEO (TIC2000-1673-C06). UCLM-4: We have developed a number of guidelines to refactor the micro-architecture of an object-oriented system and adapted them web services. USE-3.2: The core service of our architecture is the quality trader, which offers the basic services for the procurement of web services, i.e., consistence, conformance and optimality checks. Key features of the proposal are that the language it uses is symmetric, it allows for two-way matching, temporal constraints, uncontrollable and controllable quality parameters. We have studied the limits for it to be efficient, and we are currently working on using OWL to define an ontology of quality concepts. USE-3.3: We have studied how to use QRL to annotate the UML models with which we describe our reference architecture. 3.3 Knowledge management USE-3.4: After a thoughtful study of the state of the art on feature models, we concluded that their most important drawback was the lack of automated support tools. Therefore, we started working on this, and the result is a prototype that is available at ¡http://www. tdg-seville.info/topics/spl. UCLM-6: We have developed a number of mechanisms to describe and organise both the sources and kinds of knowledge involved in the development of a typical software project. The validation was performed in the context of a software maintenance process using SPEM. As a result, we have defined an ontology and an agent-based support system. Furthermore, we have designed a system that uses CBR techniques to reuse maintenance knowledge. TIC2003-02737-C02 3.4 Risk management UCLM-7: We have worked on risk management in the context of agile methodologies, but we are in arrears with this task. So far, we have prepared a research report with our main conclusions. 3.5 Provisioning system USE-2.1: We are currently working on the design of a trading system to outsource web services. So far, we have identified the roles involved, the protocols to connect them, and a general purpose negotiation framework. We still need to refine the protocols and to study their properties regarding correctness, completeness and complexity. UCLM-5: We have implemented a tool that helps determine whether two web services are interchangable, despite their interfaces being different. It requires metadata to describe an interface, the constraints on each method, and the protocol that governs the order in which the methods can be invoked; building on this information, it is able to produce a similarity metric to decide whether two services are interchangeable. Currently, we are working on adjusting the similarity criteria and thresholds in practice. USE-2.2: The trading system is based on current state-of-the-art industrial standards such as WS-Agreement, WS-Resource, WS-Negotiation, and UDDI. The work we have developed is fully compatible with these standards, which ensures that it will be easily interoperable. UCLM-3: We have proposed a new technique to automate the testing of black-box components, which is the key to testing web services. We have also implemented an accompanying tool that builds on a metamodel and uses introspection to generate and execute test cases. 3.6 Interaction models USE-3.5: In project GEOZOCO (TIC2000-1106-C02-01), we developed and implemented a synchronous multiparty interaction model that was suitable for open systems. Our improvements are twofold: first, we have worked on a new fairness notion that solves two well-known problems of previous approaches since it deals with both fair finiteness and conspiracies; second, we have worked on an asynchronous extension and we have developed a number of techniques to refine and/or abstract interactions and to compose and/or decompose interaction models. 3.7 Repositories USE-4.1: We have studied current web services repositories, and the conclusion is that most of them are user friendly and lack the ability to search for services using quality criteria. Our conclusions helped us define our trading system and design a knowledge extraction system able to deal with a user-friendly repository programmatically. TIC2003-02737-C02 USE-4.2: We have designed and implemented a system to extract knowledge from the Web that has proved useful to have access to user-friendly repositories programmatically. It has been validated with three popular web sites. 4 Relevance of our results 4.1 Preliminaries Before the project was funded, we started working on a number of activities whose goal was to have a good understanding of the technologies involved in the construction of a typical business solution. We got in contact with Dr. Wrembel1and embarked on two projects whose goal was to compile a series of articles on innovative solutions to common development problems, with an emphasis on web services. The result was an special issue of Inderscience’s Journal of Web Engineering and Technology [30] and a book that was published by Nova Science [29]. A little later, we embarked on editing a special issue of Springer’s Journal of Universal Computer Science on challenges and breakthroughs on software engineering [28]. As a result, we learnt what the main problems in this area are, and we established a network of international contacts that has proved fundamental to the development of AgilWeb. 4.2 Guidelines and notations USE-3.1: Regarding modelling and analysis, our publications are [53, 54, 55, 56, 57, 58]. They were cited in [33] and have been classified as recommended bibliography by the Modelling and Methodology Technical Committees of FIPA. We have collaborated with the research groups by Dr. Odell2, Dr. Cossentino3, Dr. Levy4, and Dr. Hinchey5. These collaborations led to the publication of [56], a collaboration agreement between NASA and the University of Seville, and a three-month research visit to the NASA Goddard Software Engineering Lab and to Intelligent Automation Inc. to keep working on our methodological fragment and to explore its applications to the industry. Most of these results are part of Joaqu´ın Pe˜na’s PhD thesis [59], which was defended in July 2005. Regarding the verification of use cases, the publications are [14, 15, 16, 17, 18, 19, 20, 21, 32], and they constitute the core of Beatriz Bern´ardez’s PhD thesis [22], which was defended in December 2004 and is the result of a close collaboration between both groups involved in AgilWeb since it was co-advised Amador Dur´an and Marcela Genero. Given the good results obtained, we are now active members of the National Research Network on Empirical Software Engineering (TIC2002–11494–E), which is led by Dr. Juristo6. UCLM-2: The publications related to this task are [40, 41, 42, 43, 44, 45, 46, 64]. We have contacted Mr. Garc´ıa7. and work in closed co-operation with him regarding this task, 1Dr. Robert Wrembel is a Professor of Database Systems with the University of Pozna´n. 2Dr. James Odell is the chair of the OMG Analysis and Design Task Force. 3Dr. Massimo Cossentino is the chair of the Methodology Technical Committee of FIPA. 4Dr. Renato Levy is the research head of Intelligent Automation Inc. 5Dr. Michael Hinchey is the head of the Software Engineering Laboratory of the NASA 6Dr. Natalia Juristo is a Professor of Software Engineering at the Universidad Polit´ecnica de Madrid. 7M.Sc. Ignacio Garc´ıa is a lecturer with the University Rey Juan Carlos. TIC2003-02737-C02 UCLM-3, and UCLM-5. We have also established a co-operation agreement with Soluziona Software Factory, S.A. to keep working in this task. Many of the results we have produced were developed in the context of F´elix ´ O. Garc´ıa and Javier Garz´as’s PhD’s thesis. UCLM-4: The main results have been produced in the context of Javier Garz´as’s PhD thesis. USE-3.2: The publications related to this task are [49, 50, 51], and they have got citations from [23, 74]. Furthermore, [52] was presented in a conference with an acceptance ratio of 21% and was selected for publication in a major journal [73]. Recently, we have collaborated with Dr. Tosic8whose research is very related to ours. We have developed a prototype that is available at http://www.tdg-seville.info/topics/procurement.html. Most of these publications constitute the core of future Octavio Mart´ın-D´ıaz’s PhD thesis, which is expected to be defended during the first semester of 2006. USE-3.3: See USE-3.2. Both tasks are intimately related. 4.3 Knowledge management USE-3.4: Our main publications are [7, 8, 12, 13, 75, 31]. We would like to mention that [12] was presented in a conference whose acceptance ratio was 14%, and that it was referenced by Dr. Batory9in [5, 6]. We have also established a network of contacts with whom we are working on this task, namely: Dr. Batory, Dr. Smith10, Dr. Montes de Oca11, and Dr. D´ıaz12. Some of the papers mentioned above were written in co-operation with them, and David Benavides is currently in the course of a three-month research visit to Dr. Smith’s Laboratory. UCLM-6: The publications related to this task are [25, 26, 27, 36, 47, 65, 66, 67, 68, 69, 70, 71, 76, 77]. We have established a collaboration agreement with Dr. Canfora13, and we are working with him in the development of this task. F´elix ´ O. Garc´ıa has recently paid a six-month research visit to his department, and he is currently on a new four-month visit. Furthermore, we have established a network of contacts with whom we are working within the context of Experimental Software Engineering Network (IST-2001-37482) and a Spanish network called “M´etodos de Investigaci´on y Fundamentos Filos´oficos en Ingenier´ıa del Software y Sistemas de Informaci´on” (TIC2002-12378-E). Many of the results we have produced were developed in the context of F´elix ´ O. Garc´ıa PhD’s thesis. 4.4 Risk management UCLM-7: So far, we have just produced a research report [62]. 8Dr. Vladimir Tosic is with the Department of Computer Science of the University of Western Ontario. 9Dr. Don Batory leads the group on Software Product Lines at the University of Texas at Austin. 10Dr. Barbara Smith is with the Cork Constraint Computation Centre, Ireland. 11Dr. Carlos Montes de Oca is a senior researcher with the CIMAT, M´exico. 12Dr. ´ Oscar D´ıaz is a Professor of Software Engineering with the University of the Basque Country 13Dr. Gerardo Canfora is a Professor with the Universit`a degli Studi del Sannio. TIC2003-02737-C02 4.5 Provisioning system USE-2.1 and USE-2.2: The publications related to this task are [34, 63, 35]. We have established relationships with Dr. Ludwig14, Dr. Zhang15, and Dr. Alonso16. These collaboration has led to a three-month research visit to the IBM T.J. Watson Research Center that will be held during the forth quarter of 2005. Furthermore, we got additional budget to pay a six-month research visit to Dr. Tavangarian’s institute17. There, we collaborated on setting up a testing bed for our trader in the context of web services for mobile e-learning services. UCLM-5: We are working in co-operation with Dr. Augusto18. Andr´es Flores will soon pay a research visit to his department to keep working on this task. UCLM-3: The publications related to this task are [37, 38, 39, 48, 61] and the results are part of Javier Garz´as’s PhD thesis. Currently, we are members of the Experimental Software Engineering Network (IST-2001-37482) and are working in co-operation with other members in the development of this task. Our co-operation with Soluziona Software Factory, S.A. is also closely related to this task. 4.6 Interaction models USE-3.5: Our main publication is [60]. We have established relationships with Dr. Fiadeiro19, and the results were published in [28]. Most of the work on fairness was at the core of David Ruiz’s PhD thesis [72]. The results on our new version of the multiparty interaction model and the associated techniques for refinement/abstraction or composition/decomposition were developed in the context of Joaqu´ın Pe˜na’s PhD thesis [54, 55, 56, 59]. 4.7 Repositories USE-4.1: Recall that this task was divided into two new tasks regarding the study of ontologies and the repositories. As for the former, our only publication is [24]. Keep reading for additional details on the latter. USE-4.2: The publications regarding this task are [1, 2, 3]. We have established relationships with Dr. Sykara20, and we got additional budget for Jos´e L. Arjona to pay a six-month research visit to her laboratory. Unfortunately, the visit was not carried out due to personal problems. However, the collaboration keeps alive and has led to Jos´e L. Arjona’s PhD thesis [4], which was defended in March 2004. 14Dr. Heiko Ludwig co-leads the work on WS-Agreement at the IBM T.J. Watson Research Center. 15Dr. Lian J. Zhang leads the group on outsourcing web services at the IBM T.J. Watson Research Center. 16Dr. Gustavo Alonso leads the research group on web services architectures at the ETH Z¨urich. 17Dr. Djamshid Tavangarian leads a research group on mobile devices of the University of Rostock. 18Dr. Juan C. Augusto is a Lecturer with the University of Ulster at Jordanstown. 19Dr. Jos´e L. Fiadeiro is a Professor of Software Science and Engineering with the University of Leicester. 20Dr. Katia Sykara is a Professor of Intelligent Systems with the Carnegie Mellon University TIC2003-02737-C02 5 Conclusions and further work Although we had problems with our initial schedule, we have been able to overcome them and currently almost all of the tasks that were planned are progressing adequately. Only a couple of tasks are a little delayed due to the problems we had to hire qualified people and the delay to publish the final release of the standard on which we rely. We do not think this will be a serious problem since we plan on finishing them in a few months. We have produced a number of publications in major journals and conferences, and 11 PhD thesis are currently under development and will conclude before or little after the project is finished, namely: Ignacio Garc´ıa (cf. UCLM-2, UCLM-3, UCLM-4, UCLM-5), Gabriela Aranda (cf. UCLM-6), Andr´es Flores (cf. UCLM-2, UCLM-3, UCLM-4, UCLM-5), ´ Oscar Rodr´ıguez (cf. UCLM-6), Juan P. Soto (cf. UCLM-6), Manuel Mart´ınez (cf. UCLM-6), Mateus Ferreira (cf. UCLM-6), Pablo Fern´andez (cf. USE-2.1, USE-2.2), Manuel Resinas (cf. USE-2.1, USE2.2), Octavio Mart´ın (cf. USE-2.1, USE-3.2, USE-3.3), and David Benavides (cf. USE-3.4). As a conclusion, the results we planned on producing are almost ready and we can embark on the final stage, that consists mostly of implementing the tools we have designed. References [1] J.L. Arjona, R. Corchuelo, J. Pe˜na, D. Ruiz. First International Atlantic Web Intelligence Conference AWIC 2003: Coping with Web Knowledge. Springer-Verlag LNAI 2663. Madrid, Spain. pp. 165-178. 2003 [2] J.L. Arjona, R. Corchuelo, D. Ru´ız, J. Pe˜na, M. Toro. 15th Conference on Advanced Information Systems Engineering CAISE’03: Knowledge Channels. Bringing the Knowledge in the Web to Software Agents. 2003 [3] J.L. Arjona, R. Corchuelo, M. Toro. International Conference on Web Intelligence IEEE/WIC 2003: A Knowledge Extraction Process Specification for Today’s non-semantic web. pp. 61-68. 2003 [4] J.L. Arjona. Bringing the Knowledge on the Web to Software Agents. University of Sevilla. 2005 [5] D. Batory. Feature models, grammars, and propositional formulas. To be published at Sofware Product Line Conference (SPLC 2005), 2005 [6] D. Batory. A tutorial on feature oriented programming, the ahead tool suite. In Summer school on Generative, Transformation Techniques in Software Engineering, 2005 [7] D. Benavides, A. Ruiz-Cort´es, R. Corchuelo, A. Dur´an. Seeking for extra-functional variability. In Proceedings of the ECOOP Workshop on Modeling Variability for Object-Oriented Product Lines, 2003 [8] D. Benavides, A. Ruiz-Cort´es, R. Corchuelo, O. Mart´ın-D´ıaz. SPL needs an automatic holistic model for software reasoning with feature models. In International Workshop on Requirements Reuse in System Family Engineering, pages 27–33, Madrid, Spain, 2004. Universidad Polit´ecnica de Madrid [9] D. Benavides, A. Ruiz-Cort´es, , M.A. Serrano, C. Montes de Oca. A first approach to build product lines of multi-organizational web based systems (mows). LNCS, Innovative Internet Community Systems, IICS 2004 3473:91–98, 2005 [10] D. Benavides, A. Ruiz-Cort´es. A firts approach to build product lines of mows. In Proceedings of the ZOCO02 workshop, El Escorial, Madrid, 2002 [11] D. Benavides, A. Ruiz-Cort’s, P. Trinidad. Coping with automatic reasoning on software product lines. In Proceedings of the 2nd Groningen Workshop on Software Variability Management, 2004 [12] D. Benavides, A. Ruiz-Cort´es, P. Trinidad. Automated reasoning on feature models. LNCS, Advanced Information Systems Engineering: 17th CAiSE, LNCS(3520):491–503, 2005 [13] D. Benavides, A. Ruiz-Cort´es, P. Trinidad. Using constraint programming to reason on feature models. In The Seventeenth International Conference on Software Engineering, Knowledge Engineering (SEKE’05), July 2005