ARGO: Improvement of Processes for Decision-Making in Software Engineering Project Management TIC2001-1143-C03

Jornada de Seguimiento de Proyectos en Tecnolog´ıas Inform´aticas, 2003 Programa Nacional de Tecnolog´ıas de la Informaci´on y las Comunicaciones ARGO: Improvement of Processes for Decision-Making in Software Engineering Project Management TIC2001-1143-C03 Jos´e Javier Dolado ∗ Universidad del Pa´ıs Vasco - Euskal Herriko Unibertsitatea Jos´e Riquelme † Universidad de Sevilla Javier Tuya ‡ Universidad de Oviedo Abstract This coordinated project develops, studies and researches into several key activities for software engineering project management. The final goal is to provide the project manager with a set of guides and procedures for carrying out a scientific decision making in software engineering projects. In this project, we deal with a wide range of management activities from an integrative viewpoint, including both initial phases of the software life cycle (estimation, simulation and planning) and others implemented at the end of the life cycle (software testing, process assessment, post-mortem analysis, etc.). Besides that, and by using experimental design, we study several aspects of the UML modelling and the software maintenance phase. In the area of cost estimation, we intend to improve the estimation process by applying metaheuristics algorithms,simulation and data mining. The combination of these techniques tries to overcome the problem of the lack of data at the beginning of the project and to increase the reliability of the predictions. We validate different behaviours in a software development project with project databases (simulated, actual, commercial). By using data mining, we synthesize the management rules more useful for software project management. In the software testing phase we assess, by means of metaheuristics, the testing strategies that are economically efficient. The main issue addressed in software testing is test case generation, both for imperative programs and non imperative (SQL and databases). In the area of process assessment, we look at the effectivity of the last international proposals (ISO-SPICE) in software development firms. We use metaheuristics techniques and information visualization techniques as horizontal techniques, meaning that they will pervade the three subprojects. Since the process of decision making over all the life cycle requires of an information system that allows to collect and to use the data according to the needs, we will develop a system for data ∗Email: [email protected] †Email: [email protected] ‡Email: [email protected] TIC2001-1143-C03 collection and for decision-making support. This decision support system will be parameterised specifically for our integrated monitoring of software projects. Keywords: Software engineering, software project management, cost estimation, metrics, data mining, experimentation, project simulation, information visualization, software testing, process assessment, metaheuristics algorithms 1 Project Overview Our argo project is devoted to the study of the different techniques that can be used by humans to cope with the management decision problems in Software Engineering. We think that the management and decision processes can enhance (or may limit) the improvements obtained by the technical instruments available in software technology. In many cases the successful outcome of a project depends more on the manager than in the tools or techniques used in the project. That is why this project studies a wide range of quantitative techniques which are amenable to be used by the software project manager. Software Engineering will not be a true engineering discipline until the people who really develop the software uses the quantitative techniques for assessing, monitoring, verifying and controlling the products, resources, quality and processes. The joint interest of the three argo subgroups is to improve the methods for managing the quantitative information that software project managers use for getting the project done reliably, on time and within budget. Our endeavors deal with projects, i.e, how to achieve a goal in which people have to carry out tasks with a final purpose, and with changing situations during the process of development. We found very inspiring the greek myth of the Argonauts: a group of mythological heroes who were assembled to rescue the golden fleece, which was guarded by a dragon. The three subprojects were named according to the three demigods that commanded the argo vessel: Jason, Heracles, Idas. The project began on 28-Dec-2001 and will end on 27-Dec-2004. 1.1 Subproject 1: Jason. Implementation of Procedures for Estimation, Experimentation, Assessment and Visualization of Software Engineering Projects Among the topics addressed by this subproject are: •Experimentation in software engineering. In our field there are constant proposals of new languages and techniques and there are many occasions where the proposals are accepted without further examination. In other fields, such as medicine or civil engineering, new solutions are not adopted until they have been positively evaluated. Although the number of articles dealing with experimentation is growing, there is still the need for further experimentation in software processes and products [7]. •Software process assessment. The goal is to have a set of indexes that allow the organization to monitor the status of its software processes. Then, there is the need of having a model and a set of tools that can help to evaluate the quality of the software houses. We may use the Capability Maturity Model (CMM) and the standard SPICE (ISO/IEC TR-15504), Software Process Improvement and Capability dEtermination. The latter is used as the reference model for applying it in our research. TIC2001-1143-C03 •Cost estimation models. There is no doubt about the importance of estimating and controlling the costs in software development. Basically there are two approaches: parametric models in which the models are driven by data and non-algorithmic methods in which we use heuristics or metaheuristics methods for finding an optimal solution, which is not guaranteed to appear. Neural networks and genetic programming fall in this category. •Metaheuristics algorithms applied to software engineering. The term metaheuristic algorithm refers to a set of generic algorithms that can search for optimal solutions in wide multidimensional spaces. We apply them in different situations and purposes (e.g., in cost estimation). Many times, the problems that software engineers face only require a set of good solutions, not an optimal alternative. 1.2 Subproject 2: Heracles. Definition and Design of a System of Metrics for the Assessment and Estimation of Software Engineering Projects •Databases of software projects. Given the difficulty of obtaining large amounts of data from actual projects, we simulate dynamic models in order to have databases big enough for post processing. There are real cases in which the decision process is improved by using the data collected [6]. In any of the two situations (actual data or simulated data) the plain question is: Now that we have much data, what can we do with it? The procedures of data mining can deal with large amounts of data. •Data Mining. The concept of Data Mining is defined as the search for hidden relationships and global patterns that are embedded within large databases. These relationships, or rules, are an invaluable knowledge about the objects related to the data base. However, the main technical problem is that the search for the correct relationships is computationally very expensive. Therefore, we need machine learning and metaheuristics algorithms for adapting the best search strategy. •Software Process Improvement. Current software process models (CMM, SPICE, ...) strongly recommend the application of statistical control and measurement guides to define, implement and evaluate the effects of different process improvement. However, whilst quantitative modelling has been widely used in other fields, it has not been considered enough in the field of software process improvement. During the last decade software process simulation has been used to address a wide diversity of management problems. Some of these problems are related to strategic management, technology adoption, understanding, training and learning, and risk management, among others. We combine traditional estimation models with an intensive utilization of dynamic simulation models of the software process. TIC2001-1143-C03 UMLexperimentation Sideeffectsexperiments SideeffectsAnalysis 2002 2003 ProjectRepresentation JASON HERACLES IDAS UMLandOML 2002ARGOMEETING 2003ARGOMEETING PhD PhD PhD PhD PhD PhD ProcessAssessment ProcessAssessment PhD jspMCYT CostAnalysis EvolutionaryAlgorithms Decision-makingforSDPs Preprocessingdata:Editing Predictioninmaintainability Preprocessingdata:Featureselection FuzzyrulesforSDPs jspMCYT-JISBD jspMCYT-JISBD SoftwareTestingAutomation MetaheuristicsAlgorithms Tabusearchtestcasegeneration Databasetesting Instrumentation(Tabu) SQLcoverage Figure 1: Topics by subgroup. 1.3 Subproject 3: Idas. Process Improvement in the Monitoring, Verification and Validation of Software Engineering Projects •Software verification and validation (V&V). Among the activities of V&V, software reviews and testing are the main tasks performed. In case of reviews, and according to the [IEEE Std 1028-1997] there are management reviews, technical reviews, inspections, walkthroughs and auditing. Inspections are a widely acknowlegded useful technique that, unfortunately, suffers from different defects [5]. •Software testing with metaheuristic algorithms. Because software testing is an expensive process, there is an increasing interest in the use of different techniques and tools to automate this process. In this project we focus on the automation of test case generation using metaheuristic algorithms [1]. There are previous works in the use of different optimization algorithms and approaches such as mutation testing, but metaheuristic techniques are not much used (with the exception of genetic algorithms). •Software testing for database systems. Although there are some tools that are useful for performing regression testing of database intensive programs, the automated support for test case generation and test case reduction, when a test case is composed by a set TIC2001-1143-C03 of database entries and some input parameters, is a topic not well covered by current research. Our goal is to improve the automation for this testing process when programs are mainly written using SQL sentences and run against a database [2]. •Software verification and validation process. The goal is that the definition of processes and metrics for supporting the testing process be integrated with the use of automation tools. 2 Managing the ARGO Project 2.1 Coordinating the Jason-Heracles-Idas subprojects We have had a very fluent coordination among the projects at the subproject level, and also at the general level, with technical meetings which served as catalysts for new ideas. The core of the whole group had participated in previous projects (CICYT TIC 1179-E, CICYT TIC99-0351 and others), and this has made it easier to have a smooth collaboration in the tasks of organization and communication. We have had two annual meetings for the entire project, within a workshop format, in which the works in progress were presented to the whole group and debated. As result of that interaction new ideas have emerged and some studies have been refocused. The project has a web page with the information that may be relevant for public interest (http://www.sc.ehu.es/jiwdocoj/argo/argo.html). Also, thanks to the effort put in this project we have continued organizing the Workshop Apoyo a la Decisi´on en Ingenier´ıa del Software, ADIS (http://www.sc.ehu.es/jiwdocoj/remis/cfpadis2003.htm)–Decision Support in Software Engineering. Figure 1 shows the temporal evolution of the topics researched. The large black dots represent the doctoral thesis that were defended by members of the project. Subgroup Idas has planned several Ph.D. Dissertations for 2004. In July of every year we have the [2002- 2003-2004] argo meeting. Figure 2 shows the expected research to be carried out until the end of the project. 2.2 Topics developed in the Jason subproject The results of this subproject have been focused on •Experimentation in Software Engineering. Experiment about side effects. We applied a crossover design on different tests involving fragments of C code that included increment and decrement operators. Each test had a side effect version and a side-effect free counterpart. The variables measured in the treatments were the number of correct answers and the time spent in answering. The results show that the side effect operators considered significantly reduce performance in comprehension-related tasks, providing empirical justification for the belief that side effects are harmful. Experiments about the dynamic modeling in UML. The goal of the first empirical study was to compare the semantic comprehension of three different notations for TIC2001-1143-C03 JASON HERACLES IDAS 2003 2004 2004ARGOMEETING ProjectRepresentation SideeffectsAnalysis SoftwareCostManagementMethods ProjectVisualization ProcessAssessmentConclusions Experimentssummary ComplexityManagement DataStreams Textualclustering Automaticreductionofdatabases NewAEcoding Knowledgeextractionfromtexts ApplicationdevelopmentApplicationdevelopment Othermetaheuristics Toolintegration SQL/DatabaseTestautomation Experimentation,benchmarks SoftwareTestingProcesses PhD PhD PhD PhD PhD PhD Softwaremetricsandquality Decisionsupportfordynamictasks Softwareprojectsimulator Projectestimationandfuzzylogic Figure 2: The immediate future of the project. representing the dynamic behaviour in UML: (a) sequence diagrams, (b) collaboration diagrams, and (c) state diagrams. We performed a 3x3 factorial experimental design with repeated measures. The metrics collected were total time and total score. The main conclusion of this study is that the comprehension of the dynamic modelling in object oriented designs depends on the diagram type and on the complexity of the document. The software project design written in the UML notation is more comprehensible, when the dynamic behaviour is modelled in a sequence diagram. While if it is implemented using a collaboration diagram, the design turns out to be less comprehensible as the application domain, and consequently, the document is more complex. •Ontology development. This work described the initial stages of building an ontology of the activities of software (V&V) as key elements in the management of a software project. There is a need for information in the V&V phases, as many decisions have to be taken with information and data which varies from project to project. The current work has its roots in the issues related to the integration of information in software project management. Among the tasks, data and decisions that the project manager has to deal with, those related to V&V of the software system are critical for a successful outcome. The goal in this work is to model the data, concepts, terms and relations used in the phases, processes and activities of software V&V from the project manager point TIC2001-1143-C03 of view. To do so, we have looked into ontologies in order to model the aforementioned concepts and as a way to integrate the different sources of information. •Bayesian networks in software engineering. Bayesian networks are becoming increasingly popular within the software engineering research community as an effective method of analysing collected data. This work deals with the creation and the use of Bayesian networks and bayesian classifiers in project management. We illustrate this process with examples in the context of software estimation. We highlight some of the difficulties and challenges of using bayesian networks and bayesian classifiers. We discuss how the bayesian approach can be used as a viable technique in Software Engineering in general and for project management in particular, and we also examine the challenges and the open issues. •Software process assessment. The international standard ISO/IEC 15504 provides a guide for software process assessment. Although one of the requirements of this model is its relevancy to all sectors and size of organizations, the limited availability of resources in SMEs (Small and Medium Enterprises) makes its effective application difficult. In this line of research, a method to infer the capability of software lifecycle processes is established. This new method is based on ISO/IEC 15504 and is aimed to define the capability levels of the processes in software SMEs. Ratings are calculated by analysing the answers obtained from a questionnaire which also constitutes an assessment guide for the enterprises to improve their software processes. 2.3 Topics developed in the Heracles subproject •Evolutionary algorithms and data mining. The application of evolutionary algorithms (EA) as search heuristic for data mining techniques is an important topic in the Heracles subproject. EAs have been used to obtain hierarchical set of decision rules with equal accuracy and less number of rules than classical methods. In addition, the EAs have allowed us to discover association rules from numerical attributes and qualitative rules for the decision-making task. We have applied these data mining methods to databases from software development projects (SDP). The aim is to provide decision methods in order to help the project manager to take decisions at any time in the development, by having the adequate number of rules for taking actions. •Data preprocessing. Other research direction consists in improving the data preprocessing techniques. In this sense, we have designed a method of editing (selection of examples or search of representative patterns) with two interesting characteristics: important reduction of the number of instances and lower computational cost. We have also developed a feature selection method (extraction of the most relevant attributes from a database) without distance or statistical calculations. •Integrated Framework for Simulation-Based Software Process Improvement. We have developed an integrated framework for software process improvement according to the CMM. This framework combines traditional estimation static models with an intensive utilization of dynamic simulation models of the software process. The aim of this framework is to support a qualitative and quantitative assessment for software process TIC2001-1143-C03 improvement and decision making to achieve a higher software development process capability according to the CMM. The framework is double-integrated. First, it is based on the systematic integration of dynamic modules to build a dynamic model to model each maturity level proposed in the reference model. As a consequence, a hierarchical set of dynamic models is developed following the same hierarchy of levels suggested in CMM. Second, the dynamic models of the framework are integrated with the use of different static techniques commonly used in planning, control, and process evaluation. The ideas underlying this framework have been materialized in two final products: a hierarchy of dynamic models, and a prototype of a tool to software process improvement design. •Decision making in software development projects. The Simulators of Software Development Projects based on dynamic models have represented a significant advance in front of the traditional techniques of estimating. These simulators enable us to know the evolution of a project before, during and after its execution. But its use in the estimate of the project before beginning the execution has been slowed down by the great number of attributes of the project that it is necessary to know previously. We have presented the improvements that have been added to the simulator developed in our department to facilitate the use of them, and the developed tool, in which we have used data mining and fuzzy logic techniques with the databases generated by the simulator. In this last case, the project manager can know, depending on the decisions that he or she takes, the level of execution of the project objective. •Forecasting in software development projects. The application of certain data mining techniques to numeric databases of SDPs, allows us to obtain qualitative information about the project evolution. Many of these techniques are descriptive, like the clustering, that is why we do not have capacity of forecasting results (project’s variables) a priori, from a new data set (project’s attributes) of a SDP. To estimate these variables from a new set of values of the attributes, we are proposing some methods (linear regression, k-nearest neighbour, data mining, forecasting), so that the obtained results can be compared between them. The objective is to check if the project variables can be forecasted, from a new series of attributes values, without having to simulate the whole project, and with low error margins. 2.4 Topics developed in the Idas subproject •Metaheuristics with Tabu search. Among the different metaheuristic techniques, we have selected Tabu Search as the first technique to experiment with, because of its capabilities of adaptative memory and responsive exploration. The main effort has been the definition of adequate cost functions, search criteria, memory management (short term and long term tabu lists) and backtracking that would be appropiate to the structure of our problem. Tabu search is used to generate test cases for imperative programs using the criteria of branch coverage and has been the subject of an intensive experimentation with programs ranging from few decisions to more than thirty decisions. All results have been compared with random techniques and outperform it (measured in time and test cases generated). Also, Tabu Search reaches the 100% coverage faster than random search. Although it is TIC2001-1143-C03 difficult to compare to other techniques because there is not a standard set of programs for benchmarking, comparison with other published results that use genetic algoritms gives that tabu search is faster. •Tool development. An instrumentation tool has also been developed. This tool analyzes the program structure and produces the instrumented program, that is plugged into a module with common data structures and the modules including the test case generation algorithm. The resulting executable program is used for test case generation, allowing us to quickly perform the experimentations. •Testing SQL programs. In the area of testing support for SQL programs, the problem of test case generation is more difficult that when we use imperative programs, because of test cases are also composed by database entries. Because of this, the first efforts have been addressed to define some kind of criteria of test completeness. We have defined a coverage metric for database testing that is calculated by taking into account the structure of the SQL query sentence, the relational structure of the data model and the data instances contained in the database. An algorithm has been developed that computes the coverage and supplies advices for: –reducing the data instances (rows) in the tables used by the SQL program –completing the database with new instances in order to attain a higher coverage –detecting possible bugs that could be latent in the database structure. •Evaluation of the usefulness of the algorithm and the coverage metric. It has been performed using an actual database obtained from an industrial maintenance process (the main table is composed by more than a thousand rows), and the algorithm has proven useful for reducing the test database size for detecting additional test cases and for reviewing the data model. 3 Outcomes of the project and main results We have been very prolific in presenting the works at refereed international conferences, but we have taken care of other activites, such as the global results of Table 1 show. Besides the figures of Table 1, we can say that some of the articles have appeared in the key journals and conferences of our fields. The set of topics researched cover an ample set of quantitative techniques. 3.1 Some published results Here we list some of the works that represent the topics that we have researched with successful results. The list is not exhaustive (obviously) and only tries to show how the members of the project are progressing. Many publications are indexed in different databases, such as ISI, Compendex, etc. The list of journals of the published works include Artificial Intelligence Communications, Empirical Software Engineering, IEE Proceedings–Software Engineering, IEEE Trans. on Power Systems, IEEE Trans. on Software Engineering, IEEE Trans. on Systems, Man and Cybernetics (Part B), Information Sciences, Information and Software Technology,