Systematic Mapping Protocol for Process Development in System-of-Systems

205 APPENDIX D PDSoS - Systemic Mapping Protocols Initial Remarks Our mapping was conducted in the context of software development processes for SoS. This mapping was carried out from April/2015 to October/2015 and updated from September/2019 to December/2018 and involved six researchers, among them four experts. We used the guidelines established by Kitchenham and Charters (2007) that is divided in three phases: planning, conducting, and reporting In the first phase, the research objectives and the systematic review protocol are defined. In the second phase, the primary studies are identifying, selecting, and evaluating second to the inclusion and exclusion criteria defined in the planning phase. For each selected studies, data are extracted and synthesized. In the third phase, a final report is organized, presented and discussed. The next sections are presented present these three phases in details. Phase I - Planning In this phase, the SM protocol was established, including the research questions, search strategy, selection criteria, quality assessment criteria and data extraction and synthesis method. The SM protocol was elaborated by the author and validated by the experts. Research Question In order to structure and simplify the research questions development, we used the PICO model (KITCHENHAM; CHARTERS,2007), which includes the consideration of four elements: P opulation, I ntervention, C omparison and O utcomes. In the Population is described the population group that is observed by the intervention; In the Intervention identifies what will be investigated this SM; Comparison is identified what will be used to compare in the context of the SM; and Outcomes are the expected effects of the proposed intervention. To this SM the 206 APPENDIX D. PDSoS - Systemic Mapping Protocols Population comprises the developers and researchers of SoS and systems engineering community. Intervention for our SM is related to the context of SoS development processes. Comparison it not applicable. Lastly, in Outcomes characterize all primary studies identified in the literature that report SoS development processes. Furthermore, we hope to identify SoS characteristics, type of empirical evaluation, technical factors, activities and/or artifacts addressed or mentioned by the processes, factors that can influence the development of the SoS, and trends and challenges discussed by the studies. The research questions are: RQ-01 Have SoS been constructed based on development processes? Which are those processes? This RQ aims at identifying processes, but not excluding other initiatives (e.g., methods, frameworks, and models) proposed for building SoS. RQ-02 Which SoS characteristics have been considered in SoS development processes? We aim at investigate the main characteristics mentioned by the authors based on the five Maier’s characteristics (MAIER,1998). RQ-03 Which technical factors and management activities were addressed or mentioned in the SoS development processes? This RQ, the objective is to identify the technical factors (e.g., communication, cost dependence, risk management) that were used or influenced to build of the SoS development processes. RQ-04 Which main challenges and trends have been discussed by the authors of the primary studies? This RQ aims at identifying the main challenges and trends of SoS development processes discussed by the authors of the primary studies. RQ-05 How have been established the SoS development processes? The objective is to identify approaches used to building SoS development processes and to verify which technologies help in the development those processes (e.g, SOA, MDA, MDD). RQ-06 What do have influenced the development of the SoS? This RQ aims at identify the factors that influence (e.g., constituents architecture, several life cycle) the interoperability in SoS development processes. Search Strategy In order to determine the search strategy of selection of the primary studies we choose four search sources: publication database, related works, snowballing and expert opinion. The four search sources are presented as follows: 207 ∙Publication database: For identifying primary studies is necessary to define which publication database will be adopted and the search string that will be used on these publication databases. At these SM, we adopted the publication database ACM Digital Library, IEEE Xplore, Science Direct, Scopus, and ISI Web of Science. They were chosen to be recognized and widely used in Software Engineering, as example: (PERTENSEN et al.,2015) (KITCHENHAM et al.,2010) (MADEYSKI et al.,2014) (MUNIR et al.,2014) (SILVA et al.,2015c) (MACHADO et al.,2013) (NARCISO et al.,2014) (GUESSI et al.,2015). To define the search string, five steps were taken: (i) search by synonyms of “development process” in systematic literature reviews of traditional software development processes; (ii) search in dictionaries; (iii) consulting experts; (iv) pilot search; and (v) use of control studies (CALINESCU; KWIATKOWSKA,2008a) (CARBON et al.,2008a) (FARROHA; FARROHA,2011a) (RAMOS et al.,2012a). The pilot research showed that the term “development technique” and the plural of the synonyms of “development process” present in the search string not influenced the total number of primary studies returned by the engine search. Furthermore, it was noted that 50% of control studies were not returning at these search, which could be a failure of the search string. An analysis and modifications were done together with experts achieving a higher return than 231% 1 of primary studies when the search string returned all control studies. The search string was designed to cover variations and synonyms for terms related to “development process” and “systems-of-systems” and adapted for meeting particularities of each engine search. The search string used in our SM is presented in Table 23. The first column represents the research areas and the second column the corresponding terms. Table 23 – Search string Area Terms Development Process (“Development Method” OR “Development Approach” OR “Development Design” OR “Design Flow” OR “Development Procedure” OR “Development Life Cycle” OR “Agile Development” OR “Software Product Line”) AND Systems-of-Systems (“System of Systems” OR “System-of-Systems” OR “Systems-of-Systems” OR “Systems of Systems”) ∙related works: 1 the search string of the pilot research returned 183 primary studies, after the modifications the search string returned the total of 423 ones. 208 APPENDIX D. PDSoS - Systemic Mapping Protocols It was also considered the studies cited as related works in the relevant primary studies found in the engine search; ∙snowballing: To increase the accuracy of our SM was used the snowballing. The snowballing refers to using the reference list of a paper or the citations to the paper to identify additional papers (WOHLIN,2014a). If using the reference is called of backward snowballing, on the other hand, using the citation is known as forward snowballing. The steps following in our SM for snowballing procedure is described in (WOHLIN,2014a). ∙expert opinion: Studies suggested by experts in SoS domain also was considered. Although the indication of studies by expert can be considered as bias, we have adopted this source aiming to not lose any important evidence. Selection criteria In order to accordingly select the studies to answer our research questions, we established the Inclusion (IC) and Exclusion Criteria (EC), which it was used in the first and second selection. In the first selection, the application of the criteria were limited to primary studies metadata (i.e., title, abstract, and keywords). At the second selection, the criteria were also considered in reading full study. In the case of any discordance in the first selection, the studies were included and analyzing in the second selection. In the second selection, in case of discordance, the study was discussed together with experts until reaching an agreement. The IC and EC are: Inclusion Criteria IC.1: The study discusses SoS development process. IC.2: The study addresses SoS characteristics, problems or activities related to SoS development processes. Exclusion criteria EC.1: The study is not related to SoS. EC.2: The study does not discuss any SoS development process. EC.3: Development process is not the main focus of the study. EC.4: The study is an editorial, keynote, opinion, tutorial, poster or panel. EC.5: The study is duplicated or there is newer or a more complete one about the same research. EC.6: The study compiles results of other studies. EC.7: The study is not written in English. EC.8: The full text of the study is not available. 209 Quality Assessment Criteria In order to evaluate the primary studies were adapted the criteria described in (KHAN et al., 2001) (KITCHENHAM; CHARTERS,2007) (KITCHENHAM et al.,2010). The quality criteria and the specification scores for the SM are shown in Table 24. The possible score depends of each quality criteria. When the primary study fully matches a quality criterion, it had 2 as score and 0 otherwise. In some cases, the quality criterion is partially met by the primary studies and in those case, intermediate scores were defined, which can range from 0.5 to 1.5. The final score of the quality assessment (QA) for each primary study was calculated using the following formula: QA = i=1 ∑ 8CQ_i maximum score *100 Table 24 – Quality assessment criteria Source: Adapted from Kitchenham et al. (2010). Quality Criteria (QC) Score QC1: Do the authors clearly state the aims of the research? 0 - No 1 - Partially 2 - Totally clear QC2: Do the authors discuss the limitations of their study? 0 - No 1 - Partially 2 - Clearly discussed QC3: Do the authors state the findings clearly ? 0 - No 1 - Clearly partially 2 - Totally clear QC4: Is there evidence that the findings of the study can be used by other researchers/practitioners? 0 - No 1 - Partially reusable 2 - Totally reusable QC5: Does the study body fully meet issues provided in its abstract? 0 - No 1 - Attend partially 2 - Attend completely QC6: Do the described methods were empirically evaluated? 0 - No 1 - Partially evaluated 2 - Totally evaluated QC7: How the empirical evaluation conducted? 0 - No 0.5 - Illustrative example 1 - Case study 1.5 - Researchers/Experts 2 - Use in industry QC8: Are there any ideas for further investigation presented? 0 - No 2 - Yes Data Extraction and Synthesis Method The data extraction is designed to gather the information necessary to answer the research questions and evaluate the quality of the study (MAGDALENO et al.,2012). Thus, for each 210 APPENDIX D. PDSoS - Systemic Mapping Protocols primary study included, data were extracted and discussed with the experts. To extraction of the data we used the Start Tool 2 (FABBRI et al.,2012), Microsoft office Excel 2013 and considered three categories of data: (i) characterization of studies (e.g., author, year, and publication vehicle); (ii) infrastructure used in development of approaches (e.g., technologies, tools, and languages); and (iii) general features of the studies (e.g., SoS characteristics, technical factors, and processes) that enable us record details of the studies, organizing the synthesis in table, graphic or graphs. To share the activities of the research protocol and the steps followed by the reviewers, it becomes possible to audits and facilitates the process of a new conduction, addressing issues related. 2<http://lapes.dc.ufscar.br/tools/starttool>