Software Product Line Engineering: A Practical Experience

Software Product Line Engineering: A Practical Experience Jose-Miguel Horcas [email protected] Univ. de Málaga, CAOSD group, Spain Mónica Pinto [email protected] Univ. de Málaga, CAOSD group, Spain Lidia Fuentes [email protected] Univ. de Málaga, CAOSD group, Spain ABSTRACT The lack of mature tool support is one of the main reasons that make the industry to be reluctant to adopt Software Product Line (SPL) approaches. A number of systematic literature reviews exist that identify the main characteristics offered by existing tools and the SPL phases in which they can be applied. However, these reviews do not really help to understand if those tools are offering what is really needed to apply SPLs to complex projects. These studies are mainly based on information extracted from the tool documentation or published papers. In this paper, we follow a different approach, in which we firstly identify those characteristics that are currently essential for the development of an SPL, and secondly analyze whether the tools provide or not support for those characteristics. We focus on those tools that satisfy certain selection criteria (e.g., they can be downloaded and are ready to be used). The paper presents a state of practice with the availability and usability of the existing tools for SPL, and defines different roadmaps that allow carrying out a complete SPL process with the existing tool support. KEYWORDS SPL in practice, state of practice, tool support, tooling roadmap 1 INTRODUCTION An increasing number of software development companies are adopting Software Product Line (SPL) approaches [ 8 , 47 ]. However, the field of SPL is quite broad and continuously changing [ 57 ]. Technical issues such as the development of the SPL infrastructure, including new practices, processes, and tool support require a great investment [ 8 ]. Thus, despite the amount of successful stories about the use of SPL engineering 1 , industry has not yet solved the variability and reuse management problem and continues to experiment with their own solutions and approaches [ 13 ]. This is partly due to the fact that companies are reluctant to adopt technically sound academic approaches due to the lack of mature tool support. Hence, the success of SPL depends on good tool support as much as on complete SPL engineering processes [6]. Regarding the processes, SPL approaches typically cover the domain and application engineering processes that include activities such as variability modeling and artifact implementation (domain engineering) and requirements analysis and product derivation (application engineering) [ 4 , 11 ], but set aside other activities that are important for companies, such as the analysis of non-functional properties (NFPs) or quality attributes and the evolution of SPL’s artifacts [ 41 ]. When considered, these activities are usually integrated into the traditional SPL process by reusing existing mechanisms, which were not specifically designed for that purpose (e.g., using attributes of extended variability models to model NFPs [ 10 ]). The most common is to find companies that only adopt a minor part of an SPL methodology (e.g., implementing variability with annotations) [ 30 ], which is sometime abandoned after a short period of time because of the lack of integration among the SPL activities. Only a few companies with enough financial resources have succeeded on using SPLs, due in most cases to the development of their own tools or the use of commercial tools (e.g., pure::variants [ 12 ], Gears [40]) that are not so accessible for smaller companies. Besides, although tool support is of paramount importance for the SPL management process [ 6 ], most existing tools only cover specific phases of the SPL approach (e.g., variability modeling or artifacts implementation). Those few tools that support several phases (e.g., FeatureIDE [ 64 ]) demand to fit an implementation technique such as Feature-Oriented Programming (FOP) [ 53 ], Aspect-Oriented Programming (AOP) [ 38 ] or annotations [ 37 ], depend on the development IDE (e.g., Eclipse) or present some important limitations [ 20 ]. For instance, applying classical SPL approaches (e.g., FOP or AOP) to web engineering is challenging because of the nature of web applications that require the simultaneous use of several languages (JavaScript, Python, Groovy,...) in the same application [ 30 ]. In addition, there are few works specifically focused on studying the SPL tool support [ 6 , 43 , 51 ] and, they usually report information that is extracted from the tool documentation or reference papers without really testing the tool availability and usability. This paper explores the existing tool support for SPL from a practical point of view, although it does not pretend to be a systematic review. The objective is to check out the existence of enough mature tool support for carrying out a complete SPL process. For each activity in the domain and application engineering, we identify the desired requirements that tools should provide to deal with complex SPL processes and application domains, and analyze the 1http://splc.net/hall-of-fame/ possibilities and limitations of each tool. The paper answers the following Research Questions: RQ1: What are the tools that provide some kind of support for SPLs? Are they available, usable, and keep up to date? To answer to this question, this paper presents a state of practice of the SPL tools, focusing on their availability and usability and selecting those that a company could use in its development process to successfully adopt an SPL process (Section 4). RQ2: How the tools behave when they are used to develop complex SPLs? (Either because a specific SPL development approach is required or because SPLs for complex application domains need to be developed.) We answer to this question by empirically analyzing the most usable tools. We use a running case study based on an SPL process with demanding characteristics such as clonable features [24, 60], variable features [21, 22], attributes [10], huge feature models, and so on (Section 5). RQ3: Is it possible to carry out a complete SPL process with the existing tool support? That is, is it possible to cover all activities of complex approaches including dealing with NFPs, and managing the evolution of SPLs? We answer to this question by defining up to 12 different roadmaps of tools that partially or completely support all phases of an SPL process (Section 6). The paper is structured as follows. Section 2 discusses related work. Section 3 motivates our study showing the requirements of complex SPL approaches. Section 4 presents a state of practice of the existing tools. Section 5 analyzes the most usable tools. Section 6 defines the tool roadmaps to carry out a complete SPL. Section 7 discusses threats to validity and Section 8 concludes the paper. 2 RELATED WORK Existing works have investigated the SPL processes in great detail [17, 56, 57], but has only surfaced its tool support [6, 43, 51]. 2.1 Software Product Line processes Multiple systematic literature reviews (SLRs) and surveys have been published covering different aspects of SPL engineering [ 11 , 16 , 17 , 48 , 56 , 57 ], such as the level of alignment in the topics covered by academia and industry [ 11 ], the level of tool support [ 16 ] or the most researched topics in SPL [ 56 , 57 ]. From these studies the phases and topics of SPL engineering in which academia and industry are more interested or that deserve more attention can be identified. These SLRs highlight some interesting conclusions. For instance, architecting [ 15 ] is the dominating SPL activity, covered by 38% of surveyed papers (56% of them are industry papers) [57]. However, there are other activities that are becoming important in the context of SPL with the emerging of new application domains, such as the Internet of Things (IoT) [ 55 ] or Cyber-Physical Systems [ 35 ], and that are not receiving the required attention. Examples of these activities are the optimization of large-scale variability models [ 48 , 49 ], the variability modeling of quality attributes [ 31 , 68 ], the management of NFPs [ 34 , 48 ], and the evolution of the SPL models [ 32 , 41 ]. This imposes new challenges to the existing development and analyses processes, as well as to the tool support. Even so, as exposed by those SLRs [ 11 , 57 ], the variability in quality attributes is a concern only in 6% of the papers, and NFPs are discussed only in 5% of all the papers [57]. 2.2 Software Product Line tools Few works study the tool support for the SPL processes specifically [ 6 , 16 , 43 , 51 ]. They are SLRs or surveys that are normally done only from the perspective of the documentation found for each tool, and the characteristics listed and discussed in that documentation. Moreover, most of the details about tools are covered in gray literature, thesis, and websites, that are not usually considered as primary studies in SLRs. Commercial tools (e.g., Gears [ 40 ] and pure::variants [ 12 ]) present the additional problem of the intellectual property protection of their technical details [ 6 ]. Concretely, in [ 12 ], a demonstration of pure::variants across the product line lifecycle is described, but it only surfaces the tool without providing further technical details. The same occurs with Gears in [40]. One of the most recent systematic studies [ 6 ] covers tools documented in research papers from 1997 until 2015 (although the study was published in 2017). This is an interesting study to know the general characteristics of these tools (e.g., technology used in its implementation, if it has a graphical or textual notation, etc.). Others similar, but older studies are [ 51 ] (published in 2014) and [ 43 ] (published in 2010). However, the most recent tools reported by these studies date from 2013 and 2005, respectively. The conclusion is that these kinds of studies are not enough to select the most appropriate tool to provide support for an SPL process. This is basically because only information about the highlevel phases covered by each tool is provided, omitting the details about the specific topics covered for each phase. In addition, the information is extracted from the tool documentation or a reference paper, and thus, these studies stay outdated very soon because, in most of the cases, they are not trying and striving directly with the tools, downloading, installing, and executing the tools or even checking their online availability — i.e., many of the tools included in existing studies are not available at all. There are even tools referenced in these papers that have never been implemented [ 51 ]. 3 MOTIVATION AND CASE STUDY An SPL approach should cover at least the domain engineering and the application engineering processes with their typical phases and activities (Figure 1): (1) variability and dependency modeling in the Domain Analysis (DA) phase; (2) feature selection and product configuration in the Requirements Analysis (RA) phase; (3) the development of reusable software artifacts in the Domain Implementation (DI) phase; and (4) variability resolution and product generation in the Product Derivation (PD) phase [ 4 ]. However, demanding requirements of real SPL projects expose the needs of additional activities that are present in any software engineering process and that are essentials for the successful adoption of SPLs in industry. Some examples are the analysis of NFPs or quality attributes, or the evolution of the SPL’s artifacts, among others. To illustrate those points, we present WeaFQAs [ 29 , 31 , 34 ], an example of an SPL process that extends the engineering framework for SPL [ 52 ] and demands specific needs in each of the SPL phases according to the current applications and domains where WeaFQAs may be used. In this section, we describe the particular needs that WeaFQAs imposes in each phase of the classical SPL framework and enumerate the requirements that tools should satisfy. 3.1 Case Study: the WeaFQAs SPL process WeaFQAs [ 29 , 31 , 34 ] is an SPL process to manage the operationalizations of quality attributes. The operationalization of a quality attribute (e.g., security) is the association of a function (e.g., the encryption of a message) to a goal (e.g., providing security). WeaFQAs introduces the concept of Functional Quality Attribute (FQA) as the specific functionality that is incorporated into the applications to fulfill the desired quality attribute. WeaFQAs promotes the variability modeling and customization of FQAs separately from the applications, and their later incorporation into them, exploiting the benefits of SPLs (e.g., reusability, adaptability,. . . ). WeaFQAs follows the classic framework for SPL engineering [ 52 ] (Figure 1) — i.e., DA, RA, DI, and PD., but needs to extend it to take into account the specific problematic of FQAs, which requires additional activities in each of those phases (see activities in bold in Figure 1). 3.2 Requirements of complex SPLs Domain Analysis (DA) . Modeling a family of FQAs starts with the characterization of the FQAs and their variability modeling (Figure 2). For each quality attribute (e.g., security) the FQAs that are required to satisfy it are modeled (e.g, encryption, authentication). For instance, characterizing the encryption FQA includes the identification of the different encryption algorithms, the available security frameworks that implement it, along with their variables and parameters such as the key length, block cipher mode of operation, and padding; the dependency relationships between encryption and others FQAs (e.g., hashing); as well as the usage context [34]. Nevertheless, modeling the FQAs variability is more complex that only considering their high degree of variability. Since each FQA (e.g., encryption) can be applied in several points of the application with different configurations (e.g., RSA or AES algorithms), each FQA needs to be modeled as a clonable feature [ 24 , 60 ] (see Encryption[1..*] in Figure 2). Moreover, some FQAs include variables that require to provide specific values (e.g., 512 kB as the average size of the messages to be encrypted). This requires to model them as variable features [ 21 , 22 ] (see MessageSize in Figure 2). Here, it is worthy to differentiate between variable features [ 21 , 22 ] that are those that require to provide a value (e.g., integer, string, float) during configuration; and features with attributes [ 10 ], which can be used to model NFPs such as the performance or price of features (see Execution time and Energy consumption attributes of the RSA feature in Figure 2). The incorporation of clonable and variable features brings into play the definition of complex crosstree constraints. Examples are modeling the dependency between features that are children of different clonable features, or a dependency involving a numerical value (see cross-tree constraints in Figure 2). Thus, a requirement about variability modeling is: DA.Req1. Support for complex variability modeling including clonable features, variable features, features with attributes, and complex constraints. WeaFQAs also considers NFPs such as performance and energy efficiency in the context of an FQA configuration. Although dealing with NFPs in feature models as attributes [ 10 ] is a wide accepted approach [ 27 ], it presents some issues. Firstly, NFPs usually compete and conflict with each other. For example, using the AES encryption algorithm has a high energy efficiency but provides a poor Domain Analysis (DA) (including FQAs characterization, variability modeling, dependency modeling, NFPs modeling,...) Domain Implementation (DI) (including FQAs software architectures, models, source code, usage of third-party libraries,...) Mapping Requirements Analysis (RA) (including feature selection, optimization of FQA configurations, stepby-step configuration,...) Product Derivation (PD) (including variability resolution, weaving FQAs, multi-line SPL, propagation of evolving changes, validation, verification,...) Configuration Features Artifacts Customer needs Domain knowledge Product New requirements Problem Space Solution Space FQAs Domain Engineering FQAs Application Engineering Figure 1: The WeaFQAs SPL process. performance in execution time [ 34 ]. These relationships cannot be handled with the basic constraints of feature models (e.g., includes and/or excludes) and require more specific treatments (e.g., use of the NFR+ Framework) [ 18 ]. Secondly, sometimes the management of NFPs needs to be postponed until the requirements analysis or even product derivation phase because the NFPs’ information about a feature is not available until the product is completely generated and tested, unless domain experts provide predictions and estimations of them. For example, to obtain the real energy efficiency or memory footprint of a product is necessary to evaluate it as a whole configuration and not as an independent features. DA.Req2. Support for dealing with and managing NFPs that can be associated to individual features or complete configurations. These two requirements appear in other current domains, such as the IoT [ 35 , 55 ], where clonables features are essential to model the variability of the different devices, and NFPs need to be modeled when deployments have to be generated according to the tradeoff between different NFPs, such as latency and battery consumption. Another challenge that WeaFQAs poses is the size of its variability model. Considering 20 FQAs modeled with a total of 178 features and 23 constraints, there are 5 . 72 e 24 configurations. Variability models of this size are unmanageable and thus is desirable to modularize them, as for example, using composite variability models as defined for the CVL language [ 22 ] (see Context feature in Figure 2). Another solution to modularize an SPL that could fit very well with WeaFQAs is the use of a multi product line (MultiPLs) approach [ 58 ]. In a MultiPL approach, the FQAs for each quality attribute (e.g., the encryption, authentication and hashing FQAs for the security attribute) would be defined in separated SPLs. These SPLs would then be composed when used in a specific application. DA.Req3. Support for modeling/managing huge feature models. Requirements Analysis (RA) .WeaFQAs allows creating the FQAs configurations according to the application requirements, but also generating optimum configurations based on NFPs, such as performance or energy efficiency. Generating optimum configurations is an intractable problem when dealing with huge variability models or models containing variable features. As discussed in ,ĂƐŚŝŶŐ΀ϭΎ΁>ŽŐŐŝŶŐ΀ϭΎ΁ ŶĐƌLJƉƚŝŽŶ΀ϭΎ΁ &YƐ ϭΎ ŽŶƚĞdžƚ DĞƐƐĂŐĞ^ŝnjĞ/ŶƚĞŐĞƌ DĞƐƐĂŐĞdLJƉĞ ^ƚƌŝŶŐ &ŝůĞKďũĞĐƚ ŽŶƚĞdžƚ ^ ůŐŽƌŝƚŚŵ Z^  džĞĐƵƚŝŽŶƚŝŵĞс΀ϭϱϬϳϬϬ΁ŵƐ :ĂǀĂdžĐƌLJƉƚŽ /ŵƉůĞŵĞŶƚĂƚŝŽŶ ŽƵŶĐLJĂƐƚůĞ <ĞLJ/ŶƚĞŐĞƌ yD> &ŽƌŵĂƚ dyd ŽŶƐŽůĞ ,ĂŶĚůĞƌ &ŝůĞ^LJƐƚĞŵ   ^,ϭ &ƵŶĐƚŝŽŶ ^,Ϯ  ŽŶƚĞdžƚ ŽŶƚĞdžƚ Şś    ɩʳʰʳʰɩɥɫɯŜſɩƀ ŜŜŜ LJƉŚĞƌĞĚ DĞƐƐĂŐĞƐ >ĞŐĞŶĚ ŶĞƌŐLJĐŽŶƐƵŵƉƚŝŽŶс΀ϰϭϲ΁:  ĨĞĂƚƵƌĞ ŶƵŵĞƌŝĐĂů ΀ϭΎ΁ ĐůŽŶĂďůĞ ĐŽŵƉŽƐŝƚĞ ŽƉƚŝŽŶĂů ŵĂŶĚĂƚŽƌLJ ϭΎ Žƌ džŽƌ ϭϭ ĂƚƚƌŝďƵƚĞ Figure 2: Excerpt of the WeaFQAs’ variability model (complete model available in [36]). DA.Req2 , sometimes it is necessary to generate several configurations (or even all) in order to evaluate them and finding the best configuration before delivering the final product, or to reason about the system’s variability. In those cases, specific formalizations of variability models (e.g., CNF [ 10 ] or BDD [ 25 ]) and reasoners and solvers (e.g., SAT [ 67 ], CPS [ 54 ]) need to be used depending on the type of analysis (e.g., calculate number of configurations, finding the optimum configuration, or generate all configurations). RA.Req1. Support for analyzing and generating optimum configurations from huge space configurations, based on different criteria (e.g., NFPs). In this phase, another activity that WeaFQAs considers is assisting application engineers when choosing a configuration. In this sense, WeaFQAs provides advises about the most appropriate selections based on the application engineers’ goals. For example, to decide the most secure encryption algorithm or the most efficient framework that implements the RSA algorithm. This kind of assistant requires to manage partial configurations that will be completed in a step-by-step process. RA.Req2. Support for partial and step-by-step configurations. Domain Implementation (DI) .There are frameworks and third party libraries that provide implementations of FQAs ready to be used, such as the Java Security package, the Apache Commons library, and the Spring Framework. So, implementing the FQAs’ artifacts from scratch using a specific variability mechanism such as FOP or AOP is not an advisable option. Instead, the challenge posed by WeaFQAs is to handle the variability of existing artifacts in order to configure their functionality according to the selected features during the RA phase. Furthermore, the recurrent nature of the FQAs makes them suitable to be used in many different domains. Domains like web engineering involve multiple types of programming languages (e.g., JavaScript, Python, Groovy), and markup languages (e.g., HTML, CSS, XML) where applying typical variability development paradigms of SPL such as FOP or AOP is extremely difficult or even impossible [30]. DI.Req1. Support for using and combining different variability mechanisms (FOP, AOP, annotations,...) independently from the language, and applying them to existing implementations. Additionally, WeaFQAs supports the management of FQAs at the architectural level. This means that WeaFQAs can handle architectural configurations of FQAs defined in any modeling language based on MOF (Meta-Object Facility), by using Model-Driven Engineering. DI.Req2. Support for managing artifacts variability at different abstraction levels: from software architecture models to code. Product Derivation (PD) .In this phase, beyond the automatic generation of the product (here FQAs configurations), WeaFQAs has a specific need: the weaving of the FQAs into the final application. Following the WeaFQAs approach, where FQAs are modeled separately from the application, each generated configuration needs to be incorporated (woven) in different places of the application. In WeaFQAs this weaving activity can be performed together with the product derivation or in an independent activity (e.g., following a MultiPL [ 58 ]). In both cases, WeaFQAs supports the weaving process at the architectural level (using MOF-compliant models) or at the code level (by defining custom transformations) [29]. PD.Req1. Support for weaving products or multi product lines. Finally, an important activity in an SPL process is the evolution engineering. When the applications’ requirements change and/or the technology of the FQAs evolves, domain knowledge and artifacts need to be updated and the changes need to be propagated to the different deployed configurations. In some domains, as for example multi-tenants applications [ 32 ], with hundreds of configurations deployed, this activity should be performed automatically. PD.Req2. Support for automatic propagation of evolving changes to the deployed products. In the rest of the paper we take into account these requirements in order to answer our research questions. 4 STATE OF PRACTICE This section answers our first research question: RQ1: What are the tools that provide some kind of support for SPLs? Are they available, usable, and keep up to date? We analyze the current state of practice of SPL tools to identify which ones are available online, and are really usable for industry, ϮϬϬϬ ϮϬϮϬ ϮϬϬϭ ϮϬϬϮ ϮϬϬϯ ϮϬϬϰ ϮϬϬϱ ϮϬϬϲ ϮϬϬϳ ϮϬϬϴ ϮϬϬϵ ϮϬϭϬ ϮϬϭϭ ϮϬϭϮ ϮϬϭϯ ϮϬϭϰ ϮϬϭϱ ϮϬϭϲ ϮϬϭϳ ϮϬϭϴ ϮϬϭϵ ϬϬϭ &Dd /ŶǀĂƌ >/^ƚŽŽůŬŝƚ KWd/Ͳ^>d ^ĐĂƚƚĞƌdŽŽů KZ^> s>Ϯ ZͲKd^ /D> ĞĐŝƐŝŽŶ<ŝŶŐ KsDK&Ͳs^ KW>ZKKZ^džƚĞŶƐŝŽŶ ZD 'ĞŶƌĐŚ ,ĞƉŚĂĞƐƚƵƐ<ƵŵďĂŶŐ ><Ͳ&ĞĂƚƵƌĞDŽĚĞůŝŶŐdŽŽů /^Ddϰ^W> DŽ^W> Dh^W>h^^ƚŽŽůŬŝƚ ZDWͲƚŽŽůZĞƋƵŝ>ŝŶĞ ^LJƐŝƉŚƵƐͲ/sDDsZDK s/^/dͲ&zĂDyD>Ͳ&D sD tĞď&D> sh>E ĞddLJ &>D ^W>Z sĂƌŝĂDŽƐ KĚLJƐƐĞLJ &ĂDĂ ,LJĚƌĂ &Dd 'ůĞŶĐŽĞ ,^ &D/>/Z ĂƉƚĂŝŶ&ĞĂƚƵƌĞ DK^<ŝƚƚϰ^W> WƌŽĚƵĐƚůŝŶZ WĂĐŽŐĞŶ &ĞĂƚƵƌĞDŽĚĞů^> ^ϮdϮŽŶĨŝŐƵƌĂƚŽƌ ĨŵƉ&ĞĂƚƵƌĞDŽĚĞůŝŶŐWůƵŐͲŝŶ ůĂĨĞƌ ^W>Kd <ĞƌŶĞů,ĂǀĞŶ W>h^ sDtdy&ĞĂƚƵƌĞ sZ &ĞĂƚƵƌĞDĂƉƉĞƌ W>hD dŽŽůĂLJ DW>DͲDĂdĞ>Ž s/d> sŝsŝŽŶĐĞƌŶDĂƉƉĞƌ hŶĚĞƌƚĂŬĞƌ DŽ^/^s>dŽŽů sZdŽŽůƵŶĚůĞ 'ĞĂƌƐ ƉƵƌĞǀĂƌŝĂŶƚƐ ĂƐLJͲWƌŽĚƵĐĞƌ &ĞĂƚƵƌĞ/ sŝĞǁ/ŶĨŝŶŝƚLJ ƐƉĞĐƚ: &ĞĂƚƵƌĞнн &ĞĂƚƵƌĞ,ŽƵƐĞ , ^W>dŽŽů ǀyŐŝŶĞ /н DĞƚĂĚŝƚнн / ydŽ& KͲ>dŽŽů^ƵŝƚĞ &ĞĂƚƵƌĞŽŵŵĂŶĚĞƌ > ys> DƵŶŐĞ ŶƚĞŶŶĂ ĞůƚĂ: Ξ Ξ Ξ Ξ <s> ;ϭϵϵϯͿ ;ϭϵϵϵͿ tĞŽdŝŶ WƵ>^ ;ϭϵϵϴͿ ;ϭϵϵϳͿ ϮϬϬϬ ϮϬϮϬ ϮϬϬϭ ϮϬϬϮ ϮϬϬϯ ϮϬϬϰ ϮϬϬϱ ϮϬϬϲ ϮϬϬϳ ϮϬϬϴ ϮϬϬϵ ϮϬϭϬ ϮϬϭϭ ϮϬϭϮ ϮϬϭϯ ϮϬϭϰ ϮϬϭϱ ϮϬϭϲ ϮϬϭϳ ϮϬϭϴ ϮϬϭϵ DĞƚĂĚŽĐ&D ,ŽůŵĞƐ ǀĂŝůĂďůĞĂŶĚƵƐĂďůĞŝŶŐƌĞĞŶ EŽƚĂǀĂŝůĂďůĞƚŽĚŽǁŶůŽĂĚŽƌŶŽƚǁŽƌŬŝŶŐŝŶƌĞĚ WĞƌŝŽĚƵŶĚĞƌĂĐƚŝǀĞĚĞǀĞůŽƉŵĞŶƚ >ĞŐĞŶĚ /ŶƚĞŐƌĂƚŝŽŶ WƌŽƚŽƚLJƉĞ Ξ WƌŽďůĞŵ^ƉĂĐĞ ^ŽůƵƚŝŽŶ^ƉĂĐĞ WƌŽďůĞŵΘ^ŽůƵƚŝŽŶ^ƉĂĐĞ ĐĂĚĞŵŝĐƚŽŽů Figure 3: State of practice of SPL tools. Domain Analysis (DA): 37% Requirements Analysis (RA): 36% Domain Implementation (DI): 17% Product Derivation (PD): 7% All phases: 3% SPL phases covered Available and usable: 19% Not available, complex setup, or not working: 81% Availability and Usability Figure 4: Summary of existing tools. practitioners, and the SPL community. The goal is to collect all possible tools related with SPL to check their status before considering them to analysis. This does not pretend to be a systematic review of tools, but an in-depth study to identify existing tools. Research method .We performed a manual search on different sources. First, we identified SLRs [ 6 , 51 ] and surveys [ 43 ] about SPL tools. We also searched the proceedings of the demonstration and tool track in some of the most relevant events about SPL and variability (e.g., SPLC, VaMoS), for the period not covered by the SLRs and surveys (2015-2018). Second, for each reported tool we searched for its availability (i.e., its website, code repository or executable). When the information was not available in the paper we performed a manual search on web search engines (e.g., Google) to localize the tool by applying the following search strings: <<name of the tool>> , tool , SPL , Software Product Line , and variability . Finally, we glanced at the tool by downloading, installing, and launching it to check its correct functioning. Data extraction form . We used Google Forms 2 to capture the basic information about the availability of the tools: name, brief description, URL, main reference, SPL’s phases covered, type of tool (academic, commercial, prototype), first and last release date, availability, current status and integration with other tools. This data has been extracted from the information found in reference papers, the official website, and the code repository of the tool. The only inclusion criteria for this form is that the tool is directly related with SPL or is being used in the context of SPL. Result . To illustrate the state of practice, we have built a timeline (Figure 3) with all SPL tools published until February 2019 3 . We found 97 tools. As summarized in Figure 4, only 3% of them cover all phases of the SPL process. Moreover, there seem to be more interest in the problem space than in the solution space since the DA (37% of the tools) and the RA (36%) are the phases most covered by the tools. The DI and PD phases are only covered by 17% and 7% of the tools respectively. The main problem with SPL tools is the fact that only 19% are available online to be downloaded and used, have an easy installation process or can be directly used online in a web browser (green timelines in Figure 3). Characteristics that make them attractive to be used in the community and industry. The other 81% are obsolete tools, have complex installation process, work with errors, or they are not available at all (red timelines in Figure 3). This evidences that there are lot of tools but most of them are academic or prototypes tools that are abandoned shortly when the associated research project ends. The state of practice gives a wide vision of the current state of art of the SPL tools and helps practitioners to select appropriate tools. The main artifacts developed that allows replicating and/or improving this state of practice are available in [36]. 5 TOOL SUPPORT FOR COMPLEX SPLS This section answers our second research question, selecting first a subset of the tools identified in Section 4: RQ2: How the tools behave when they are used in complex SPLs? 5.1 Tool selection Four inclusion criteria (IC) are considered relevant to answer RQ2: IC1: The tool is available online to be downloaded or used. IC2: The installation process is straightforward and does not require complicate settings, additional dependencies or thirdparty plugins that can be obsolete. IC3: The tool is under active development or the latest release is a final stable version. IC4: The tool covers at least one of the main phases of the SPL process defined in Section 3 and in [4, 52]. Five exclusion criteria (EC) were used to exclude tools that we do not consider appropriate to be used in a professional environment: EC1: The tool is a prototype or a preliminary or beta version. EC2: The tool does not work or works with errors. 2https://forms.gle/JfH9bKHHTgCLc31R7 3A .csv file with the tools information is available in [36]. EC3: The tool is commercial and the owner does not provide an alternative free limited version. EC4: The tool is a compilation of independent tools not related to SPL (e.g., a CASE tool). EC5: There is another similar, more actual or useful tool, or the tool has been integrated within another. By applying our inclusion and exclusion criteria we have chosen the SPL tools to be analyzed in this section (Table 1). Note that there are many other tools that are available and usable (e.g., Feature- House [ 5 ], AHEAD [ 9 ],...), but they have been excluded by EC5 since they are integrated within other tools like FeatureIDE [ 64 ]. Others are very updated (e.g., ProductlineRE [ 26 ]) but have so many obstacles to be installed, so they do not pass IC2. Others are exclusive for a specific domain (e.g., FMCAT [ 7 ] focusing on the analysis of dynamic services product line). Others are generic tools that are not specific of SPL, even if they are used as part of some SPL tools in concrete phases of the SPL engineering process (e.g., AspectJ to implement artifacts following an AOP approach). 5.2 Tool analysis In this section we analyze the selected tools to check whether or not they satisfy the requirements of a complex SPL process like the WeaFQAs process presented in Section 3. All artifacts developed and used throughout the different phases to test the tools are available to repeat the experiments in [ 36 ]. This includes the variability models in different formats and the implementation code of the artifacts. 5.2.1 Domain Analysis (DA) phase .Apart from variability and dependency modeling, the DA process should take into account requirements DA.Req1, DA.Req2, and DA.Req3. Experiments. We have modeled the variability of the FQAs (Figure 2) with the selected tools. The WeaFQAs variability model includes clonable features ( Encryption [1..*] ), variable features ( MessageSize: Integer ), features with attributes ( Execution time and Energy consumption of the encryption algorithms), composite units to modularize the variability model and to facilitate its management and configuration ( Context ), arbitrary group multiplicity ( 1..* ), and complex cross-tree constraints like those involving variable features. Analysis. Regarding DA.Req1., all tools support basic feature models (mandatory and optional features, alternative (xor) and or groups, requires and excludes constraints). However the support for advanced characteristic is very limited (Table 2). Clonable features is the most difficult characteristic to be implemented, and thus, no tool provides support for them completely. Clafer allows cloning any feature in the variability model and configuring each instance, but this is done at the configuration step and deciding whether a feature is clonable or not should be done at the domain analysis phase. FeatureIDE and pure::variants allow a similar behaviour of clonable features by inserting subtrees in the variability model. In FeatureIDE, this characteristic follows the VELVET approach of MultiPLs [ 58 ], but it is still a prototype that only supports the FeatureHouse [ 5 ] composition approach when generating code, it contains many errors and with not enough documentation. Pure::variants introduces the concept of “variant instance” as a link in the feature model to another configuration space. In contrast to Table 1: Description of the selected SPL tools. Tool Year Last update SPL phases Description S.P.L.O.T. [45] 2009 Jan. 2015 DA, RA Online tool to edit, debug, analyze, configure, share and download feature models. It offers hundreds of feature models from academics and practitioners. Available in: http://www.splot-research.org/ Glencoe [2] 2018 Jan. 2019 DA Web application to work with variability models. Model importation from DIMACS or pure::variants [ 61 ]. Several solvers for the automated analysis of FMs. Available in: https://glencoe.hochschule-trier.de/ Clafer [3] 2014 Feb. 2018 DA, RA General-purpose lightweight language for structural modeling: feature modeling and configuration, class and object modeling, and metamodeling. Several solvers and model reasoners. Available in: https://www.clafer.org/ FeatureIDE [42, 64] 2004 Nov. 2018 DA, RA, DI, PD Open-source Eclipse framework with plug-in based extension mechanism to integrate and test existing tools and SPL approaches [ 39 ] (FeatureHouse, AHEAD, AspectJ, Antenna, etc.). Available in: http://www.featureide.com/ pure::variants [61] 2003 Dec. 2018 DA, RA, DI, PD Commercial solution that supports all phases of the SPL process. Many extensions that connect pure::variants with common systems and software engineering tools [12]. Available in: https://www.pure-systems.com/ vEXgine [33] 2017 Jan. 2018 PD Customizable implementation of the CVL execution engine [ 69 ] that can be extended with custom transformation engines to support multiple variability approaches. Available in: http://caosd.lcc.uma.es/vexgine/ Clafer, the number of instances for the clonable feature has to be decided in the domain analysis phase and not at the configuration step, where this decision is normally taken. The support for variable features and for feature with attributes is confused because of the thin difference between them. Clafer allows defining variable features with a specific type (e.g., integer) that behaves as a normal feature but allows providing a value during the configuration step, and also specifying constraints about the value of that feature. However, to support attributes in Clafer (as for example to specify a utility value for each feature) we have to rely in the Clafer Multi-Objective Optimizer (ClaferMOO [ 3 ]) that is a specific reasoner for attributed-feature models, or modeling the attributes as variable (numerical) features. This implies to define an additional variable feature (e.g., integer) for each normal feature in the variability model, and make sure those variable feature are selected in the final configuration. Contrary, pure::variants offers complete support for attributes but not for variable features that in this case can be implemented as attributes. FeatureIDE supports attributes only partially, because it requires selecting the composer “Extended Feature Modeling” and then, no other composer can be selected. Neither S.P.L.O.T. nor Glencoe support clonable features, variable features, and feature with attributes. Finally, each tool provides additional characteristics for variability modeling. For instance, Glencoe and pure::variants allow mixing mandatory features within “or” groups. Glencoe, Clafer and pure::variants support arbitrary multiplicity in group features (e.g., x..y where x can be distinct from 1 and y distinct from * ). FeatureIDE and Clafer allow defining abstract features. Clafer (with constraints involving variable features) and pure::variants (with Prolog, and its own variant of OCL: pvSCL [ 61 ]) allow defining complex constraints. Concerning DA.Req2., no tool provides explicit support for dealing with NFPs, relying on features with attributes to manage NFPs. Respecting DA.Req3., first, huge feature models cannot be easily modularized within existing tools. Clafer allows defining multiple feature models as abstract classes but all of them in the same file. FeatureIDE, as discussed for clonable features, supports MultiPLs but it is in its infancy and the feature models itself cannot be divided in multiple files. In pure::variants, the support is better since it defines an Hierarchical Variant Composition to link a feature Table 2: Tool support for the Problem Space: DA and RA. Requirements and Characteristics S.P.L.O.T. Glencoe Clafer FeatureIDE pure::variants Domain Analysis (DA) DA.Req1 Basic FMs (optional, mandatory, or, xor, requires, excludes) ■ ■ ■ ■ ■ Cardinality-based FMs (clonable features or features cloning/cardinalities) □ □ l l l Variable features (variable features with type — i.e., integer, string,... ) □ □ ■ □ l Extended FMs (features with attributes) □ □ l l ■ Other extensions (complex constraints, arbitrary group multiplicity, abstract features,... ) l l ■l l DA.Req2 Support for NFPs □□□□□ DA.Req3 Modularization of FMs (composition units, multiple variability models) □ □ l l ■ Evolution of FMs (modification of features — e.g., change variability type, move feature,...) □l■l■ Requirements Analysis (RA) RA.Req1 Analysis of FMs (statistics, validation,... ) ■ ■ l■ ■ Number of configurations (model counting, independently of the FM’s size) ■ ■ l l ■ Generation of configurations (enumerate all configurations) l□ ■ ■ l Optimization of configurations (e.g., based on NFPs) □ □ l□ □ RA.Req2 Partial configurations ■□■l l Step-by-step configuration ■ □ l□ □ ■Full support. lPartial support. □No support. Table 3: Tool support for the Solution Space: DI and PD. Requirements/Characteristic FeatureIDE pure::variants vEXgine Domain Implementation (DI) DI.Req1 Different variability mechanism (FOP, AOP, annotations,... ) ■ ■ l Multi-language / Language independent (used in the same project) l■ ■ DI.Req2 Model abstraction level (architecture, design, code,...) l l l Product Derivation (PD) PD.Req1 Product derivation (variability resolution) ■ ■ ■ Weaving products □l■ Multi Product Lines l□ □ PD.Req2 Evolution changes (automatic propagation of changes) □l l ■Full support. lPartial support. □No support. model inside another one. Second, modifications of the variability models once created can be complex in some tools like S.P.L.O.T. and Glencoe, where modifying a part of the feature model usually can be only achieved by removing that part and adding it again. Contrarily, Clafer and pure::variants allow even moving features from a branch to another in a straightforward way. Findings. Table 2 summarizes the characteristics supported by the analyzed tools. S.P.L.O.T. and Glencoe are the most usable tools for the DA phase since they are available online, intuitive and easy to use and even their models can be exported to FeatureIDE and pure::variants, respectively. However, they do not provide any support for advanced characteristics. Only Glencoe and FeatureIDE use the notation proposed by Czarnecki [ 23 ] that is the most comprehensible and flexible by now (and the most used) [ 10 ]. The notation of Clafer can be tedious for variability modeling although it provides good support for variable features and acceptable support for clonable features. S.P.L.O.T. and pure::variants share a similar interface to build the feature model, following a tree structure but each of them with its own notation. It is worthy to mention that there are other tools that provide explicit support for clonable and variable features such as the tools that provide support to the CVL language [ 69 ] (e.g., MoSIS CVL Tool [ 63 ] and BVR Tool Bundle [ 65 ]). However, those tools do not meet our inclusion/exclusion criteria because they are not currently available or are not in a usable state. 5.2.2 Requirements Analysis (RA) phase .The goal of this process is to select a desired combination of FQAs according to the application requirements. This phase should also consider the optimization of the FQA configurations based on NFPs, as well as assisting application engineers when generating the configurations. Experiments. By using the selected tools, we analyze the possible configurations of the FQA’s variability model, generating different (or all possible) configurations, and finding optimum configurations based on the NFPs (e.g., performance and energy efficiency in WeaFQAs) when possible. The challenge in this step is dealing with huge variability models such as the FQAs feature model. Analysis. Regarding RA.Req1., almost all tools provide some kind of support for analyzing the variability model. This means statistics and metrics about the variability model (core features, optional features, number of constraints,...), model validation (consistency, void feature model,...), and anomalies detection (dead features, false-optional features, redundancy constraints,...). Clafer can only validate the model syntactically since it is a text plain modeling language with a formal grammar, but no further analysis about variability is carried out by default. Depending on the requested analysis, each tool uses an specific feature model formalization and/or solver to perform the analysis. For example, to calculate the number of configurations or variability degree of the feature model, S.P.L.O.T. uses a Binary Decision Diagrams (BDD) engine [ 25 ] for which counting the number of valid configurations is straightforward. Glencoe uses a Sentential Decision diagram (SDD) [ 50 ] engine that enables to determine the total number of configurations within very short times. Within pure::variants is also possible to calculate the number of configurations for each subtree under a selected feature. However, these tools calculate the number of configurations without taking into account variable features, which considerably increments the total number of configurations. The other tools (Clafer and FeatureIDE) require to generate all configurations in order to enumerate them, and thus, with these tools is not possible to calculate the number of configurations for huge models, like the FQAs feature model, in a reasonable time. For instance, using the Choco solver [ 54 ] integrated in Clafer, it takes 1 hour to generate 13 e 6configurations from a total of 5 . 72 e 24 (calculated with S.P.L.O.T.), requiring more than a billion of years to generate all configurations. FeatureIDE, in addition, generates the associated code, so it requires more time. Additionally, none of the selected tools provide support for finding optimum configurations in feature models. Only Clafer, with its ClaferMOO module, provides a multi-objective optimization mode, but this implies to use another kind of model not related with the Clafer’s variability model. Even so, with Clafer and FeatureIDE, we could generate all configurations (for small models), evaluate them based on the NFPs (e.g., performance and energy efficiency) and then finding the optimum configurations by using an specific optimization tool [ 34 ]. Other techniques such as random sampling applied to SPL [ 49 ] and formalizations of the feature models such as CNF or the use of advanced SAT solvers [ 14 , 67 ] will allow reasoning about these aspects or help to solve these issues with huge models, but this is out of scope of this paper. With respect to RA.Req2., only S.P.L.O.T. and Clafer provide a reasonable good support to manage partial configurations and step-by-step configuration. S.P.L.O.T. provides validation and statistics of the partial configuration, but also auto-completion of the configuration with less features or the configuration with more features. This is done through an online step-by-step configuration assistant. Clafer allows generating configurations from a partial one thanks to its instantiation process based on constraint definition. FeatureIDE and pure::variants allow generating partial configurations and calculate the number of valid configurations from that partial configurations, but they do not incorporate a guided process like S.P.L.O.T. to assist the user. Findings. No tool allows generating all configurations efficiently for huge variability models like the required in WeaFQAs. Tools are able to calculate that the complete FQAs’ feature model has 5 . 72 e 24 different configurations, but without taking into account variable features (e.g., numerical values). In fact, nowadays, with the existing tool support for SPL it is not possible to generate optimum configurations of products based on some criteria like NFPs [ 34 , 49 ]. 5.2.3 Domain Implementation (DI) phase . This phase focuses on the implementation of the variable artifacts (e.g., models, code). Experiments. We have implemented some of the FQAs using different variability mechanisms, concretely AOP, FOP, and annotations. We have reused third party libraries like the Java Security package for the Hashing FQA, the BouncyCastle library for Encryption, and the SLF4J API for Logging. Analysis. Regarding DI.Req1., FeatureIDE is the tool that provides best support for different variability mechanisms. Concretely, it supports FOP using the FeatureHouse approach or AHEAD, AOP with AspectJ, and annotations with Antenna (Java comments), Colligens (C preprocessor) or Munge (Android), among others [ 44 ]. However, it is not possible to combine different approaches in different parts of the application (e.g., annotations and AHEAD) or to use different languages (e.g., Java and JavaScript). Actually, only the combination of FeatureHouse with Java and AspectJ is supported. The pure::variants tool also provides good support for AOP (e.g., AspectJ and AspectC++), and annotations (e.g., for Java, JavaScript, C++) with its own variation points system, but not for FOP. The family model [ 61 ] of pure::variants allows describing the variable architecture/code and to connect it via appropriate rules to the feature model. Nevertheless, most of the advanced options of pure::variants are only available in the commercial version. In the case of vEXgine, it is possible to use and combine different variability mechanisms but the resolution of that variability needs to be delegated to an external engine [33]. Concerning DI.Req2., FeatureIDE and pure::variants offer very good support for implementing the variability at the code level as discussed in DI.Req1, while vEXgine needs specific extensions to work at code level [ 30 ]. At a high abstraction level (architecture and design), both pure::variants and vEXgine offer the best support. However, pure::variants requires the commercial version to manage high abstract models (e.g., UML), and vEXgine requires to define the appropriate model transformations despite it supports any MOF-compliant model [ 33 ]. FeatureIDE offers the possibility to combine FeatureHouse and UML, but actually, this integration is not completely operable. Findings. With existing tools, it is very difficult to apply the variability mechanisms (e.g., AOP, FOP) to third party libraries like those that implement FQAs, and the solution is usually encapsulating the behaviour of those libraries in entities of the specific approach (e.g., aspects or features) or implementing the FQAs from scratch using a specific variability mechanism. Moreover, no tool supports an effective variability mechanism to be applied over several languages (Java, Python, JavaScript) in the same project. 5.2.4 Product Derivation (PD) phase .The product derivation phase is in charge of generating the final product (configurations of FQAs) by resolving the variability of the artifacts (FQAs) according to the selection of features made in the RA phase. Then, those configurations of the FQAs needs to be incorporated into the final application by some combination mechanism (weaving, or MultiPL). Also, the propagation of changes in the final products when the requirements changes or the domain artifacts evolve needs to be considered in this phase. Experiments. We have generated the final products by resolving the variability of the FQAs with different configurations based on the configuration models specified in the RA phase. When possible, we have incorporated those configurations to existing applications. For example, weaving the FQAs’ configurations to an electronic voting (e-voting) application [ 31 ]. Then, we have evolved the FQAs’ variability model and try to update the generated configurations. Analysis. Variability resolution and product derivation is achieved by all analyzed tools. A limitation in FeatureIDE is that only one composer (e.g., FeatureHouse, annotations) can be selected for an SPL application and thus the combination of different approaches requires to build and integrate a custom composer within FeatureIDE. Apart from resolving the variability, PD.Req1. cannot be completely satisfied. In fact, only vEXgine provides complete support for weaving FQAs by defining custom model transformations [ 29 ]. The flexibility of pure::variants allows integrating other tools like Git to partially support mixing variants [ 59 ]. FeatureIDE integrates the VELVET approach [ 58 ] for MultiPL, but this is a prototype and in this case the product derivation is not fully operable. Regarding PD.Req2. the support for propagating changes in the variability model to the existing configurations exists but is limited. In pure::variants, the source code of the product variants can be evolved by using merge operations from Git [ 28 , 59 ]. Also, with the help of specific model transformations and evolution algorithms [ 31 , 32 ], vEXgine can evolve the deployed artifacts, but the effort of defining those transformations is considerable. Findings. Apart from the basic activity in this phase existing tools have not paid attention to advanced characteristics (e.g., weaving, MultiPL, evolution). However, those characteristic could be incorporate in some tools thanks to their extension mechanisms such as the possibility to define new composers in FeatureIDE [ 39 ] or the custom engines and model transformations of vEXgine [33]. 6 SPL TOOLS ROADMAP This section answers our third research question: RQ3: Is it possible to carry out a complete SPL process with the existing tool support? To answer RQ3, based on the analysis in the previous section, we define some practical roadmaps to completely carry out an SPL process with the existing tool support (Figure 5). The answer for RQ3 is that existing tools do support the complete process of SPL, but with many limitations. As shown in Figure 5, Roadmap 1 with