scieee AI-readable full text Open interactive document viewer

Multi-objective test case prioritization in highly configurable systems: a case study

Parejo Maestre, José Antonio; Sánchez Jerez, Ana Belén; Segura, Sergio; Ruiz-Cortés, Antonio; López-Herrejón, Alberto E.; Egyed, Alexander

Publisher
Zenodo
DOI
10.5281/zenodo.14755608
Repository
zenodo.org
Language
en

Abstract

Supplementary material for the article "Multi-objective test case prioritization in highly configurable systems: a case study" published in Journal of Systems and Software (https://doi.org/10.1016/j.jss.2016.09.045). For the sake of replicability, the source code of our algorithm, the Drupal attributed feature model, experimental results and statistical analysis scripts in R are publicly available.

Full text

Supplementary Material Wednesday 6th July, 2016 Abstract This document provides supplementary material for the paper ’Multi-Objective Test Case Prioritization in Highly Configurable Systems: A Case Study’. Specifically, all the p-values of the null hypothesis statistical tests are provided. For each experiment two tables of p-values are provided: the first table contains the original p-values obtained by the specific statistical test used for comparison, second we provide a table with the corrected p-values obtained by applying the Holms post-hoc procedure for the correction of multiple comparisons. 1 Experiment 1 Table 1: Original p-values for the comparison of mono vs. multiobjective prioritization using functional objectives. Functional Multi-Objective Functional Mono-Objective Connectivity Dissimilarity Pairwise VCoverage Connectivity Dissimilarity 0.0273 0.9657 0.0015 0.0181 Connectivity Pairwise 0.0534 0.0001 0.0000 0.0000 Connectivity VCoverage 0.8897 0.0317 0.0000 0.0000 Dissimilarity Pairwise 0.0001 0.0000 0.0000 0.0000 Dissimilarity VCoverage 0.0350 0.7851 0.0002 0.0073 Pairwise VCoverage 0.0302 0.0000 0.0000 0.0000 Connectivity Dissimilarity Pairwise 0.0385 0.0000 0.0000 0.0000 Connectivity Dissimilarity VCoverage 0.0212 0.9048 0.0001 0.0085 Connectivity Pairwise VCoverage 0.0078 0.0000 0.0000 0.0000 Dissimilarity Pairwise VCoverage 0.0033 0.0000 0.0000 0.0000 1 Table 2: Adjusted p-values for the comparison of mono vs. multiobjective prioritization using functional objectives, obtained by applying the Holms correction procedure to the p-values of table 1. Functional Multi-Objective Functional Mono-Objective Connectivity Dissimilarity Pairwise VCoverage Connectivity Dissimilarity 0.2730 1.0000 0.0255 0.2172 Connectivity Pairwise 0.2730 0.0021 0.0000 0.0000 Connectivity VCoverage 1.0000 0.2730 0.0000 0.0000 Dissimilarity Pairwise 0.0021 0.0000 0.0000 0.0000 Dissimilarity VCoverage 0.2730 1.0000 0.0036 0.1095 Pairwise VCoverage 0.2730 0.0000 0.0000 0.0000 Connectivity Dissimilarity Pairwise 0.2730 0.0000 0.0000 0.0000 Connectivity Dissimilarity VCoverage 0.2332 1.0000 0.0021 0.1105 Connectivity Pairwise VCoverage 0.1095 0.0000 0.0000 0.0000 Dissimilarity Pairwise VCoverage 0.0528 0.0000 0.0000 0.0000 2 Experiment 2 Table 3: Original p-values for the comparison of mono vs. multiobjective prioritization using non-functional objectives. Non-Functional Mono-Objective Changes Faults Size Changes Faults 0.00000 0.01701 0.00000 Changes Size 0.02599 0.00000 0.87455 Faults Size 0.00000 0.44713 0.00000 Changes Faults Size 0.00000 0.00003 0.00000 Table 4: Adjusted p-values for the comparison of mono vs. multiobjective prioritization using non-functional objectives, obtained by applying the Holms correction procedure to the p-values of table 3. Non-Functional Mono-Objective Changes Faults Size Changes Faults 0,0000 0,0680 0,0000 Changes Size 0,0780 0,0000 0,8943 Faults Size 0,0000 0,8943 0,0000 Changes Faults Size 0,0000 0,0002 0,0000 2 3 Experiment 3 Table 5: Original p-values for the comparison of mono vs. multiobjective combinations of functional and non-functional objectives. Connectivity Dissimilarity Pairwise VCoverage Changes Faults Size Changes Connectivity 0.10728 0.57544 0.00003 0.00178 0.07416 0.00000 0.28069 Changes Dissimilarity 0.01376 0.78506 0.00258 0.02599 0.42377 0.00000 0.06237 Changes Pairwise 0.00211 0.00000 0.00000 0.00000 0.00000 0.00000 0.00004 Changes VCoverage 0.48749 0.12530 0.00000 0.00005 0.00524 0.00000 0.98092 Connectivity Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.63728 0.00000 Connectivity Size 0.14291 0.37543 0.00001 0.00044 0.04338 0.00000 0.45212 Dissimilarity Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.76915 0.00000 Dissimilarity Size 0.01069 0.73405 0.00649 0.05582 0.67716 0.00000 0.02172 Faults Pairwise 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Faults VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.71103 0.00000 Pairwise Size 0.00014 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Size VCoverage 0.12771 0.45791 0.00002 0.00094 0.06517 0.00000 0.32113 Changes Connectivity Dissimilarity 0.00899 0.94279 0.00044 0.00899 0.44636 0.00000 0.04042 Changes Connectivity Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00001 0.00000 Changes Connectivity Pairwise 0.00242 0.00000 0.00000 0.00000 0.00000 0.00000 0.00012 Changes Connectivity Size 0.10108 0.51186 0.00001 0.00081 0.05458 0.00000 0.29377 Changes Connectivity VCoverage 0.17123 0.35507 0.00000 0.00018 0.03947 0.00000 0.53055 Changes Dissimilarity Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00092 0.00000 Changes Dissimilarity Pairwise 0.00048 0.00000 0.00000 0.00000 0.00000 0.00000 0.00001 Changes Dissimilarity Size 0.01905 0.79982 0.00002 0.00315 0.22856 0.00000 0.06237 Changes Dissimilarity VCoverage 0.03097 0.61527 0.00019 0.00325 0.20696 0.00000 0.11829 Changes Faults Pairwise 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Changes Faults VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.00001 0.00000 Changes Pairwise Size 0.00003 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Changes Pairwise VCoverage 0.00477 0.00000 0.00000 0.00000 0.00000 0.00000 0.00017 Changes Size VCoverage 0.01376 0.96566 0.00012 0.00689 0.32113 0.00000 0.03763 Connectivity Dissimilarity Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.98465 0.00000 Connectivity Dissimilarity Size 0.11159 0.43498 0.00003 0.00130 0.06101 0.00000 0.36516 Connectivity Faults Pairwise 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Connectivity Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.09889 0.00000 Connectivity Faults VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.24235 0.00000 Connectivity Pairwise Size 0.00130 0.00000 0.00000 0.00000 0.00000 0.00000 0.00005 Connectivity Size VCoverage 0.00926 0.91238 0.00060 0.01164 0.55598 0.00000 0.02947 Dissimilarity Faults Pairwise 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dissimilarity Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.02229 0.00000 Dissimilarity Faults VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.18261 0.00000 Dissimilarity Pairwise Size 0.00011 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dissimilarity Size VCoverage 0.02470 0.71253 0.00000 0.00073 0.14291 0.00000 0.07575 Faults Pairwise Size 0.00001 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Faults Pairwise VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Faults Size VCoverage 0.00000 0.00000 0.00000 0.00000 0.00000 0.05021 0.00000 Pairwise Size VCoverage 0.00005 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 3 Table 6: Adjusted p-values for the comparison of m mono vs. multiobjective combinations of functional and non-functional objectives, obtained by applying the Holms correction procedure to the p-values of table 5. Connectivity Dissimilarity Pairwise VCoverage Changes Faults Size Changes Connectivity 1.0000 1.0000 0,00318 0.0000 1.0000 0.0000 1.0000 Changes Dissimilarity 1.0000 1.0000 0,20382 1.0000 1.0000 0.0000 1.0000 Changes Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes VCoverage 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 1.0000 Connectivity Faults 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Connectivity Size 1.0000 1.0000 0,00114 0.0000 1.0000 0.0000 1.0000 Dissimilarity Faults 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Dissimilarity Size 1.0000 1.0000 0,48026 1.0000 1.0000 0.0000 1.0000 Faults Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Faults VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Pairwise Size 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Size VCoverage 1.0000 1.0000 0,00216 0.0000 1.0000 0.0000 1.0000 Changes Connectivity Dissimilarity 1.0000 1.0000 0,04048 1.0000 1.0000 0.0000 1.0000 Changes Connectivity Faults 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Connectivity Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Connectivity Size 1.0000 1.0000 0,00114 0.0000 1.0000 0.0000 1.0000 Changes Connectivity VCoverage 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 1.0000 Changes Dissimilarity Faults 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Dissimilarity Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Dissimilarity Size 1.0000 1.0000 0,00216 0.0000 1.0000 0.0000 1.0000 Changes Dissimilarity VCoverage 1.0000 1.0000 0,01767 0.0000 1.0000 0.0000 1.0000 Changes Faults Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Faults VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Pairwise Size 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Pairwise VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Changes Size VCoverage 1.0000 1.0000 0,01176 1.0000 1.0000 0.0000 1.0000 Connectivity Dissimilarity Faults 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Connectivity Dissimilarity Size 1.0000 1.0000 0,00318 0.0000 1.0000 0.0000 1.0000 Connectivity Faults Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Connectivity Faults Size 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Connectivity Faults VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Connectivity Pairwise Size 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Connectivity Size VCoverage 1.0000 1.0000 0,0534 1.0000 1.0000 0.0000 1.0000 Dissimilarity Faults Pairwise 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Dissimilarity Faults Size 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Dissimilarity Faults VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Dissimilarity Pairwise Size 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Dissimilarity Size VCoverage 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 1.0000 Faults Pairwise Size 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Faults Pairwise VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Faults Size VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 Pairwise Size VCoverage 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 4 Table 7: Original p-values for the comparison of combinations of non-functional objectives vs. mixed combinations of functional and nonfunctional prioritization objectives. Changes Faults Changes Size Faults Size Changes Faults Size Changes Connectivity 0.00000 0.51186 0.00000 0.00000 Changes Dissimilarity 0.00000 0.14291 0.00000 0.00000 Changes Pairwise 0.00000 0.00094 0.00000 0.00000 Changes VCoverage 0.00000 0.87455 0.00000 0.00000 Connectivity Faults 0.19010 0.00000 0.16733 0.00382 Connectivity Size 0.00000 0.63563 0.00000 0.00000 Dissimilarity Faults 0.03305 0.00000 0.37342 0.00026 Dissimilarity Size 0.00000 0.09136 0.00000 0.00000 Faults Pairwise 0.00000 0.00000 0.00000 0.00000 Faults VCoverage 0.10390 0.00000 0.10288 0.00142 Pairwise Size 0.00000 0.00007 0.00000 0.00000 Size VCoverage 0.00000 0.50571 0.00000 0.00000 Changes Connectivity Dissimilarity 0.00000 0.13770 0.00000 0.00000 Changes Connectivity Faults 0.02534 0.00000 0.00000 0.28501 Changes Connectivity Pairwise 0.00000 0.00109 0.00000 0.00000 Changes Connectivity Size 0.00000 0.50571 0.00000 0.00000 Changes Connectivity VCoverage 0.00000 0.69123 0.00000 0.00000 Changes Dissimilarity Faults 0.26387 0.00000 0.00005 0.91238 Changes Dissimilarity Pairwise 0.00000 0.00021 0.00000 0.00000 Changes Dissimilarity Size 0.00000 0.28937 0.00000 0.00000 Changes Dissimilarity VCoverage 0.00000 0.24774 0.00000 0.00000 Changes Faults Pairwise 0.00000 0.00000 0.00000 0.00000 Changes Faults VCoverage 0.02408 0.00000 0.00000 0.36009 Changes Pairwise Size 0.00000 0.00003 0.00000 0.00000 Changes Pairwise VCoverage 0.00000 0.00305 0.00000 0.00000 Changes Size VCoverage 0.00000 0.16233 0.00000 0.00000 Connectivity Dissimilarity Faults 0.04338 0.00000 0.38384 0.00039 Connectivity Dissimilarity Size 0.00000 0.44636 0.00000 0.00000 Connectivity Faults Pairwise 0.00000 0.00000 0.00000 0.00000 Connectivity Faults Size 0.40727 0.00000 0.02146 0.02116 Connectivity Faults VCoverage 0.71253 0.00000 0.07907 0.09515 Connectivity Pairwise Size 0.00000 0.00065 0.00000 0.00000 Connectivity Size VCoverage 0.00000 0.17736 0.00000 0.00000 Dissimilarity Faults Pairwise 0.00000 0.00000 0.00000 0.00000 Dissimilarity Faults Size 0.89722 0.00000 0.00109 0.14028 Dissimilarity Faults VCoverage 0.67904 0.00000 0.04040 0.07272 Dissimilarity Pairwise Size 0.00000 0.00014 0.00000 0.00000 Dissimilarity Size VCoverage 0.00000 0.23994 0.00000 0.00000 Faults Pairwise Size 0.00000 0.00001 0.00000 0.00000 Faults Pairwise VCoverage 0.00000 0.00000 0.00000 0.00000 Faults Size VCoverage 0.98855 0.00000 0.00736 0.14827 Pairwise Size VCoverage 0.00000 0.00005 0.00000 0.00000 5 Table 8: Adjusted p-values for the comparison of combinations of non-functional objectives vs. mixed combinations of functional and nonfunctional prioritization objectives, obtained by applying the Holms correction procedure to the p-values of table 7. Changes Faults Changes Size Faults Size Changes Faults Size Changes Connectivity 0.000 0.000 0.000 0.000 Changes Dissimilarity 1.000 0.000 0.889 0.889 Changes Pairwise 1.000 0.000 1.000 1.000 Changes VCoverage 0.000 0.033 0.000 0.000 Connectivity Faults 0.000 1.000 0.000 0.000 Connectivity Size 0.000 0.000 0.000 0.000 Dissimilarity Faults 1.000 0.000 0.052 1.000 Dissimilarity Size 1.000 0.000 1.000 1.000 Faults Pairwise 0.000 0.008 0.000 0.000 Faults VCoverage 0.000 1.000 0.000 0.000 Pairwise Size 0.000 0.001 0.000 0.000 Size VCoverage 0.000 0.000 0.000 0.000 Changes Connectivity Dissimilarity 1.000 0.000 0.316 1.000 Changes Connectivity Faults 0.000 0.003 0.000 0.000 Changes Connectivity Pairwise 0.000 0.000 0.000 0.000 Changes Connectivity Size 1.000 0.000 0.889 0.889 Changes Connectivity VCoverage 1.000 0.000 1.000 1.000 Changes Dissimilarity Faults 0.000 0.033 0.000 0.000 Changes Dissimilarity Pairwise 0.000 1.000 0.000 0.000 Changes Dissimilarity Size 0.000 0.000 0.000 0.000 Changes Dissimilarity VCoverage 1.000 0.000 0.052 1.000 Changes Faults Pairwise 1.000 0.000 1.000 1.000 Changes Faults VCoverage 0.000 0.008 0.000 0.000 Changes Pairwise Size 0.000 1.000 0.000 0.000 Changes Pairwise VCoverage 0.000 0.001 0.000 0.000 Changes Size VCoverage 0.000 0.000 0.000 0.000 Connectivity Dissimilarity Faults 1.000 0.000 0.316 1.000 Connectivity Dissimilarity Size 0.000 0.003 0.000 0.000 Connectivity Faults Pairwise 0.000 0.000 0.000 0.000 Connectivity Faults Size 1.000 0.000 0.889 0.889 Connectivity Faults VCoverage 1.000 0.000 1.000 1.000 Connectivity Pairwise Size 0.000 0.033 0.000 0.000 Connectivity Size VCoverage 0.000 1.000 0.000 0.000 Dissimilarity Faults Pairwise 0.000 0.000 0.000 0.000 Dissimilarity Faults Size 1.000 0.000 0.052 1.000 Dissimilarity Faults VCoverage 1.000 0.000 1.000 1.000 Dissimilarity Pairwise Size 0.000 0.008 0.000 0.000 Dissimilarity Size VCoverage 0.000 1.000 0.000 0.000 Faults Pairwise Size 0.000 0.001 0.000 0.000 Faults Pairwise VCoverage 0.000 0.000 0.000 0.000 Faults Size VCoverage 1.000 0.000 0.316 1.000 Pairwise Size VCoverage 0.000 0.003 0.000 0.000 6 Table 9: Original p-values for the comparison of functional objectives vs. mixed combinations of functional and nonfunctional prioritization objectives Mixed Multi-Objective Functional Multi-Objective Con Pai Dis Pai Pai VCov Con Dis Dis VCov Con VCov Con Pai VCov Con Dis Pai Dis Pai VCov Con Dis VCov Chgs Con 0.00018 0.00000 0.00008 0.49352 0.56243 0.08590 0.00001 0.00010 0.00000 0.51186 Chgs Dis 0.00000 0.00000 0.00000 0.91238 0.75579 0.00981 0.00000 0.00000 0.00000 0.60186 Chgs Pai 0.21757 0.18686 0.28069 0.00000 0.00000 0.00754 0.85949 0.19339 0.94279 0.00000 Chgs VCov 0.00629 0.00000 0.00395 0.11602 0.10942 0.37543 0.00063 0.00557 0.00022 0.11159 Con Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Size 0.00012 0.00000 0.00002 0.33061 0.36009 0.09136 0.00001 0.00005 0.00000 0.45212 Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Size 0.00002 0.00000 0.00002 0.75579 0.50571 0.01415 0.00000 0.00001 0.00000 0.58858 Faults Pai 0.00020 0.36009 0.00012 0.00000 0.00000 0.00000 0.00574 0.00007 0.02347 0.00000 Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Pai Size 0.01666 0.89722 0.02599 0.00000 0.00000 0.00043 0.13770 0.01339 0.27219 0.00000 Size VCov 0.00041 0.00000 0.00020 0.37027 0.53685 0.10311 0.00003 0.00019 0.00001 0.53055 Chgs Con Dis 0.00000 0.00000 0.00000 0.96566 0.83700 0.01039 0.00000 0.00000 0.00000 0.73405 Chgs Con Faults 0.00000 0.00001 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Con Pai 0.14028 0.22485 0.22119 0.00000 0.00000 0.00689 0.70540 0.10728 0.87455 0.00000 Chgs Con Size 0.00010 0.00000 0.00003 0.37027 0.53055 0.08590 0.00000 0.00005 0.00000 0.49352 Chgs Con VCov 0.00008 0.00000 0.00005 0.29377 0.30272 0.10728 0.00002 0.00004 0.00000 0.41823 Chgs Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Dis Pai 0.04545 0.53685 0.04760 0.00000 0.00000 0.00087 0.29377 0.03253 0.48149 0.00000 Chgs Dis Size 0.00002 0.00000 0.00001 0.78506 0.98092 0.02875 0.00000 0.00001 0.00000 0.92757 Chgs Dis VCov 0.00002 0.00000 0.00000 0.75579 0.82208 0.04042 0.00000 0.00001 0.00000 0.75579 Chgs Faults Pai 0.00005 0.37027 0.00004 0.00000 0.00000 0.00000 0.00267 0.00002 0.01806 0.00000 Chgs Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Pai Size 0.00540 0.90480 0.00524 0.00000 0.00000 0.00007 0.06955 0.00250 0.14291 0.00000 Chgs Pai VCov 0.34026 0.06517 0.59521 0.00000 0.00000 0.01712 0.78506 0.33541 0.58858 0.00000 Chgs Size VCov 0.00000 0.00000 0.00000 0.87455 0.84449 0.01536 0.00000 0.00000 0.00000 0.95041 Con Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Dis Size 0.00022 0.00000 0.00009 0.47554 0.44636 0.07106 0.00001 0.00014 0.00000 0.47554 Con Faults Pai 0.00019 0.28501 0.00013 0.00000 0.00000 0.00000 0.00434 0.00010 0.02008 0.00000 Con Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Pai Size 0.07737 0.37027 0.13515 0.00000 0.00000 0.00295 0.48149 0.06806 0.67716 0.00000 Con Size VCov 0.00002 0.00000 0.00001 0.91997 0.63563 0.02172 0.00000 0.00001 0.00000 0.67716 Dis Faults Pai 0.00024 0.48149 0.00018 0.00000 0.00000 0.00000 0.00669 0.00012 0.03586 0.00000 Dis Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Pai Size 0.02347 0.54319 0.03673 0.00000 0.00000 0.00043 0.22119 0.01855 0.40186 0.00000 Dis Size VCov 0.00000 0.00000 0.00000 0.56243 0.82208 0.02008 0.00000 0.00000 0.00000 0.78506 Faults Pai Size 0.00031 0.35009 0.00012 0.00000 0.00000 0.00001 0.00370 0.00012 0.02008 0.00000 Faults Pai VCov 0.00017 0.47554 0.00010 0.00000 0.00000 0.00000 0.00250 0.00006 0.01855 0.00000 Faults Size VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Pai Size VCov 0.00610 0.85949 0.00800 0.00000 0.00000 0.00014 0.11379 0.00315 0.19010 0.00000 7 Table 10: Adjusted p-values for the comparison of functional objectives vs. mixed combinations of functional and nonfunctional prioritization objectives, obtained by applying the Holms correction procedure to the p-values of table 10. Mixed Multi-Objective Functional Multi-Objective Con Dis Con Pair Con VCov Dis Pair Dis VCov Pair VCov Con Dis Pair Con Dis VCov Con Pair VCov Dis Pair VCov Chgs Con 0.02772 0.00000 0.01360 1.00000 1.00000 1.00000 0.00201 0.01670 0.00000 1.00000 Chgs Dis 0.00000 0.00000 0.00000 1.00000 1.00000 1.00000 0.00000 0.00000 0.00000 1.00000 Chgs Pair 1.00000 1.00000 1.00000 0.00000 0.00000 0.92742 1.00000 1.00000 1.00000 0.00000 Chgs VCov 0.79254 0.00000 0.52535 1.00000 1.00000 1.00000 0.08883 0.71853 0.03256 1.00000 Con Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Size 0.01956 0.00000 0.00378 1.00000 1.00000 1.00000 0.00201 0.00885 0.00000 1.00000 Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Size 0.00378 0.00000 0.00378 1.00000 1.00000 1.00000 0.00000 0.00201 0.00000 1.00000 Faults Pair 0.03000 1.00000 0.01956 0.00000 0.00000 0.00000 0.73472 0.01204 1.00000 0.00000 Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Pair Size 1.00000 1.00000 1.00000 0.00000 0.00000 0.06149 1.00000 1.00000 1.00000 0.00000 Size VCov 0.05904 0.00000 0.03000 1.00000 1.00000 1.00000 0.00543 0.02888 0.00201 1.00000 Chgs Con Dis 0.00000 0.00000 0.00000 1.00000 1.00000 1.00000 0.00000 0.00000 0.00000 1.00000 Chgs Con Faults 0.00000 0.00201 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Con Pair 1.00000 1.00000 1.00000 0.00000 0.00000 0.85436 1.00000 1.00000 1.00000 0.00000 Chgs Con Size 0.01670 0.00000 0.00543 1.00000 1.00000 1.00000 0.00000 0.00885 0.00000 1.00000 Chgs Con VCov 0.01360 0.00000 0.00885 1.00000 1.00000 1.00000 0.00378 0.00716 0.00000 1.00000 Chgs Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Dis Pair 1.00000 1.00000 1.00000 0.00000 0.00000 0.12180 1.00000 1.00000 1.00000 0.00000 Chgs Dis Size 0.00378 0.00000 0.00201 1.00000 1.00000 1.00000 0.00000 0.00201 0.00000 1.00000 Chgs Dis VCov 0.00378 0.00000 0.00000 1.00000 1.00000 1.00000 0.00000 0.00201 0.00000 1.00000 Chgs Faults Pair 0.00885 1.00000 0.00716 0.00000 0.00000 0.00000 0.36579 0.00378 1.00000 0.00000 Chgs Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Chgs Pair Size 0.70200 1.00000 0.68644 0.00000 0.00000 0.01204 1.00000 0.34750 1.00000 0.00000 Chgs Pair VCov 1.00000 1.00000 1.00000 0.00000 0.00000 1.00000 1.00000 1.00000 1.00000 0.00000 Chgs Size VCov 0.00000 0.00000 0.00000 1.00000 1.00000 1.00000 0.00000 0.00000 0.00000 1.00000 Con Dis Faults 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Dis Size 0.03256 0.00000 0.01512 1.00000 1.00000 1.00000 0.00201 0.02198 0.00000 1.00000 Con Faults Pair 0.02888 1.00000 0.02054 0.00000 0.00000 0.00000 0.57288 0.01670 1.00000 0.00000 Con Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Con Pair Size 1.00000 1.00000 1.00000 0.00000 0.00000 0.40120 1.00000 1.00000 1.00000 0.00000 Con Size VCov 0.00378 0.00000 0.00201 1.00000 1.00000 1.00000 0.00000 0.00201 0.00000 1.00000 Dis Faults Pair 0.03504 1.00000 0.02772 0.00000 0.00000 0.00000 0.83625 0.01956 1.00000 0.00000 Dis Faults Size 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Faults VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Dis Pair Size 1.00000 1.00000 1.00000 0.00000 0.00000 0.06149 1.00000 1.00000 1.00000 0.00000 Dis Size VCov 0.00000 0.00000 0.00000 1.00000 1.00000 1.00000 0.00000 0.00000 0.00000 1.00000 Faults Pair Size 0.04495 1.00000 0.01956 0.00000 0.00000 0.00201 0.49580 0.01956 1.00000 0.00000 Faults Pair VCov 0.02635 1.00000 0.01670 0.00000 0.00000 0.00000 0.34750 0.01038 1.00000 0.00000 Faults Size VCov 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 Pair Size VCov 0.77470 1.00000 0.97600 0.00000 0.00000 0.02198 1.00000 0.42525 1.00000 0.00000 8 4 Experiment 4 Table 11: Original p-values for the comparison of for functional vs. nonfunctional prioritization objectives Changes Faults Size Faults Size Changes Faults Changes Size Changes Faults Size Connectivity 0.00084 0.00000 0.54957 0.00000 0.00000 0.52428 0.00000 Dissimilarity 0.40186 0.00000 0.06101 0.00000 0.00000 0.18686 0.00000 Pairwise 0.00669 0.00000 0.00000 0.00000 0.00000 0.00009 0.00000 VCoverage 0.08769 0.00000 0.00001 0.00000 0.00000 0.00058 0.00000 Connectivity Pairwise 0.00000 0.00000 0.00462 0.00000 0.00000 0.02408 0.00000 Dissimilarity Pairwise 0.00000 0.00000 0.00000 0.00000 0.00000 0.00003 0.00000 Pairwise VCoverage 0.00000 0.00000 0.00204 0.00000 0.00000 0.01415 0.00000 Connectivity Dissimilarity 0.45791 0.00000 0.05708 0.00000 0.00000 0.18686 0.00000 Dissimilarity VCoverage 0.27642 0.00000 0.09136 0.00000 0.00000 0.25977 0.00000 Connectivity VCoverage 0.00098 0.00000 0.45791 0.00000 0.00000 0.53685 0.00000 Connectivity Pairwise VCoverage 0.00000 0.00000 0.00052 0.00000 0.00000 0.00358 0.00000 Connectivity Dissimilarity Pairwise 0.00000 0.00000 0.00382 0.00000 0.00000 0.02061 0.00000 Dissimilarity Pairwise VCoverage 0.00000 0.00000 0.00016 0.00000 0.00000 0.00150 0.00000 Connectivity Dissimilarity VCoverage 0.31184 0.00000 0.06376 0.00000 0.00000 0.18048 0.00000 9