UEQ KPI Value Range based on the UEQ Benchmark

UEQ KPI Value Range based on the UEQ Benchmark Andreas Hinderks University of Seville Martin Schrepp SAP SE Francisco José Domínguez Mayo University of Seville María José Escalona Cuaresma University of Seville Jörg Thomaschewski University of Applied Science Emden/Leer December 2018 Contents Contents 1 Introduction 2 1.1 UEQKPI........................................... 2 1.2 ResearchMethodology.................................... 3 2 UEQ KPI Value Range 5 2.1 UEQ KPI Value Range - Benchmark Category Bad . . . . . . . . . . . . . . . . . . . . 6 2.2 UEQ KPI Value Range - Benchmark Category Below Average . . . . . . . . . . . . . . 6 2.3 UEQ KPI Value Range - Benchmark Category Above Average . . . . . . . . . . . . . . 7 2.4 UEQ KPI Value Range - Benchmark Category Good . . . . . . . . . . . . . . . . . . . 7 2.5 UEQ KPI Value Range - Benchmark Category Excellent . . . . . . . . . . . . . . . . . 8 List of Figures 9 List of Tables 10 Bibliography 11 1 Introduction 1 Introduction The formula described here is to combine for every participant the value and assessed importance of each UEQ scale. Therefore, we calculated for each participant the relative importance of each scale and multiplied this with the value of the UEQ scale. This allows to generate a meaningful UX KPI based on the UEQ, that we have named UEQ KPI. The steps are described in detail below. The method presented in this paper is based on the User Experience Questionnaire (UEQ) [Laugwitz et al. 2008] and its benchmark [Schrepp 2017a]. The objective of the UEQ is to allow a quick assessment done by end users covering a preferably comprehensive impression of user experience. It should allow the users to express feelings, impressions, and attitudes that arise when experiencing the product under investigation in a very simple and immediate way. It consists of 26 items that are grouped into 6 scales (Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation and Novelty). Each scale represents a distinct UX quality aspect. In this paper, we describe an approach to calculate the possible value range of the UEQ KPI [Schrepp 2017b]. 1.1 UEQ KPI The central idea of our approach is to combine for every participant the value and perceived importance of each UEQ scale [Schrepp 2017a]. Therefore, we calculated for each participant the relative importance of each scale and multiplied this with the value of the UEQ scale. Finally, we have generated a UX KPI based on the UEQ. We have named the resulting value UEQ KPI. The steps are described in detail below. •The result of the UEQ represents a value per participant Paiand for every scale: attractiveness Ai, perspicuity Pi, efficiency Ei, dependability Di, stimulation Si, and novelty Ni. •The result of the six additional questions is a value of the perceived importance of each scale: attractiveness ai, perspicuity pi, efficiency ei, dependability di, stimulation si, and novelty ni. •We have calculated the relative importance per participant using the given perceived importance of each scale. For example for attractiveness Awi=ai/(ai+pi+ei+di+si+ni). •The UEQ KPI per participant can be calculated by multiplying the mean value of the UEQ scale with its relative importance: UEQ KP Ii=Awi∗Ai+Pwi∗Pi+Ewi∗Ei+Dwi∗Di+ 2 Introduction Swi∗Si+Nwi∗Ni. The UEQ KPI has been calculated using the following formula: UEQ KPI =1 n n X i=1 (Awi∗Ai+Pwi∗Pi+Ewi∗Ei+Dwi∗Di+Swi∗Si+Nwi∗Ni)(1.1) We have used this formula to calculate the possible value range of the UEQ KPI with a combination of the UEQ Benchmark and theoretical values. 1.2 Research Methodology The formula for calculating the UEQ KPI consists of two parts for each factor of the UEQ: The first is the value of the UEQ factor and the second is the perceived importance per UEQ factor. The UEQ KPI is calculated from both values. In order to determine the value range of the UEQ KPI, both values should be considered. Theoretically, the two values can have the following ranges: •UEQ factor: Between -3 and +3 •Perceived importance: Between 1 and 7 If the values are calculated with the UEQ KPI formula, this results is in a value range for the UEQ KPI between -3 and +3. This is the theoretically assumed value range of the UEQ KPI. However, this value range is not very helpful. Another way to calculate the possible value range is to include the UEQ benchmark. A benchmark already exists for the UEQ, which specifies a practical value range for the factors of the UEQ [Schrepp 2017a]. The results of an evaluation of a product can thus be better interpreted. The figure 1.1 shows the benchmark categories for each factor. Figure 1.1: UEQ Benchmark 3 Introduction From the UEQ benchmark, the mean values per factor can be extracted, as are given in table 1.1. ATT PER EFF DEP STI NOV Bad -0.155 -0.185 -0.235 -0.115 -0.255 -0.355 Below Avarage 0.930 0.855 0.755 0.955 0.740 0.500 Above Avarage 1.340 1.315 1.220 1.305 1.145 0.875 Good 1.630 1.725 1.620 1.560 1.425 1.220 Excellent 2.125 2.200 2.140 2.075 2.025 1.950 Table 1.1: UEQ Benchmark - Mean Value for each Factor (ATT-Attractiveness , PER-Perspicuity, EFF-Efficiency, DEP-Dependability, STI-Stimunlation, NOV-Novelty) We use these mean values of the benchmark to calculate the possible range of values of the UEQ KPI with practical data. In this way, the value range of the UEQ KPI can be calculated on the basis of the following data: •UEQ factor: Mean values per UEQ factor from the benchmark categories (Bad, Below Avarage, Above Avarage, Good, Excellent) •Perceived importance: Between 1 and 7 The UEQ KPI was calculated for all benchmark categories. All possible combinations of all benchmark categories and for each perceived importance factor (1 to 7 per UEQ factor) were calculated. Chapter 2 presents the results of the calculation. 4 UEQ KPI Value Range 2 UEQ KPI Value Range From the UEQ KPI formula (see Section 1.1) and the ranges of values (see Section 1.2), the following mean values and the minimum and maximum values for each benchmark categorie result: UEQ Benchmark UEQ KPI Mean Value UEQ KPI Min Value UEQ KPI Max Value Bad -0.224 -0.286 -0.162 Below Avarage 0.768 0.645 0.891 Above Avarage 1.164 1.038 1.290 Good 1.502 1.375 1.628 Excellent 2.080 2.018 2.143 Table 2.1: UEQ KPI Value Range - Overview The figure below shows the table (see table 2.1) graphically. Figure 2.1: UEQ KPI Value Range for each Benchmark Range This gives the possible range of values for the UEQ KPI based on the UEQ benchmark of: UEQ KPI Value Range between -0.286 and 2.143 The value range for the perceived importance per UEQ factor should be recorded in an additional benchmark, so that the value range of the UEQ KPI can be interpreted in an even better manner. 5 UEQ KPI Value Range Although a theoretical value range from -3 to +3 is possible, it will not occur in a practical application. Rather, it shows that the real value range is smaller and more positive. 2.1 UEQ KPI Value Range - Benchmark Category Bad Possible range of values for the UEQ KPI based on the UEQ benchmark category bad. The UEQ KPI value range for the UEQ benchmark category bad is between -0.286 and -0.126 with -0.224 as the mean value. Figure 2.2: UEQ KPI Value Range - Benchmark Category Bad 2.2 UEQ KPI Value Range - Benchmark Category Below Average Possible range of values for the UEQ KPI based on the UEQ benchmark category Below Average. The UEQ KPI value range for the UEQ benchmark category Below Average is between 0.645 and 0.891 with 0.768 as the mean value. 6 UEQ KPI Value Range Figure 2.3: UEQ KPI Value Range - Benchmark Category Below Average 2.3 UEQ KPI Value Range - Benchmark Category Above Average Possible range of values for the UEQ KPI based on the UEQ benchmark category Above Average. The UEQ KPI value range for the UEQ benchmark category Above Average is between 1.038 and 1.290 with 1.164 as the mean value. Figure 2.4: UEQ KPI Value Range - Benchmark Category Above Average 2.4 UEQ KPI Value Range - Benchmark Category Good Possible range of values for the UEQ KPI based on the UEQ benchmark category Good. The UEQ KPI value range for the UEQ benchmark category Good is between 1.375 and 1.628 with 1.502 as the mean value. 7 UEQ KPI Value Range Figure 2.5: UEQ KPI Value Range - Benchmark Category Good 2.5 UEQ KPI Value Range - Benchmark Category Excellent Possible range of values for the UEQ KPI based on the UEQ benchmark category Excellent. The UEQ KPI value range for the UEQ benchmark category Excellent is between 2.018 and 2.143 with 2.080 as the mean value. Figure 2.6: UEQ KPI Value Range - Benchmark Category Excellent 8