REBECCA – D7.5: REBECCA system evaluation and acceptance by patients and health professionals

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 965231. Horizon 2020 EUROPEAN COMMISSION Directorate-General for Research and Innovation People, Combating Diseases D7.5 – REBECCA system evaluation and acceptance by patients and health professionals Project Acronym: REBECCA Grant Agreement No: 965231 Project Full Title: Research on breast cancer induced chronic conditions supported by causal analysis of multi-source data Starting Date: 1/4/ 2021 Duration in Months: 54 H2020SC1-DTH-12-2020 D7.5 2 | 93 Deliverable version 1.0 Date 25 September 2025 Nature of deliverable Report Dissemination level Public Work Package WP7 – Improving research and clinical workflows using multi-source RWD Lead Beneficiary KI Author(s) / Contributor(s) Alkyoni Glympi (KI), Ioannis Ioakimidis (KI) Status Submitted H2020SC1-DTH-12-2020 D7.5 3 | 93 Document History Version1 Issue Date Status2 Content and Changes 0.1 18 September 2025 Draft First draft 1.0 25 September 2025 Submitted Reviewed internally. Final version 1 Please use a new number for each new version of the deliverable. Use “0.#” for Draft and PeerReviewed. “x.#” for Submitted and Approved”, where x>=1.Add the date when this version was issued and list the items that have been added or changed. 2 A deliverable can be in one of these stages: Draft, Peer-Reviewed, Submitted and Approved. H2020SC1-DTH-12-2020 D7.5 4 | 93 Table of Contents Document History ..................................................................................................................... 3 Table of figures ........................................................................................................................... 6 Abbreviations and Acronyms ................................................................................................ 9 Executive Summary ................................................................................................................. 10 1 Introduction ....................................................................................................................... 11 1.1 Aim ................................................................................................................................................................. 12 1.2 Intended audience ................................................................................................................................... 12 1.3 Project partners and acronyms ........................................................................................................... 12 1.4 Methodology ............................................................................................................................................. 12 1.5 Organization of the document ........................................................................................................... 14 2 System evaluation tools ................................................................................................ 15 2.1 Evaluation framework ............................................................................................................................. 15 2.1.1 The User Experience Questionnaire ............................................................................................. 15 2.1.2 The System Usability Scale .............................................................................................................. 16 2.1.3 The Technology Acceptance Model ............................................................................................ 17 2.2 Evaluation forms ....................................................................................................................................... 19 3 System evaluation from WP5 ..................................................................................... 20 3.1 Spain – REBECCA-OST ............................................................................................................................ 20 3.1.1 Long UEQ Results at 6 months ...................................................................................................... 20 3.1.2 Long UEQ Results at 12 months ................................................................................................... 22 3.2 Norway – REBECCA-CRF ........................................................................................................................ 26 3.2.1 BCP evaluation of the pApp ............................................................................................................ 26 3.2.2 HC evaluation of the pApp ............................................................................................................. 31 3.2.3 BCP vs HC Evaluation ........................................................................................................................ 35 4 System evaluation from WP6 ..................................................................................... 40 4.1 Spain - REBECCA-INCLIVA-Inter-QoL .............................................................................................. 40 4.1.1 BCP evaluation of the pApp ............................................................................................................ 40 4.1.2 HCP evaluation of the system ........................................................................................................ 44 4.2 Norway - REBECCA-SUH-Inter-QoL .................................................................................................. 47 4.2.1 BCP evaluation of the pApp ............................................................................................................ 47 4.2.2 HCP evaluation of the system ........................................................................................................ 52 4.3 Sweden - REBECCA-KI-Feas-QoL ....................................................................................................... 54 4.3.1 Evaluation of the pApp – UEQ-S ................................................................................................... 54 4.3.2 Evaluation of the pApp – SUS Questionnaire .......................................................................... 56 4.3.3 Evaluation of the consultation ....................................................................................................... 56 5 System evaluation from WP7 ..................................................................................... 59 5.1 Norway - REBECCA-PROST .................................................................................................................. 59 5.1.1 Evaluation of the pApp – UEQ-S ................................................................................................... 59 5.1.2 Evaluation of the pApp – SUS Questionnaire .......................................................................... 60 H2020SC1-DTH-12-2020 D7.5 5 | 93 5.1.3 Additional questions .......................................................................................................................... 61 6 Result summary – internal evaluation ..................................................................... 63 6.1 UEQ results ................................................................................................................................................. 63 6.2 SUS results .................................................................................................................................................. 64 6.3 Additional questions results ................................................................................................................ 64 7 REBECCA final stakeholder workshop – external evaluation .......................... 66 7.1 Workshop’s key insights ........................................................................................................................ 66 7.2 Results of the External Expert System Evaluation ........................................................................ 68 Conclusions ................................................................................................................................ 70 References .................................................................................................................................. 71 Annex ............................................................................................................................................ 72 A. Evaluation forms .............................................................................................................. 73 B. REBECCA final stakeholder workshop ..................................................................... 88 H2020SC1-DTH-12-2020 D7.5 6 | 93 Table of figures Figure 1. UEQ scales and an illustrative example of the aggregated results across the six outcome scales. .................................................................................................................................. 15 Figure 2. UEQ-Short item structure and an illustrative example for a benchmark comparison of pragmatic, hedonic, and overall quality ........................................................... 16 Figure 3. SUS Items and Scoring Interpretation for Assessing System Usability ............ 17 Figure 4. Modified Technology Acceptance Model ................................................................... 19 Figure 5. An example of the integrated version of the patient evaluation for WP6, created in REDCap ................................................................................................................................... 19 Figure 6. System evaluation WP5 - Spain ...................................................................................... 20 Figure 7. Flow chart of responses analysed at 6 months – Spain, WP5 ............................. 20 Figure 8. Visual Representation of Mean Long UEQ Item Scores – Spain, BCP, WP5, n=48, 6 months ........................................................................................................................................ 22 Figure 9. Flow chart of responses analysed at 12 months – Spain, BCP WP5 ................. 23 Figure 10. Visual Representation of Mean Long UEQ Item Scores - Spain, BCP, WP5, n=52, 12 months ...................................................................................................................................... 24 Figure 11. System evaluation WP5 - Norway ................................................................................ 26 Figure 12. Visual Representation of Mean UEQ Item Scores – Norway, BCP WP5, n=30 ........................................................................................................................................................................ 27 Figure 13. Mean SUS score – Norway, WP5 BCP, n=19 ............................................................ 28 Figure 14. Perceived potential of the pApp to enhance QoL after primary treatment, Norway, BCP WP5, n=28 ....................................................................................................................... 29 Figure 15. Feelings about privacy - Norway, BCP WP5, n=27 ................................................ 29 Figure 16. Perception of being monitored - Norway, BCP WP5, n=27 .............................. 30 Figure 17. Overall satisfaction - Norway, BCP WP5, n=27 ....................................................... 30 Figure 18. Visual Representation of Mean UEQ Item Scores – Norway, HC WP5, n=36 ........................................................................................................................................................................ 32 Figure 19. Mean SUS score – Norway, HC, WP5 n=32 .............................................................. 32 Figure 20. Perceived potential of the pApp to enhance QoL after primary treatment, Norway, HC WP5, n=37 ......................................................................................................................... 33 Figure 21. Feelings about privacy - Norway, HC WP5, n=36 .................................................. 33 Figure 22. Perception of being monitored - Norway, HC WP5, n=37 ................................ 34 Figure 23. Overall satisfaction - Norway, HC WP5, n=35 ......................................................... 35 Figure 24. UEQ mean values BCP vs HC, WP5, Norway ............................................................ 35 Figure 25. SUS score BCP vs HC, WP5, Norway ........................................................................... 36 Figure 26. QoL Improvement, BCP vs HC, WP5, Norway ......................................................... 37 Figure 27. Feelings about privacy, BCP vs HC, WP5, Norway ................................................. 37 Figure 28. Felt monitored, BCP vs HC, WP5, Norway ................................................................ 38 H2020SC1-DTH-12-2020 D7.5 7 | 93 Figure 29. Overall Satisfaction, BCP vs HC, WP5, Norway ....................................................... 38 Figure 30. System evaluation WP6 - Spain .................................................................................... 40 Figure 31. Visual Representation of Mean UEQ Item Scores - Spain, BCP WP6, n=15 41 Figure 32. Mean SUS score – Spain, BCP WP6, n=15 ............................................................... 42 Figure 33. Perceived QoL improvement after primary treatment, Spain, BCP WP6, n= 15 ................................................................................................................................................................... 42 Figure 34. Feelings about privacy, Spain, BCP WP6, n= 15 ..................................................... 43 Figure 35. Felt monitored. Spain, BCP WP6, n= 13 .................................................................... 43 Figure 36. Overall satisfaction, Spain, BCP WP6, n= 15 ............................................................ 44 Figure 37. Visual Representation of Mean UEQ Item Scores - Spain, HCP WP6, n=5 .. 45 Figure 38. Mean SUS score – Spain, HCP WP6 n=5 ................................................................... 45 Figure 39. System evaluation WP6 - Norway ................................................................................ 47 Figure 40. Visual Representation of Mean UEQ Item Scores - Norway, BCP WP6, n=28 ........................................................................................................................................................................ 48 Figure 41. Mean SUS score – Norway, BCP, WP6 n=23 .......................................................... 49 Figure 42. Perceived QoL improvement after primary treatment, Norway, BCP WP6, n= 26 ................................................................................................................................................................... 49 Figure 43. Feelings about privacy, Norway, BCP WP6, n= 28 ................................................ 50 Figure 44. Felt monitored. Norway, BCP WP6, n= 28 ................................................................ 51 Figure 45. Overall satisfaction. Norway, BCP WP6, n= 26 ....................................................... 51 Figure 46. Visual Representation of Mean UEQ Item Scores – Norway, HCP WP6, n=3 ........................................................................................................................................................................ 52 Figure 47. Mean SUS score – Norway, HCP, WP6 n=3 .............................................................. 53 Figure 48.Visual Representation of Mean UEQ Item Scores - Sweden, BCS, WP6, n=53 ........................................................................................................................................................................ 55 Figure 49. SUS Results - Sweden, BCS, WP6, n=51 .................................................................... 56 Figure 50. Trust in the guidanceSweden, BCS, WP6, n=53 .................................................. 57 Figure 51. Recommendations' alignment with participants’ needsSweden, BCS, WP6, n=53 .............................................................................................................................................................. 57 Figure 52. Perceived improvement in QoLSweden, BCS, WP6, n=53 .............................. 58 Figure 53. Visual Representation of Mean UEQ Item Scores - Norway, PCP WP7, n=20 ........................................................................................................................................................................ 60 Figure 54. Mean SUS score – Norway, PCP, WP7, n=23 ........................................................... 60 Figure 55. Perceived QoL improvement after primary treatment, Norway, PCP WP7, n= 21 ................................................................................................................................................................... 61 Figure 56. Feelings about privacy, Norway, PCP WP7, n= 21 ................................................. 61 Figure 57. Felt monitored. Norway, PCP WP7, n= 21 ................................................................ 62 Figure 58. Overall satisfaction. Norway, PCP, WP7, n= 18 ....................................................... 62 Figure 59. Distribution of stakeholder groups in the final workshop, n=15 .................... 68 H2020SC1-DTH-12-2020 D7.5 8 | 93 Figure 60. Average rating per stakeholder group, n=15 .......................................................... 69 H2020SC1-DTH-12-2020 D7.5 9 | 93 Abbreviations and Acronyms BCP Breast Cancer Patient BCS Breast Cancer Survivor EHR Electronic Health Records HC Healthy Controls HCP Health Care Professionals KI Karolinska Institutet pApp REBECCA Patient App PCP Prostate Cancer Patients PEOU Perceived Ease of Use PS Perceived Security PT Perceived Trust PU Perceived Usefulness QoL Quality of Life RWD Real-World Data SUS System Usability Scale TAM Technology Acceptance Model UEQ User Experience Questionnaire UEQ-S User Experience Questionnaire – Short version WP Work Package H2020SC1-DTH-12-2020 D7.5 16|93 A short version of the UEQ (UEQ-S) is also available, containing 8 items that cover the same six quality dimensions while reducing respondent burden, making it particularly suitable for studies where questionnaire length must be minimized. Figure 2. UEQ-Short item structure and an illustrative example for a benchmark comparison of pragmatic, hedonic, and overall quality In both the full and short versions, responses are given on a 7-point semantic differential scale, with opposing adjectives placed at each end (e.g., “complicated” vs. “easy”). Scores for each scale are calculated by averaging the corresponding item ratings, resulting in values ranging from -3 (most negative) to +3 (most positive). According to the UEQ benchmark, values: • above +0.8 are considered positive • values between -0.8 and +0.8 are neutral, and • values below -0.8 are negative. The analysis also allows for comparison against an international benchmark database, classifying results into categories such as “excellent,” “good,” “above average,” “below average,” or “poor” for each dimension. 2.1.2 The System Usability Scale The SUS is a widely used, reliable tool for assessing the perceived usability of a system. It consists of 10 statements rated on a 5-point Likert scale, ranging from “Strongly disagree” to “Strongly agree,” with items alternating between positive and negative phrasing to minimize response bias. To calculate the SUS score, responses are converted to numerical values (0–4), adjusted for positive or negative items, summed, and multiplied by 2.5 to yield a score ranging from 0 to 100. While not a percentage, this score reflects overall usability, with 68 considered the benchmark for “average” usability. Scores above 68 are generally interpreted as above average, whereas scores below this threshold may indicate usability issues. In addition to absolute values, results can be interpreted using adjective ratings such as “good,” “excellent,” “poor” (Figure 3). H2020SC1-DTH-12-2020 D7.5 17|93 Figure 3. SUS Items and Scoring Interpretation for Assessing System Usability 2.1.3 The Technology Acceptance Model TAM was first conceptualized by Fred Davis in the 1980s and stands as a pivotal tool in elucidating how individuals make decisions regarding the adoption of new technology. TAM originated from the domain of information systems research, specifically emerging in the late 1980s and early 1990s, as it was initially developed by Fred Davis in 1986 and was later extended by Davis in collaboration with Richard Bagozzi in 1989. TAM aimed to provide insights into the factors influencing individuals’ acceptance and adoption of new information technologies, especially concerning the potential adoption of new technology-based systems within existing operational frameworks (i.e., close to the goal of the REBECCA system evaluation).5 The foundational premise of TAM is that perceived usefulness (PU) and perceived ease of use (PEOU) are critical determinants of an individual’s intention to use a technology, which, in turn, influences the actual usage. PU refers to the user’s belief that a particular technology will enhance one’s performance, productivity, or enjoyment while performing a technology-assisted task. On the other hand, PEOU refers to the user’s perception about the ease of the technology use. Currently TAM and various adapted versions of it relevant to specific deployment requirements, much like in the case of REBECCA is widely used in academia, industry, in policymaking and as well as in the health domain. The appropriate uses of TAM, in general, include user-centric evaluations for understanding and predicting user behaviour in adopting a wide range of technologies, from mobile applications to social media platforms and artificial 5 Marangunić N, Granić A. Technology acceptance model: a literature review from 1986 to 2013. Universal access in the information society. 2015 Mar;14:81-95. H2020SC1-DTH-12-2020 D7.5 18|93 intelligence.1 Similarly, in the domain of Health, TAM has previously been used for investigating the acceptance of various technologies, such as electronic health records (EHRs), remote sensing, wearable health devices, and mobile health applications, both by patients and by involved health professionals.6 In order to better fit the needs of the REBECCA project, a modified version of TAM7 is used (Figure 4), proposed for health-related users’ acceptance of m-health services. This modified TAM incorporates key factors including Perceived Usefulness, Perceived Ease of Use, Perceived Security, and Perceived Trust. These dimensions were chosen to comprehensively assess the medical experts’ and patients’ viewpoints on the REBECCA system. In this adaptation of the TAM evaluation the term “Perceived Usefulness” delves into the extent to which users believe that the REBECCA system aids them in achieving their objectives, such as enhancing QoL (patients), decision making process (medical experts) and provided treatment (both). It essentially gauges the practical benefits perceived by the users. Additionally, the term “Perceived Ease of Use” focuses on the degree to which individuals believe that using the system would be effortless and straightforward. This aspect investigates the system’s user-friendliness, which is crucial for widespread acceptance and adoption. Furthermore, “Perceived Security” assesses the users’ confidence in the system’s ability to safeguard their sensitive information and maintain their privacy, a paramount concept in the field of medical technology, as patients’ data security is critical in practice, and so is the perception of personal data security by the patients themselves. Finally, the “Perceived Trust” evaluation component delves into the users’ confidence and faith in the reliability, integrity, and competence of the REBECCA system to perform the described/communicated goals. The perceived trust that the user experiences during the use of the system is a fundamental component in cultivating and sustaining a positive user experience, fostering long-term relationships, and encouraging sustained and long-term usage of the technology. 6 AlQudah AA, Al-Emran M, Shaalan K. Technology acceptance in healthcare: A systematic review. Applied Sciences. 2021 Nov 9;11(22):10537. 7 Alloghani M, Hussain A, Al-Jumeily D, Abuelma'atti O. Technology Acceptance Model for the Use of M-Health Services among health related users in UAE. In2015 International Conference on Developments of E-Systems Engineering (DeSE) 2015 Dec 13 (pp. 213-217). IEEE. H2020SC1-DTH-12-2020 D7.5 19|93 2.2 Evaluation forms The internal assessment of the final iteration of the REBECCA system was conducted via evaluation forms from the clinical studies in Norway and Spain, as well as from the feasibility study in Sweden. As mentioned above, the evaluation forms consisted of three sections: the UEQ, the SUS, and additional project-specific questions based on the TAM (Figure 5). They were electronically created and distributed through REDCap, translated into the respective national languages, and administered to patients, HC at the study exit, and to HCP at the end of the study. The complete evaluation forms are provided in Annex A. Figure 5. An example of the integrated version of the patient evaluation for WP6, created in REDCap Figure 4. Modified Technology Acceptance Model H2020SC1-DTH-12-2020 D7.5 20|93 3 System evaluation from WP5 3.1 Spain – REBECCA-OST During the observational study in Spain, the system was evaluated through the UEQ long form, and the evaluation was conducted at two time points: 6 months and 12 months (Figure 6). Figure 6. System evaluation WP5 - Spain 3.1.1 Long UEQ Results at 6 months At 6 months after study’s onset, seventy-three BCP were given the Long UEQ form and from those 48 were included in the analysis (Figure 7). *Inconsistences: Each scale (group of questions) should measure the same thing. If a person gives very different answers within the same scale (for example, both very positive and very negative), this is a sign the answers might not be reliable. If this happened on three or more scales, that person’s responses are considered inconsistent and are removed from the analysis. *Critical length: Identifies cases where participants provide the same response to a large number of items, which may indicate disengagement (e.g., choosing the middle category for nearly all items). Based on Figure 7. Flow chart of responses analysed at 6 months – Spain, WP5 H2020SC1-DTH-12-2020 D7.5 21|93 empirical findings across several UEQ datasets, responses in which more than 15 items have identical ratings are flagged under this criterion and also excluded from the analysis. Table 1 below presents the results of the Long UEQ used to evaluate the system at 6 months. It shows the mean value, variance and standard deviation for each of the 26 scales rated. The "Left" and "Right" columns show the two extremes of the scale (e.g., annoying vs. enjoyable) participants used to evaluate each aspect of the app. These scales are further grouped into the six overarching dimensions (shown with different colours on the right side of the table): Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty. Table 1. Overview of UEQ Ratings at 6 months – Spain, BCP, WP5, n=48, 6 months Item Mean Variance Std. Dev. No. Left Right Scale 1 2.0 1.4 1.2 48 annoying enjoyable Attractiveness 2 -2.5 1.2 1.1 48 not understandable understandable Perspicuity 3 1.1 2.3 1.5 47 creative dull Novelty 4 2.6 0.8 0.9 48 easy to learn difficult to learn Perspicuity 5 2.0 1.7 1.3 44 valuable inferior Stimulation 6 0.5 1.4 1.2 44 boring exciting Stimulation 7 1.5 1.8 1.4 44 not interesting interesting Stimulation 8 1.2 2.2 1.5 44 unpredictable predictable Dependability 9 1.5 2.7 1.6 44 fast slow Efficiency 10 -0.4 2.3 1.5 44 inventive conventional Novelty 11 1.8 1.3 1.1 45 obstructive supportive Dependability 12 1.7 2.2 1.5 45 good bad Attractiveness 13 2.4 1.4 1.2 45 complicated easy Perspicuity 14 2.0 1.0 1.0 45 unlikable pleasing Attractiveness 15 -0.3 2.7 1.7 44 usual leading edge Novelty 16 2.0 1.2 1.1 45 unpleasant pleasant Attractiveness 17 2.1 1.5 1.2 45 secure not secure Dependability 18 1.4 2.0 1.4 46 motivating demotivating Stimulation 19 1.7 1.4 1.2 46 meets expectations does not meet expectations Dependability 20 1.8 1.2 1.1 46 inefficient efficient Efficiency 21 2.4 1.4 1.2 46 clear confusing Perspicuity 22 1.8 1.9 1.4 46 impractical practical Efficiency 23 2.2 1.8 1.3 46 organized cluttered Efficiency 24 1.2 1.9 1.4 45 attractive unattractive Attractiveness 25 2.0 2.0 1.4 45 friendly unfriendly Attractiveness 26 0.3 1.8 1.3 45 conservative innovative Novelty H2020SC1-DTH-12-2020 D7.5 22|93 In Figure 8 that follows, a visual representation of these results is provided. The shaded purple area in the centre marks the neutral range, with values between -0.8 and +0.8. Values above +0.8 reflect a positive evaluation, while values below -0.8 indicate a negative evaluation. Figure 8. Visual Representation of Mean Long UEQ Item Scores – Spain, BCP, WP5, n=48, 6 months The UEQ results show that most items were rated positively, particularly in the dimensions of Attractiveness, Efficiency, and Dependability, which scored above the neutral range. Perspicuity received mixed ratings, with “not understandable /understandable” falling into the negative area, indicating usability challenges. Stimulation and Novelty were closer/into to the neutral range, suggesting the system was perceived as functional but less exciting or innovative. 3.1.2 Long UEQ Results at 12 months At 12 months sixty-nine BCPs were given the Long UEQ form and from those 52 were included in the analysis (Figure 9). H2020SC1-DTH-12-2020 D7.5 23|93 *For a detailed explanation of inconsistencies and critical length, see p. 20 Table 2. Overview of UEQ Ratings – Spain, BCP, WP5, n=52, 12 months Ite m Mea n Varian ce Std. Dev. No . Left Right Scale 1 1.4 1.6 1.3 52 annoying enjoyable Attractiveness 2 -1.8 3.1 1.8 52 not understandable understandable Perspicuity 3 0.7 2.0 1.4 52 creative dull Novelty 4 2.2 2.0 1.4 52 easy to learn difficult to learn Perspicuity 5 1.5 2.3 1.5 51 valuable inferior Stimulation 6 0.1 1.8 1.3 52 boring exciting Stimulation 7 1.2 1.3 1.2 52 not interesting interesting Stimulation 8 1.1 1.7 1.3 52 unpredictable predictable Dependability 9 0.8 2.9 1.7 52 fast slow Efficiency 10 -0.4 1.9 1.4 52 inventive conventional Novelty 11 1.0 1.7 1.3 52 obstructive supportive Dependability 12 1.6 1.9 1.4 52 good bad Attractiveness 13 2.0 2.0 1.4 52 complicated easy Perspicuity 14 1.7 1.4 1.2 51 unlikable pleasing Attractiveness 15 -0.4 2.4 1.6 52 usual leading edge Novelty 16 1.8 1.4 1.2 52 unpleasant pleasant Attractiveness 17 1.5 1.9 1.4 49 secure not secure Dependability 18 0.8 1.3 1.1 49 motivating demotivating Stimulation 19 1.1 1.7 1.3 50 meets expectations does not meet expectations Dependability 20 1.1 1.4 1.2 49 inefficient efficient Efficiency 21 2.2 1.2 1.1 50 clear confusing Perspicuity 22 1.2 2.1 1.4 50 impractical practical Efficiency 23 2.1 1.4 1.2 50 organized cluttered Efficiency 24 0.8 1.9 1.4 50 attractive unattractive Attractiveness 25 1.8 1.5 1.2 49 friendly unfriendly Attractiveness 26 -0.2 2.1 1.4 48 conservative innovative Novelty In Figure 10 that follows, a visual representation of these results is provided. The shaded purple area in the center marks the neutral range, with values between -0.8 and +0.8. Values above +0.8 reflect a positive evaluation, while values below -0.8 indicate a negative evaluation. Figure 9. Flow chart of responses analysed at 12 months – Spain, BCP WP5 H2020SC1-DTH-12-2020 D7.5 24|93 Figure 10. Visual Representation of Mean Long UEQ Item Scores - Spain, BCP, WP5, n=52, 12 months The UEQ results indicate overall positive ratings across most dimensions, with Attractiveness, Efficiency, and Dependability scoring above neutral. Perspicuity showed variability, with “not understandable/understandable” rated negatively, highlighting potential clarity issues. Stimulation and Novelty remained close to neutral, suggesting limited perceived excitement or innovation. H2020SC1-DTH-12-2020 D7.5 25|93 3.1.2.1 Comparison of UEQ results 6 months vs 12 months In both evaluations, Attractiveness, Efficiency, and Dependability were consistently rated positively. Stimulation shifted from positive to the neutral range, while Novelty remained in the neutral range. Perspicuity was the weakest scale among the positively rated scales in both cases, driven mainly by low scores for “not understandable /understandable.” However, the user experience ratings decreased in all categories, reflecting a less positive perception of the system over time. Table 3. Comparison of UEQ results (6 months vs 12 months) – Spain, WP5 Scale 6-Month Mean (N=48) 12-Month Mean (N=52) Change Attractiveness 1.783 1.510 ▼ 0.273 Perspicuity 1.160 1.115 ▼ 0.045 Efficiency 1.703 1.295 ▼ 0.408 Dependability 1.722 1.173 ▼ 0.549 Stimulation 1.326 0.875 ▼ 0.451 Novelty 0.206 -0.061 ▼ 0.267 H2020SC1-DTH-12-2020 D7.5 32|93 Figure 18. Visual Representation of Mean UEQ Item Scores – Norway, HC WP5, n=36 Values between -0.8 and 0.8 represent a neural evaluation Values > 0,8 represent a positive evaluation Values < -0,8 represent a negative evaluation As seen in the results, the ease of use was the only quality that was evaluated positively. The rest of the pragmatic qualities (supportive, clear, efficient) as well as the hedonic qualities, reflecting how exciting, interesting, inventive, or innovative the system felt to users, received more moderate ratings, falling into to the neutral range. 3.2.2.2 SUS Questionnaire Thirty-six HC completed the questionnaire, of whom 4 responses were excluded due to incomplete or invalid data, resulting in 32 valid responses for analysis. The average SUS score obtained was 58, which falls within the range of marginally acceptable usability (Figure 19). Figure 19. Mean SUS score – Norway, HC, WP5 n=32 3.2.2.3 Additional questions When HC were asked whether they felt the REBECCA pApp enhance their QoL over half of participants responded, “I don’t know”, while around 40% answered “No” and only a very small proportion said “Yes.” These results should be interpreted with caution, given the observational nature of the study, in which no intervention was implemented and the development stage of the pApp. Additionally, for HC – who are healthy and not cancer patients – it is probably even more difficult to imagine potential improvements in their QoL. H2020SC1-DTH-12-2020 D7.5 33|93 Most participants reported feeling secure or very secure about their privacy while using the pApp, with nearly half selecting “Very Secure” and over one-third selecting “Secure”. Only a small percentage felt insecure, and around 14% were uncertain. These results suggest a generally high level of trust in the pApp’s privacy measures. 0% 10% 20% 30% 40% 50% 60% Yes No I don't know Improve QoL 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Very Insecure Insecure I don’t know Secure Very Secure Feelings about privacy Figure 20. Perceived potential of the pApp to enhance QoL after primary treatment, Norway, HC WP5, n=37 Figure 21. Feelings about privacy - Norway, HC WP5, n=36 H2020SC1-DTH-12-2020 D7.5 34|93 Most participants (68%) reported that they did not feel monitored while using the pApp. A smaller proportion (16%) felt monitored, while another 16% indicated they had not thought about it. This suggests that the majority of users were comfortable with the pApp’s level of oversight, though a small subset did perceive a sense of monitoring. Figure 22. Perception of being monitored - Norway, HC WP5, n=37 Overall satisfaction with the pApp was mixed. While nearly half of participants reported being “Satisfied” or “Very Satisfied” (46%), about one-fifth expressed dissatisfaction (20%). The largest group (34%) remained neutral, suggesting that although many participants had a positive experience, there is room for improvement to increase engagement and reduce ambivalence. 0% 10% 20% 30% 40% 50% 60% 70% 80% Yes No Haven't thought about it Felt monitored H2020SC1-DTH-12-2020 D7.5 35|93 Figure 23. Overall satisfaction - Norway, HC WP5, n=35 3.2.3 BCP vs HC Evaluation 3.2.3.1 UEQ Evaluation Figure 24 illustrates a comparative analysis of mean item values between the BCP and HC. Both groups rated the pragmatic quality of the pApp higher compared to the hedonic. Notably, BCP demonstrate higher means in all the item rated and evaluated positively 3 out of the 4 items in the pragmatic quality compared to 1 out of 4 of the HC. These findings reflect differing ways participants experienced the pApp and that might be because BCP who were more likely to benefit from the pApp, appeared more optimistic compared to the HC. Figure 24. UEQ mean values BCP vs HC, WP5, Norway 0% 5% 10% 15% 20% 25% 30% 35% 40% Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied Overall saticfaction H2020SC1-DTH-12-2020 D7.5 36|93 3.2.3.2 SUS Evaluation BCP provided higher ratings for the pApp on the SUS compared to HC. However, both groups’ scores fell within the range of marginally acceptable usability. Figure 25. SUS score BCP vs HC, WP5, Norway 3.2.3.3 Additional questions The figures below present a comparison of responses from BCP and HC to a selection of the additional questions administered at 12 months of the observational study. As this was an observation study, the results should be interpreted with caution. Figure 26 shows perceptions of whether the pApp improved QoL. Most participants in both groups reported no improvement with a higher percentage of BCP. A larger proportion of HC (around 55%) selected “I don’t know,” compared to about onequarter of BCP, while only a small percentage of both groups perceived a positive impact. H2020SC1-DTH-12-2020 D7.5 37|93 Figure 26. QoL Improvement, BCP vs HC, WP5, Norway Figure 27 illustrates feelings about privacy. Both groups reported high levels of confidence, with over 80% of participants indicating that they felt secure or very secure. Very few participants felt insecure or very insecure, and uncertainty was more common among HC than BCP. Figure 27. Feelings about privacy, BCP vs HC, WP5, Norway Figure 28 reflects whether participants felt monitored while using the pApp. The majority of both groups (around 70%) stated that they did not feel monitored. A small proportion of HC (around 15%) reported feeling monitored, while nearly one-third of BCP said they had not thought about it. 0% 10% 20% 30% 40% 50% 60% 70% Yes No I don't know Improve QoL Patients Healthy Controls 0% 10% 20% 30% 40% 50% 60% Very Insecure Insecure I don’t know Secure Very Secure Feelings about privacy Patients Healthy Controls H2020SC1-DTH-12-2020 D7.5 38|93 Figure 28. Felt monitored, BCP vs HC, WP5, Norway The fourth figure presents overall satisfaction. Satisfaction levels were relatively high in both groups (higher in BCP group), with the majority reporting being satisfied or very satisfied. Neutral responses were more common among HC, while dissatisfaction remained low in both groups, and only a few HC reported being very dissatisfied. Figure 29. Overall Satisfaction, BCP vs HC, WP5, Norway Overall, BCP and HC reported similar perceptions of privacy with low levels of discomfort or perceived monitoring. However, HC tended to be less satisfied and more uncertain about potential QoL benefits. In general, HC appeared more cautious and 0% 10% 20% 30% 40% 50% 60% 70% 80% Yes No Haven't thought about it Felt monitored Patients Healthy Controls 0% 5% 10% 15% 20% 25% 30% 35% 40% Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied Overall satisfaction Patients Healthy Controls H2020SC1-DTH-12-2020 D7.5 39|93 less enthusiastic in their responses compared to BCP, with a higher frequency of “I don’t know,” “Neutral,” and occasional “Very dissatisfied” ratings. This difference likely stems from HC limited personal connection to the challenges of illness. Without firsthand experience of the pApp’s intended support, they may find it harder to recognize clear benefits, which can result in a more critical or emotionally detached evaluation. H2020SC1-DTH-12-2020 D7.5 40|93 4 System evaluation from WP6 4.1 Spain - REBECCA-INCLIVA-Inter-QoL The system’s evaluation in the intervention study in Spain, involves both BCP and HCP. It includes the UEQ-S, the SUS, and additional project-specific questions. At the time of writing this report, fifteen BCP and five HCP had completed the evaluation form. Figure 30. System evaluation WP6 - Spain 4.1.1 BCP evaluation of the pApp 4.1.1.1 UEQ-S Fifteen BCP evaluated the pApp using the UEQ-S. Table 6 shows the mean value and standard deviation for each of the 8 items rated. All items received positive ratings, with mean values above 0.8. Pragmatic qualities such as clarity, ease of use, supportiveness, and efficiency were consistently well-evaluated, confirming that the system is perceived as functional and user-friendly. Hedonic qualities, including excitement, inventiveness, and innovativeness, were also rated positively, suggesting that participants found the system engaging and stimulating overall. H2020SC1-DTH-12-2020 D7.5 41|93 Table 6. Overview of UEQ Ratings –Spain, BCP WP6, n=15 Item Mean Std. Dev. Negative Positive Scale 1 1.8 1.4 obstructive supportive Pragmatic Quality 2 2.3 1.5 complicated easy Pragmatic Quality 3 1.5 1.7 inefficient efficient Pragmatic Quality 4 2.0 2.1 confusing clear Pragmatic Quality 5 1.0 1.6 boring exciting Hedonic Quality 6 1.4 1.6 not interesting interesting Hedonic Quality 7 1.9 1.4 conventional inventive Hedonic Quality 8 1.6 1.4 usual leading edge Hedonic Quality In Figure 31 that follows, a visual representation of these results is provided. The shaded purple area in the centre marks the neutral range, with values between -0.8 and +0.8. Values above +0.8 reflect a positive evaluation, while values below -0.8 indicate a negative evaluation Figure 31. Visual Representation of Mean UEQ Item Scores - Spain, BCP WP6, n=15 H2020SC1-DTH-12-2020 D7.5 48|93 Table 12. Overview of UEQ Ratings - Norway, BCP WP6, n=28 Item Mean Std. Dev. Negative Positive Scale 1 1.1 1.4 obstructive supportive Pragmatic Quality 2 2.1 1.2 complicated easy Pragmatic Quality 3 0.6 2.1 inefficient efficient Pragmatic Quality 4 1.5 1.6 confusing clear Pragmatic Quality 5 0.3 1.9 boring exciting Hedonic Quality 6 0.6 1.9 not interesting interesting Hedonic Quality 7 0.3 1.8 conventional inventive Hedonic Quality 8 -0.3 2.0 usual leading edge Hedonic Quality *Evaluation from 1 to 7 In Figure 40 that follows, a visual representation of these results is provided. The shaded purple area in the centre marks the neutral range, with values between -0.8 and +0.8. Values above +0.8 reflect a positive evaluation, while values below -0.8 indicate a negative evaluation. Figure 40. Visual Representation of Mean UEQ Item Scores - Norway, BCP WP6, n=28 H2020SC1-DTH-12-2020 D7.5 49|93 As seen in the results, supportiveness, ease of use, and clarity were rated positively, standing out as strengths of the system. The other pragmatic quality (efficiency) and all hedonic qualities (excitement, interest, inventiveness, innovativeness) received more moderate scores, falling within the neutral range. This suggests that while the system was perceived as functional and easy to use, it was viewed as moderately engaging or innovative from an experiential perspective. 4.2.1.2 SUS Questionnaire Twenty-eight BCP completed the questionnaire, of whom five responses were excluded due to incomplete data, resulting in twenty-three valid responses for analysis. The average SUS score obtained was 71.7, which falls within the range of acceptable usability (Figure 41). Figure 41. Mean SUS score – Norway, BCP, WP6 n=23 4.2.1.3 Additional questions Figure 42 illustrates BCP perceptions of whether the pApp improved their QoL. Over half (around 55%) reported that the pApp did not improve their QoL, while about onethird (35%) felt it had a positive effect. A smaller proportion (approximately 10%) were uncertain. Figure 42. Perceived QoL improvement after primary treatment, Norway, BCP WP6, n= 26 0% 10% 20% 30% 40% 50% 60% Yes No I don't know Improve QoL H2020SC1-DTH-12-2020 D7.5 50|93 The following figure (Figure 43) presents BCP feelings about privacy when using the pApp. The majority expressed positive views, with around half (50%) reporting that they felt secure and more than one-third (35%) feeling very secure. Only a small proportion indicated negative perceptions, with about 10% reporting that they felt very insecure and a few participants (under 5%) stating they felt insecure. Figure 43. Feelings about privacy, Norway, BCP WP6, n= 28 Figure 44 shows participants’ perceptions of whether they felt monitored while using the pApp. The vast majority (79%) reported that they did not feel monitored, while about 15% stated that they had not thought about it. Only a very small proportion (around 7%) indicated that they did feel monitored. These results suggest that most participants did not experience the pApp as intrusive. 0% 10% 20% 30% 40% 50% 60% Very Insecure Insecure I don’t know Secure Very Secure Feelings about privacy H2020SC1-DTH-12-2020 D7.5 51|93 Figure 44. Felt monitored. Norway, BCP WP6, n= 28 Most participants were satisfied (about 75% combined “Satisfied” and “Very satisfied”), while only around 15% expressed dissatisfaction. These finding suggests a generally positive experience. Figure 45. Overall satisfaction. Norway, BCP WP6, n= 26 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Yes No Haven't thought about it Felt monitored 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied Overall satisfaction H2020SC1-DTH-12-2020 D7.5 52|93 4.2.2 HCP evaluation of the system 4.2.2.1 UEQ-S Three HCP evaluated the REBECCA system using the UEQ-S. The Table 13 below shows the mean value and standard deviation for each of the 8 items rated. Table 13. Overview of UEQ Ratings - Norway, HCP WP6, n=3 Item Mean Std. Dev. Negative Positive Scale 1 1.7 0.6 obstructive supportive Pragmatic Quality 2 2.0 0.0 complicated easy Pragmatic Quality 3 1.0 1.0 inefficient efficient Pragmatic Quality 4 1.7 0.6 confusing clear Pragmatic Quality 5 2.0 1.0 boring exciting Hedonic Quality 6 2.3 0.6 not interesting interesting Hedonic Quality 7 1.0 1.0 conventional inventive Hedonic Quality 8 0.7 1.2 usual leading edge Hedonic Quality *Evaluation from 1 to 7 Figure 46 that follows, a visual representation of these results is provided. Figure 46. Visual Representation of Mean UEQ Item Scores – Norway, HCP WP6, n=3 Values between -0.8 and 0.8 represent a neural evaluation Values > 0,8 represent a positive evaluation Values < -0,8 represent a negative evaluation H2020SC1-DTH-12-2020 D7.5 53|93 4.2.2.2 SUS Questionnaire Three HCP completed the SUS questionnaire. The average SUS score obtained was 78.3, which falls within the range of acceptable usability (Figure 47). Figure 47. Mean SUS score – Norway, HCP, WP6 n=3 4.2.2.3 Additional questions To complement the standardized questionnaires, HCP were asked targeted questions to assess their experience with the REBECCA system (Annex A.3). The following tables summarize their responses (n = 3). HCP were asked whether they felt the REBECCA system was integrated into their clinical workflow. Table 14. Perceived integration of the REBECCA system into clinical workflow - Norway, HCP WP6, n=2 Answer Count (Frequency) Percentage % Yes 2 100% No 0 The following question evaluated whether the clinical dashboard supported HCP in enhancing their workflow. Table 15. Usefulness of the clinical dashboard in improving workflow - Norway, HCP WP6, n=2 Answer Count (Frequency) Percentage % Yes 2 100% No 0 HCP rated also their level of trust in the pApp’s data for monitoring patients’ lifestyles. Table 16.Trust in pApp data for monitoring patient lifestyles - Norway, HCP WP6, n=3 Answer Count (Frequency) Percentage % Not at all 0 0% A little 0 0% Moderately 0 0% A lot 3 100% Extremely 0 0% H2020SC1-DTH-12-2020 D7.5 54|93 The following question captured HCP overall satisfaction with using the REBECCA system. Table 17. Overall satisfaction with the REBECCA system experience - Norway, HCP WP6, n=3 Answer Count (Frequency) Percentage % Very satisfied 0 0% Satisfied 2 67% Neutral 1 33% Dissatisfied 0 0% Very dissatisfied 0 0% 4.3 Sweden - REBECCA-KI-Feas-QoL The REBECCA-KI-Feas-QoL was conducted in Stockholm by KI in collaboration with AMAZONA. The study evaluated the usability and feasibility of the REBECCA system, as well as changes in BCS QoL, using data collected from the pApp, Garmin wearable devices, browser history, and REDCap questionnaires among BCS. Of the 61 BCS initially enrolled, 54 were included in the final analysis. 4.3.1 Evaluation of the pApp – UEQ-S To evaluate the user experience of the pApp, the UEQ-S was used, and 55 responses were collected. Two responses were excluded from the analysis: one due to a critical pattern, where inconsistent ratings within the same scale indicated a low-quality response, and one due to critical length, defined as 15 or more identical answers, suggesting low engagement. As a result, 53 responses were included in the final analysis. Table 18 shows the mean value and standard deviation for each of the 8 items rated. The "Negative" and "Positive" columns show the two extremes of the scale participants used to evaluate each aspect of the pApp. These items are grouped into two broader dimensions of user experience: Pragmatic Quality and Hedonic Quality. Pragmatic Quality refers to how useful, easy to use, and clear the pApp is, while Hedonic Quality reflects how engaging, exciting, and interesting the pApp feels. H2020SC1-DTH-12-2020 D7.5 55|93 Table 18. Overview of UEQ, - Sweden, BCS, WP6 n=53 Item Mean Std. Dev. Negative Positive Scale 1 0.2 1.3 obstructive supportive Pragmatic Quality 2 1.8 1.2 complicated easy Pragmatic Quality 3 0.2 1.2 inefficient efficient Pragmatic Quality 4 1.1 1.5 confusing clear Pragmatic Quality 5 -0.5 1.1 boring exciting Hedonic Quality 6 0.1 1.4 not interesting interesting Hedonic Quality 7 -0.1 1.0 conventional inventive Hedonic Quality 8 0.0 1.0 usual leading edge Hedonic Quality In Figure 48 that follows, a visual representation of these results is provided. The shaded purple area in the centre marks the neutral range, with values between -0.8 and +0.8. Values above +0.8 reflect a positive evaluation, while values below -0.8 indicate a negative evaluation. Figure 48.Visual Representation of Mean UEQ Item Scores - Sweden, BCS, WP6, n=53 H2020SC1-DTH-12-2020 D7.5 56|93 As seen in Figure 48, the pragmatic qualities of the pApp such as how easy, and clear it is, were evaluated positively. In contrast, the rest of the pragmatic aspects (supportiveness and efficiency) and all the hedonic qualities, which capture how interesting, exciting, or novel the pApp felt to BCS, received neutral ratings. This suggests that while the pApp was seen as functional, it was perceived as emotionally neutral or less engaging in terms of user experience. 4.3.2 Evaluation of the pApp – SUS Questionnaire Fifty-four participants completed the SUS questionnaire, of whom 3 responses were excluded due to incomplete or invalid data, resulting in 51 valid responses for analysis. The average SUS score obtained was 68.53, which is just around the threshold of acceptable usability. However, scores above 68 are generally considered above average. Figure 49. SUS Results - Sweden, BCS, WP6, n=51 4.3.3 Evaluation of the consultation At the end of the study (3 months from baseline), BCS were asked to evaluate the overall value and relevance of the personalized consultations. The questions aimed to capture their perceptions of the usefulness of the content, how well it addressed their individual needs, and whether it had a positive impact on their daily life and well-being. Responses were collected using a 5-point scale ranging from "Not at all" to "Completely". BCS responses are presented in the graphs below. Around half of the BCS reported that they trusted the guidance provided during the consultation “A lot”. A smaller portion expressed complete trust, while very few BCS reported low or no trust. H2020SC1-DTH-12-2020 D7.5 57|93 Figure 50. Trust in the guidanceSweden, BCS, WP6, n=53 A large portion of the BCS felt that the recommendations aligned well with their personal needs, with many indicating that the guidance provided was relevant and appropriate for their situation. Fewer participants reported only moderate alignment, and very few felt that the recommendations did not meet their needs. Figure 51. Recommendations' alignment with participants’ needsSweden, BCS, WP6, n=53 When asked about improvements in their QoL, many BCS reported noticing a slight or moderate positive change. Only a few BCS indicated either a substantial improvement or no improvement at all. 0% 10% 20% 30% 40% 50% 60% Not at all A little Moderately A lot Completely Trust the guidance N=53 0% 10% 20% 30% 40% 50% Not at all A little Moderately A lot Completely Alignment of recommendations with needs H2020SC1-DTH-12-2020 D7.5 64|93 6.2 SUS results Table 21 presents the SUS scores, indicating the perceived usability of the system across the three countries based on the different studies (WP), and participant groups. Table 21. SUS results by country, WP and participants’ group Country WP Group SUS Average Score (0 -100) Norway WP5 BCP 65 WP5 HC 58 WP6 HCP 78.3 WP6 BCP 71.7 WP7 PCP 68.13 Spain WP5 BCP Not available WP5 BCP Not available WP6 HCP 56.5 WP6 BCP 74.5 Sweden (Feasibility) WP6 BCS 68.53 52 - 70: marginal acceptable usability (OK) > 70: acceptable usability BCP in the intervention studies (WP6) in both Norway and Spain and HCP in Norway evaluated the REBECCA system with an average SUS score above 70 indicating an acceptable level of usability. Additionally, the two feasibility studies with BCS in Sweden (WP6) and PCP in Norway (WP7) reported average SUS scores above 68, which is generally considered acceptable usability as well. All other groups provided scores within the “marginally acceptable” usability range. 6.3 Additional questions results The results in Table 22 summarize the answers of patient groups and HC in the following measures: (1) Satisfaction, which reflects the percentage of participants who reported being “Very satisfied” or “Satisfied”; (2) Felt monitored, indicating the proportion of participants who answered “Yes” to feeling monitored while using the pApp and wearing the watch; and (3) QoL improvement represents the percentage of participants who reported a positive (“Yes”) change in their QoL. In the Swedish feasibility study, this question was not a simple yes/no format, as QoL improvement was not the primary aim. In this case, the percentage reflects participants who selected positive options (“completely”, “a lot”, or “moderately”). H2020SC1-DTH-12-2020 D7.5 65|93 Table 22. Satisfaction, perceived monitoring, and QoL improvement across patients Country WP Group Satisfaction Felt monitored QoL improvement Norway WP5 BCP 67% 0% 11% WP5 HC 46% 16% 3% WP6 BCP 73% 7% 35% WP7 PCP 66% 5% 10% Spain WP6 BCP 80% 0% 100% Sweden (Feasibility) WP6 BCS 44% 13% 18% (moderately) As shown in Table 22 overall satisfaction levels were high across most of the groups, with over 65% of participants reporting being satisfied or very satisfied with the system. Satisfaction rates were below 50% only among HC and BCS, likely because the system was less relevant to their needs. These two groups also reported higher feelings of being monitored (16% and 13%) compared to 0–7% in the other groups. Reported QoL improvements varied widely: Spanish patients in the intervention study reported 100% improvement, while the other groups had much lower rates, with the next highest being 35% among BCP in Norway in the intervention study (WP6). The results in Table 23 summarize four key measures for HCP: (1) Satisfaction: percentage of HCPs who were “Very satisfied” or “Satisfied”; (2) System integration: percentage of HCP who answered “Yes” to successful integration into the clinical workflow; (3) Workflow improvement: percentage of HCP who answered “Yes” to the dashboard improving their workflow; and (4) Trust in data: percentage of HCP who reported trusting the pApp data (“extremely” to “moderately”). Table 23. HCP Evaluation Results: Norway vs. Spain (WP6) Evaluation measures Norway HCP WP6 Spain HCP WP6 Overall satisfaction 67% 100% Integration of the system in the clinical workflow 100% 80% Workflow improvement using dashboard 100% 40% Trust the data collected from the pApp 100% 40% The Norwegian HCPs were less satisfied with the system compared to the Spanish HCPs. However, only 40% of the Spanish HCPs reported that the clinical dashboard improved their workflow and 60% expressed lower trust in the data, even though the majority (80%) felt that the system was integrated into the clinical workflow. H2020SC1-DTH-12-2020 D7.5 66|93 7 REBECCA final stakeholder workshop – external evaluation The REBECCA final stakeholder workshop took place on 5th of June 2025 at the EHMA Conference in Rennes. It lasted around 1,5 hours and combined presentations with interactive discussions. Coordinators and technical partners introduced the project, its outcomes, and the REBECCA Platform. Participants then engaged in a World Café-style discussion across tables representing different stakeholder groups, focusing on obstacles and enablers for system adoption. The session concluded with table representatives sharing key insights, overarching takeaways highlighted by the coordinators, and participants completing the External Expert System Evaluation Survey. A total of 17 participants attended, including a representative from an HTA (start-up), a representative from EUREGHA, policy managers, researchers, an interoperability architect, patients, students in health management, health managers, and nurses. The following summary captures key insights from the breakout discussions, highlighting valuable perspectives from several stakeholders. 7.1 Workshop’s key insights 1. User-centric data approaches The workshop stressed that digital health tools like REBECCA must start with real user and patient needs to ensure relevance, trust, and long-term engagement. Participants highlighted the importance of making data accessible not just for researchers and policymakers but also for patients and other stakeholders. Data should be presented in formats that are understandable and relatable, avoiding overly abstract or numerical presentations that could hinder engagement. Co-creation with users was seen as essential, with localised solutions reflecting individual contexts and experiences. Patients also emphasised that systems should feel rewarding over time to encourage continued use. 2. Trust, engagement, and human-centredness Building trust was identified as foundational, requiring transparency, respect, and responsiveness from those who manage data. Clear, plain-language communication was seen as critical to making scientific and policy information accessible to all user groups, with attention to tone to foster inclusivity. Participants asked about translation practices used for REBECCA communications and stressed the need for culturally appropriate translations. Neutral intermediaries such as nurses and community facilitators were recognised as key in connecting users with digital systems. Social and H2020SC1-DTH-12-2020 D7.5 67|93 cultural sensitivity was also seen as crucial, especially when working with vulnerable groups. 3. Patient trust, data sensitivities, and involvement Participants agreed that patients should be active co-creators rather than passive data sources, with their preferences and insights shaping the design and implementation of tools like REBECCA. Human-centred design should ensure patients feel respected, with empathy and personalisation at the core, particularly when using highly sensitive data such as social media history. Discussions highlighted the importance of consent and control, including the ability to opt out of certain data streams such as GPS or browsing history. A student nurse reflected that she would find such systems valuable in her future career and would recommend them to patients if trust and clarity were ensured. 4. Decentralisation and local engagement Participants underlined that digital solutions should not follow a one-size-fits-all approach. They recommended decentralising system design and governance to better reflect local needs, practices, and capacities. Special emphasis was placed on reaching rural and underserved areas, with communication strategies tailored to settings with lower literacy levels or limited connectivity. 5. Demographics, equity, and accessibility The workshop noted that system design must account for demographic differences such as age, health status, digital literacy, and rural residency. Data equity was highlighted as a priority, ensuring everyone has equal access to services and the ability to benefit from them. Particular care must be taken when handling sensitive data, such as social media and online behaviour, to ensure transparency, consent, and respect for privacy. 6. Translating expertise and bridging policy gaps Participants highlighted the need to turn scientific insights into actionable and understandable information for policymakers, while involving them early in the project cycle. Political sensitivity and resistance were recognised as barriers that can be mitigated through participatory policymaking and inclusive dialogue. Sustaining engagement over time was also seen as important, supported by clear communication of benefits and strong usability. 7. Integration of science, policy, and healthcare The workshop encouraged stronger links between research and patient care pathways, particularly when introducing innovation such as AI. The use of AI in primary care must be handled carefully, ensuring that patients understand the process and retain a sense H2020SC1-DTH-12-2020 D7.5 68|93 of control. Participants were curious about whether REBECCA would eventually transition into a standardised platform to support future scaling and implementation. Ethical and transparent governance was emphasised as vital for protecting patient safety, autonomy, and societal benefit. In conclusion, the workshop highlighted a set of key action points to guide the development of digital health tools. These include translating scientific language for policymakers and patients, promoting human-centred and participatory design, and ensuring patients are trusted co-creators. Systems should enable decentralisation, account for demographic and social sensitivities, and communicate clearly and empathetically. Building scientific and data literacy, clarifying the benefits and optionality of data collection, and designing tools that remain valuable over time were emphasised. Finally, bridging research, policy, and practice while planning for longterm sustainability emerged as essential for meaningful, lasting impact. 7.2 Results of the External Expert System Evaluation At the end of the workshop, participants were invited to complete the External Expert System Evaluation Survey (Annex B.3) to capture their perceptions and feedback regarding the REBECCA system. The following graph shows the distribution of the different stakeholder groups that filled in the evaluation. Figure 59. Distribution of stakeholder groups in the final workshop, n=15 H2020SC1-DTH-12-2020 D7.5 69|93 The External Expert System Evaluation questionnaire collected information on stakeholders’ willingness to support integration of REBECCA, the perceived importance of integration within patient management and clinical research, and evaluations of the system’s innovativeness and practicality, using 7-point Likert scales (From 1: Not at all to 7: Extremely). Figure 60. Average rating per stakeholder group, n=15 The evaluation results show generally positive perceptions of the REBECCA system across stakeholder groups, with some differences in emphasis. Willingness to support integration was high for all groups, highest among public health students and policy makers. The health professional rated the system as most important for patient management and clinical research, while researchers and students gave slightly lower scores. Perceptions of innovativeness varied, with health authorities rating it highest and students lowest. Practicality received more mixed evaluations, with health professionals expressing the most concerns. Overall, the results suggest broad acceptability of the system, but highlight the need to address practical implementation challenges, particularly for health professionals. 1.0 2.0 3.0 4.0 5.0 6.0 7.0 Willingness to Support Importance – Patient Mgmt Importance – Clinical Research Innovativeness Practicality Average rating (1-7) Avarage rating per stakeholder group Researchers and tech Policy makers Health authorities Health pro Public health students H2020SC1-DTH-12-2020 D7.5 70|93 Conclusions The internal evaluation of the REBECCA system across different countries, WPs, and participant groups provides valuable insights into its overall usability, satisfaction, and effectiveness in supporting patient care. Pragmatic quality scores were generally positive, particularly among HCPs and patients in the intervention studies (WP6), while hedonic quality scores were mostly neutral, with lower ratings from HC and BCS. In terms of usability, SUS scores were generally acceptable or marginally acceptable for all participant groups. Differences in reported QoL improvement, with Spanish patients showing the highest improvement, suggest that the system’s impact may vary according to local context. The relatively low trust in data reported by Spanish HCPs further highlights a barrier to clinical adoption, emphasizing the need for improved data transparency and integration with existing workflows. These findings were complemented by the External Expert System Evaluation conducted during the final stakeholder workshop. Results indicate broad acceptability of the REBECCA system among external stakeholders, while also highlighting practical implementation challenges, particularly for frontline health professionals. The workshop also identified key action points for digital health tool development, including human-centred and participatory design, patient involvement as trusted cocreators, clear and empathetic communication, decentralisation, consideration of demographic and social sensitivities, promotion of data literacy, and long-term sustainability. Taken together, these insights suggest that while the REBECCA system demonstrates usability and functional adequacy, future development should focus on enhancing engagement, building trust, and supporting adaptation to local contexts. Integrating the workshop recommendations will further maximise the system’s value and facilitate successful adoption in real-world settings. H2020SC1-DTH-12-2020 D7.5 71|93 References 1. Schrepp, M., Hinderks, A., & Thomaschewski, J. (2017). User Experience Questionnaire Handbook. 2. Available online: https://www.ueq-online.org 3. Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40. 4. Marangunić N, Granić A. Technology acceptance model: a literature review from 1986 to 2013. Universal access in the information society. 2015 Mar;14:81-95. 5. AlQudah AA, Al-Emran M, Shaalan K. Technology acceptance in healthcare: A systematic review. Applied Sciences. 2021 Nov 9;11(22):10537. 6. Alloghani M, Hussain A, Al-Jumeily D, Abuelma'atti O. Technology Acceptance Model for the Use of M-Health Services among health related users in UAE. In2015 International Conference on Developments of E-Systems Engineering (DeSE) 2015 Dec 13 (pp. 213-217). IEEE. H2020SC1-DTH-12-2020 D7.5 72|93 ANNEX H2020SC1-DTH-12-2020 D7.5 73|93 A. Evaluation forms A.1 Evaluation form – Patients H2020SC1-DTH-12-2020 D7.5 80|93 H2020SC1-DTH-12-2020 D7.5 81|93 H2020SC1-DTH-12-2020 D7.5 82|93 H2020SC1-DTH-12-2020 D7.5 83|93 A.3 Evaluation form - HCP H2020SC1-DTH-12-2020 D7.5 84|93 H2020SC1-DTH-12-2020 D7.5 85|93 H2020SC1-DTH-12-2020 D7.5 86|93 H2020SC1-DTH-12-2020 D7.5 87|93 A.4 Feedback questionnaire for the consultation – Feasibility study AMAZONAS H2020SC1-DTH-12-2020 D7.5 88|93 B. REBECCA final stakeholder workshop B.1 Workshop’s structure Duration: 1 hour 15 minutes Objectives: • Evaluate and validate the REBECCA system’s acceptance by patients and health professionals. • Support exploitation strategies and sustainability planning. • Discuss integration of RWD-based solutions into healthcare systems and clinical workflows. • Engage stakeholders to explore adoption strategies and implementation pathways. Workshop Structure: 1. Introduction (15 minutes) o Project presentation and main outcomes by the coordinators. o Platform introduction by the technical partners. 2. Interactive Discussion – World Café Technique (35 minutes) o Participants rotated across three tables, each representing a stakeholder group: decision-makers/policymakers, health authorities/regulatory bodies, health professionals/managers, researchers/tech innovators, and patients’ organisations. o Each table was provided with key discussion topics focusing on obstacles and enablers for adopting the REBECCA system in their respective work. o Each table nominated a representative to remain at the table and present its key insights at the end of the session. 3. Takeaways & Conclusions (25 minutes) o Table representatives shared their discussion outcomes. o Coordinators highlighted overarching takeaways and final remarks. o Participants were invited to complete the External Expert System Evaluation Survey, available via QR code or paper copy. Expected Outcomes: H2020SC1-DTH-12-2020 D7.5 89|93 • Gather concrete insights and recommendations for large-scale deployment of the REBECCA system. • Create a roadmap for integrating the system into stakeholders’ workflows. • Collect evaluation data via the External Expert System Evaluation Survey. B.2 Workshops notes REBECCA workshop at EHMA conference – summary of key insights on health data (5th June 2025, in Rennes) The REBECCA workshop explored the challenges and opportunities surrounding the use of health data, particularly personal data, with a strong focus on ethical, inclusive, and user-centred approaches. This summary includes collective reflections from breakout discussions, including valuable patient and practitioner perspectives. Workshop with: 17 participants. • Representant from an HTA company (start-up) • Representant from EUREGHA • Policy manager • Interoperability architect • Patients • Students in health management • Health managers • Nurse 1. User-centric data approaches • Start with user needs: the workshop emphasised the importance of grounding digital health tools like REBECCA in real user and patient needs to ensure relevance, trust, and long-term engagement. • Data accessibility for all: data should not only serve researchers and policymakers. Participants advocated for meaningful, secure, and broader data access to empower all stakeholders, especially patients. • Making data personal: data should be presented in formats that are relatable and understandable to users. Abstract or purely numerical forms may hinder understanding and engagement.