The Social and hUman CeNtered XR: SUN XR Project

This paper has received financial support by the Horizon Europe Research & Innovation Programme under Grant agreement N. 101092612 (Social and hUman ceNtered XR - SUN project). The Social and hUman ceNtered XR: SUN XR project Luca Greci [1], Ferdinando Bosco [1] and Vincenzo Croce [1] 1 Engineering Ingegneria Informatica spa, Rome, Italy [email protected] Abstract. In the recent years, Extended Reality (XR) technology has emerged as a powerful tool, revolutionizing various industries, offering immersive experiences and making experiences available to individuals who may face physical barriers. XR technology, combined with hyper-realistic avatars and innovative interfaces, holds great promise for revolutionizing rehabilitation practices and improving the quality of life for individuals with serious mobility and verbal communication diseases. In this context the SUN project aims to explore and develop XR solutions that merge the physical and virtual realms, focusing on human and social aspects. The project addresses challenges such as hyper-realistic human interaction interfaces, resource limitations of end-user devices, and wearable haptics for enhanced physical interactions. The project also investigates gaze-based and gesture-based interaction and leverages artificial intelligence to optimize resource allocation and improve the XR experience. This paper presents two use cases that will be developed as demonstrators of the SUN project: XR for Rehabilitation and XR for people with serious mobility and verbal communication diseases. Keywords: Extended reality, limb rehabilitation, mobility diseases, verbal communication diseases, NFT and Metaverse. 1 Introduction Extended Reality (XR) technology has emerged as a powerful tool, revolutionizing various industries and offering immersive experiences. XR technologies have the potential to greatly enhance accessibility, making experiences available to individuals who may face physical or geographical barriers. For example, individuals with disabilities or limited mobility can benefit from virtual experiences that provide them with opportunities to explore new environments, participate in educational activities, or engage in social interactions [1, 2, 3]. XR can level the playing field and ensure that everyone, regardless of physical limitations, can access and engage with various forms of content and experiences. Moreover, XR technologies offer opportunities for individuals to connect, collaborate, and engage in shared experiences regardless of their physical location [4]. Virtual reality platforms enable people from different parts of the world to come together in shared virtual spaces, fostering social inclusion and breaking down geographical barriers. 2 XR can provide individuals who may feel isolated or marginalized in their physical environments with a sense of belonging and community. XR technologies have shown promise in the healthcare sector, particularly in the realms of mental health, rehabilitation, and assistive therapy [5]. Virtual reality-based interventions can provide accessible and controlled environments for individuals with anxiety disorders [6, 7], phobias [8], facilitating exposure therapy and reducing barriers to mental healthcare. This paper presents the real-life scenarios that will be used to evaluate the technologies developed by the SUN project in the field of upper and lower body rehabilitation and supporting people with verbal communication diseases: XR for rehabilitation and XR for people with serious mobility and verbal communication diseases. 2 Upper and lower limb rehabilitation Upper and lower limb rehabilitation is a vital component of medical care for individuals with impairments or injuries affecting their arms, hands, legs, or feet. This type of rehabilitation aims to restore mobility, functionality, and independence in performing activities of daily living. Traditional rehabilitation approaches often involve repetitive exercises, manual therapy, and assistive devices to facilitate recovery [9]. While these methods have proven effective to some extent, they often lack engagement, personalization, and real-life context, leading to reduced motivation and adherence to therapy. Traditional rehabilitation approaches face several limitations that can be addressed by incorporating XR technologies. Firstly, traditional methods often lack engaging and immersive experiences, which can result in boredom and reduced motivation for patients. XR technologies, such as virtual reality (VR) and augmented reality (AR), provide a more captivating and interactive environment, offering patients a sense of presence and enjoyment during their rehabilitation sessions [10-13]. Secondly, traditional approaches may lack personalization and customization. Each patient has unique needs and abilities, and traditional therapies often follow a one-sizefits-all approach. In contrast, XR technologies enable the creation of hyper-realistic avatars (HRA) that can mimic the patient's specific body movements and characteristics. This personalization aspect enhances the relevance and effectiveness of rehabilitation exercises, as patients can see themselves represented in the virtual environment, increasing their sense of ownership and engagement [14]. Lastly, traditional rehabilitation methods often lack real-life context, making it challenging for patients to transfer their skills and progress into their daily activities. XR technologies can simulate real-world scenarios and environments, providing patients with the opportunity to practice functional movements and tasks in a safe and controlled virtual setting. By bridging the gap between therapy and real-life situations, XR-based rehabilitation can improve the transfer of skills and boost patients' confidence in performing daily activities independently [15, 16]. 3 3 Mobility and verbal communication diseases For individuals with limited mobility, XR offers virtual environments that can be accessed from the comfort of their homes. XR headsets provide a means of escape and exploration, enabling users to visit distant locations engage in physical activities, or even attend events they would otherwise be unable to access. Through augmented reality (AR), users can experience simulations that mimic realworld scenarios, allowing them to practice and improve their mobility skills in a safe and controlled environment [18]. XR technology is empowering individuals with mobility challenges by granting them newfound freedom, independence, and a sense of inclusion. People with verbal communication diseases often face significant challenges in expressing themselves and interacting with others. XR technology can bridge this communication gap by offering innovative solutions. With XR, individuals can leverage gesture recognition, eye tracking, and voice commands to communicate their needs, emotions, and thoughts more effectively [19]. AR overlays can display real-time subtitles, translating spoken language into written text, enabling smooth conversations with others. Virtual reality (VR) environments facilitate non-verbal forms of expression, such as body language and hand gestures, allowing individuals to connect with their surroundings and express themselves more fully. XR technology empowers people with verbal communication diseases to engage in meaningful social interactions and facilitates improved communication with their caregivers, family, and friends [20]. 4 Hyper-realistic avatar and XR 4.1 For rehabilitation In SUN photorealistic avatars are used and represent new cutting-edge technology compared with low quality 2d avatars and cartonized version. They offer a realistic representation of yourself in the virtual world and facial expressions that make your avatar more human. HRAs, in the context of rehabilitation, are digital representations that closely mimic the appearance and movements of individuals undergoing therapy. The hyper-realist factor represents a strong user engagement factor, in fact the experience shows that the engagement obtained by avatar that accurately resembles the user is significantly higher than the static and cartoon-like ones. HRAs are designed using advanced 3D photogrammetry body scanning techniques and motion capture technologies to replicate the patient's unique physical characteristics, including body shape, limb movements, joint angles, and even facial expressions. HRAs are then integrated into virtual reality (VR) or augmented reality (AR) environments, where patients can interact with and control their virtual counterparts. The role of HRA in rehabilitation extends beyond mere visual representation. They play a crucial role in enhancing patient engagement and motivation. By embodying the 4 patient in a virtual environment, avatars provide a strong sense of presence and immersion, making the rehabilitation experience more interactive and compelling. Patients can see themselves performing movements and tasks through the avatar's perspective, which creates a powerful visual feedback loop [21]. This visual feedback can boost motivation by providing a clear understanding of progress and improvements, as well as highlighting areas that need further attention [22]. Furthermore, customization and personalization of avatars are essential features in individualized rehabilitation. HRA can be tailored to match the patient's specific needs, abilities, and preferences. This customization includes adjusting the avatar's physical attributes, such as limb lengths or joint ranges, to accurately represent the patient's impairments. Personalization extends to the virtual environment as well, allowing patients to choose different scenarios, difficulty levels, or even incorporating gamification elements to make the rehabilitation experience more enjoyable and engaging. By providing individualized avatars, rehabilitation programs can address the specific challenges and goals of each patient, enhancing their sense of ownership and promoting a higher level of commitment to the therapy process [23]. 4.2 To enhance communication HRA, Real-time 3D Human Pose-Motion Reconstruction and XR technologies have ushered in a new era of accessibility and inclusivity for individuals facing serious mobility and verbal communication diseases. By harnessing the power of virtual environments and lifelike avatars, these innovative solutions offer transformative opportunities for individuals to overcome their challenges and engage with the world on their terms. Hyper-realistic HRA serve as digital representations of individuals, enabling them to express themselves, communicate, and interact with others in virtual spaces. For people with verbal communication diseases, these avatars become powerful tools for overcoming limitations [23]. Through the integration of XR, users can control their avatars' movements, gestures, and expressions, facilitating non-verbal communication. This empowers individuals to express their thoughts, emotions, and needs in a way that transcends verbal language barriers, leading to improved social interaction and a greater sense of connection with others [24]. 5 The SUN XR project The main objective of The Social and hUman ceNtered XR (SUN) project is to explore and create XR solutions that effectively merge the physical and virtual realms, with a specific focus on human and social aspects. By utilizing the virtual world, this initiative seeks to enhance the physical environment by introducing new possibilities for socializing and engaging with others. The solution will be offered as SUN XR Platform, including digital twins, wearable sensors and haptic interfaces, artificial intelligencebased solutions to address limitations of wearable devices, hyper-realistic avatars of real people, blockchain-based solutions for the digital asset management. In relation to 5 the latest a tokenized platform, leveraging on NFTs, will be embedded for the fair, transparent and secure management of the XR digital assets. The SUN project aims to tackle certain challenges in XR by addressing the lack of hyper-realistic and immersive human interaction interfaces, as well as resource limitations of end-user devices. To overcome these limitations, the project will focus on developing innovative wearable sensors and haptic interfaces that enable natural and convincing interaction with the virtual environment. The introduction of wearable haptics will enhance physical interactions, allowing users to engage with virtual menus and receive contextually-relevant information through tactile feedback. Furthermore, the project will explore solutions for providing body contextual information, such as skin stretch-vibration on different body parts and vibrotactile and thermal cues under fingertips. These enhancements will guide users in various tasks, including remote training, home physical exercises, and interacting with others in XR. In addition, the project will investigate and develop advanced solutions for user interaction, specifically gaze-based and gesture-based interaction. These approaches will enable users to interact with the XR environment using their gaze or gestures, enhancing the immersion and usability of the experience. To address the current limitations of wearable devices in terms of computing power, memory, and network connectivity, the project will leverage artificial intelligencebased solutions. These intelligent systems will optimize resource allocation, improve data processing capabilities, and overcome network constraints, ensuring a smoother and more efficient XR experience on wearable devices. The solutions will be demonstrated in real-life scenarios, focusing on social interaction and collaboration. 5.1 XR for rehabilitation The objective of this scenario is to enhance adherence to the physiotherapy protocol, boost patient involvement, monitor physiological conditions, and offer immediate feedback to patients by real-time classification of exercises as correctly or incorrectly executed, based on the criteria set by physiotherapists. It is widely recognized that combining visual and physical interaction yields better task performance compared to relying solely on visual interaction. Wearable haptics, such as EMG, can be employed to enhance the monitoring of physical interaction by providing contextual information about the body. Visual AI algorithms can aid in understanding body alignment, capturing outlines, and delivering personalized real-time feedback to enhance the quality of movement. Ensuring adherence and fidelity to rehabilitation is particularly challenging, especially when it comes to motor learning with personalized feedback. Physiotherapy plays a crucial role in the complete rehabilitation of conditions like orthopedic surgery, "frozen shoulder," or severe hand arthritis, but it often becomes repetitive, monotonous, and time-consuming. In fact, achieving motor recovery often requires an extensive course of physiotherapy treatment, sometimes exceeding 50 sessions. 6 Therefore, it is essential to bridge the gap between evidence and practice and effectively implement digital innovations while improving their accuracy and portability in home environments. By starting with a visual representation of an exercise, conveyed through an avatar of the therapist performing it correctly, the patient will be asked to repeat the exercise or synchronize their movements with the avatar. Wearable devices and wireless sensors will be utilized to measure and provide contextual information, including multiple IMUs (Inertial Measurement Unit) for dynamic motion, body and limb orientation kinetics and kinematics, and the maximum joint angle achieved. SEMG will assess neuromuscular potentials like activation and fatigue, while a smartwatch will monitor heart rate, oxygen saturation, and arterial blood pressure. In specific cases, a flex sensor could replace a gyroscope to measure limb angles during bending, and a force-sensitive sensor can gauge limb pressure, potentially enabling classification of various rehabilitation exercises across a sensor network. Real-time data from a camera and the sensors will be collected and sent to a GPU server, which could be cloud-based or edge-based, leveraging cloud-based AI infrastructure or edge AI technology, respectively. An AI algorithm will process the sensory input, adjusting the exercise difficulty-intensity based on muscle activation levels and providing personalized feedback on movement quality. Clinicians can also contribute their feedback at any point during the automated feedback process to enhance accuracy, ensure safety, and further improve the algorithm. Prior to the application of these complex innovative digital tools, usability testing is necessary. Additionally, the goal of this scenario is to increase user engagement among diverse populations. This can be achieved by incorporating personalization for diverse populations with varying rehabilitation needs and abilities. An iterative-convergent-mixed-methods design will be employed to assess and address any significant usability issues, optimizing the user experience and facilitating adoption. This design approach will provide transparency and guidance throughout the development and implementation of the tool within clinical pathways. 5.2 XR for people with serious mobility and verbal communication diseases Individuals with motor disabilities or those recovering from strokes often face significant challenges in communication and meeting their essential needs. The project aims to address this issue by establishing a dedicated communication pathway specifically designed for such individuals. This pathway will enable them to interact with specific social cues, transforming them into clear communication or actions. By leveraging residual abilities and assigning them meaning in terms of communication, the project will incorporate avatars in a virtual environment as needed. The objective is to develop low-cost, non-invasive tools that utilize existing biofeedback, facial expressions, and other input methods. The main challenge lies in creating a solution that allows for person-specific XR interactions in various settings such as home, workplace, or school. The project also aims to develop innovative multi-user virtual communication and collaboration 7 solutions that deliver cohesive multisensory experiences while effectively conveying relevant social cues. Successful implementation of the pilot scenario will enable individuals with communication and motor disabilities to interact with friends and family, realistically meeting in an extended environment that combines the physical and virtual realms. The person with disabilities will be represented by an avatar, interacting with others present in an augmented physical environment. Simultaneously, the individual with disabilities will experience the illusion of being present in the same physical environment as others, interacting with the new interfaces provided by the SUN project. To achieve this, SUN will develop a new generation of non-invasive bidirectional body-machine interfaces (BBMIs). These interfaces will enable smooth and highly effective interaction between individuals with different types of sensory-motor disabilities and virtual reality environments with avatars. The BBMI will utilize printed electrode arrays to record muscle activities and inertial sensors, with the sensor positions tailored to the specific motor abilities of the individuals. For example, shoulder and elbow movements may be used, and muscle activities from auricular muscles could also be recorded. These sensors will capture information during various upper limb and hand movements, and a tailored decoding algorithm will be implemented to identify the most useful signals for different tasks. This approach has already proven successful in controlling flying drones and is expected to yield promising results in this context as well. Machine learning techniques will be utilized to decode the different tasks, relying on human-machine interfaces and extracting information from electrophysiological and biomechanical signals. Furthermore, SUN aims to provide sensory feedback to the user regarding movement and interaction with the avatar in the virtual environment. This will be achieved through techniques such as transcutaneous electrical stimulation or small actuators for vibration. 6 Conclusion and future work The paper describes the general objectives of the SUN focusing on the two scenarios aimed at improving rehabilitation and communication for individuals with physical disabilities. The first scenario focuses on enhancing adherence to physiotherapy protocols and increasing patient engagement through real-time classification of exercises. The use of wearable haptics and visual AI algorithms allows for better monitoring of physical interactions and personalized feedback. The goal is to bridge the gap between evidence and practice by implementing digital innovations accurately and effectively in home environments. Various wearable devices and wireless sensors are employed to measure physiological conditions and provide contextual information. Real-time data from cameras and sensors are processed by an AI algorithm to adjust exercise difficulty and provide personalized feedback on movement quality. Clinician feedback is also incorporated to enhance accuracy and safety. 8 The second scenario addresses communication and essential needs for individuals with motor disabilities or those recovering from strokes. The project aims to establish a dedicated communication pathway using XR technologies. Hyper-realistic avatars in virtual environments will enable individuals to interact with social cues and convert them into clear communication or actions. The goal is to develop low-cost, non-invasive tools that utilize biofeedback, facial expressions, and other input methods. The project also aims to create multi-user virtual communication and collaboration solutions that effectively convey social cues. Non-invasive bidirectional body-machine interfaces (BBMIs) will be developed to enable smooth interaction between individuals with sensory-motor disabilities and virtual reality environments. Machine learning techniques will be used to decode different tasks based on electrophysiological and biomechanical signals. Finally, sensory feedback through electrical stimulation or vibration will enhance the user's experience and interaction with the avatar in the virtual environment. The future phase of this project will involve implementing and assessing the pilots in terms of usability and performance. The results obtained from the pilots will be evaluated through focus groups, which will conduct interaction analysis and assess factors such as the sense of immersion, experience sharing, and engagement. The assessment of the pilots will also involve the evaluation by professional users. For instance, in the rehabilitation scenario, a total of 10 subjects will be enrolled, and the outcomes will be evaluated using the QuichDASH scale. The QuichDASH scale is a shortened version of the DASH (Disability of the Arm, Shoulder, and Hand) scale, which can be found at https://dash.iwh.on.ca/about-quickdash. References 1. Aasen J, Galaaen K, Nilsson F, Sørensen T, Lien L, Leonhardt M. Promoting Social Participation and Recovery Using Virtual Reality-Based Interventions Among People With Mental Health and Substance Use Disorders: Qualitative Study. JMIR Form Res. 2023 Apr 27;7:e46136. doi: 10.2196/46136. PMID: 37104000; PMCID: PMC10176145. 2. Kourtesis P, Kouklari E-C, Roussos P, Mantas V, Papanikolaou K, Skaloumbakas C, Pehlivanidis A. Virtual Reality Training of Social Skills in Adults with Autism Spectrum Disorder: An Examination of Acceptability, Usability, User Experience, Social Skills, and Executive Functions. Behavioral Sciences. 2023; 13(4):336. https://doi.org/10.3390/bs13040336 3. Liu, Lu. (2023). Virtual Reality for Social Skills Training of Children and Adolescents with ASD: A Systematic Review. Journal of Education, Humanities and Social Sciences. 8. 20612067. 10.54097/ehss.v8i.4645. 4. Arlati, S., Colombo, V., Spoladore, D., Greci, L., Pedroli, E., Serino, S., ... & Sacco, M. (2019). A social virtual reality-based application for the physical and cognitive training of the elderly at home. Sensors, 19(2), 261. 5. Weiss, P. L., Kizony, R., & Feintuch, U. (2008). Virtual reality in neurorehabilitation. Textbook of Neural Repair and Rehabilitation, 1253-1265. 6. Carl E, Stein AT, Levihn-Coon A, Pogue JR, Rothbaum B, Emmelkamp P, Asmundson GJG, Carlbring P, Powers MB. Virtual reality exposure therapy for anxiety and related disorders: A meta-analysis of randomized controlled trials. J Anxiety Disord. 2019 Jan; 61:2736. doi: 10.1016/j.janxdis.2018.08.003. Epub 2018 Aug 10. PMID: 30287083. Malbos, E., 9 Rapee, R. M., Kavakli, M., & Kagan, C. (2019). Virtual reality applications for the treatment of social anxiety disorder: An overview. Journal of Clinical Psychology, 75(12), 2463-2479. 7. García-Palacios, A., Botella, C., Hoffman, H., & Fabregat, S. (2007). Comparing acceptance and refusal rates of virtual reality exposure vs. in vivo exposure by patients with specific phobias. CyberPsychology & Behavior, 10(5), 722-724. 8. Veerbeek, J. M., van Wegen, E. E., van Peppen, R. P. S., van der Wees, P. J., Hendriks, E. J., Rietberg, M. B., and Kwakkel, G., What is the evidence for physical therapy poststroke? a systematic review and meta-analysis, PLoS ONE, vol. 9, no. 2, p. e87987, 2014. 9. M. Holden, Virtual environments for motor rehabilitation: review, Cyberpsychology, Behavior, and Social Networking, vol. 8, no. 3, pp. 187–211, 2005. 10. Y. Ou, Y. Wang, H. Chang, S. Yen, Y. Zheng, and B. Lee, Development of virtual reality rehabilitation games for children with attention-deficit hyperactivity disorder, Journal of Ambient Intelligence and Humanized Computing, vol. 11, no. 11, pp. 5713–5720, 2020. 11. E. Köktürk, F. Molteni, M. Bordegoni, and M. Covarrubias, Utilization of limb orientations for a home-based immersive virtual reality rehabilitation system to treat phantom limb pain, Lecture Notes in Computer Science, pp. 484–488, 2018. 12. R. Malaka, "How computer games can improve your health and fitness, Games for Training, Education, Health and Sports, pp. 1–7, 2014. 13. Montoya MF, Muñoz JE, Henao OA. Enhancing virtual rehabilitation in upper limbs with biocybernetic adaptation: the effects of virtual reality on perceived muscle fatigue, game performance and user experience. IEEE Trans Neural Syst Rehabil Eng. 2020;28(3):740– 747. doi: 10.1109/TNSRE.2020.2968869. 14. Kim, A., Schweighofer, N. & Finley, J.M. Locomotor skill acquisition in virtual reality shows sustained transfer to the real world. J NeuroEngineering Rehabil 16, 113 (2019). https://doi.org/10.1186/s12984-019-0584-y 15. Rose FD, Attree EA, Brooks BM, Parslow DM, Penn PR, Ambihaipahan N. Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergonomics. 2000 Apr;43(4):494-511. doi: 10.1080/001401300184378. PMID: 10801083. 16. Oncioiu I., Priescu I., The Use of Virtual Reality in Tourism Destinations as a Tool to Develop Tourist Behavior Perspective. Sustainability. 2022; 14(7):4191. https://doi.org/10.3390/su14074191. 17. Demeco A, Zola L, Frizziero A, Martini C, Palumbo A, Foresti R, Buccino G, Costantino C. Immersive Virtual Reality in Post-Stroke Rehabilitation: A Systematic Review. Sensors. 2023; 23(3):1712. https://doi.org/10.3390/s23031712 18. Dechsling A, Orm S, Kalandadze T, Sütterlin S, Øien RA, Shic F, Nordahl-Hansen A. Virtual and Augmented Reality in Social Skills Interventions for Individuals with Autism Spectrum Disorder: A Scoping Review. J Autism Dev Disord. 2022 Nov;52(11):4692-4707. doi: 10.1007/s10803-021-05338-5. Epub 2021 Nov 16. PMID: 34783991; PMCID: PMC9556391. 19. Rizzo, A. A., & Kim, G. J. (2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence: Teleoperators and Virtual Environments, 14(2), 119-146. doi: 10.1162/1054746053967094 20. L. Y. Liu, S. Sangani, K. K. Patterson, J. Fung and A. Lamontagne, "Real-Time AvatarBased Feedback to Enhance the Symmetry of Spatiotemporal Parameters After Stroke: Instantaneous Effects of Different Avatar Views," in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 4, pp. 878-887, April 2020, doi: 10.1109/TNSRE.2020.2979830. 21. M. Masmoudi, d. E. Benrachou, o. Djekoune and e. Al, "Avatar-facilitated therapy and virtual reality: Next-Generation of Functional Rehabilitation Methods," in 2020 1st