Design and development of biomedical applications for post-stroke rehabilitation subjects

Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) Design&and&Development&of&Biomedical& Applications&for&Post-Stroke&Rehabilitation& Subjects& Jordi&Torner&Ribé,&Gil&Serrancolí,&Inés&Ayestaran&Arriaga,& Francesc& Alpiste,& Andrea& Iriarte,& Carles& Margelí,& Berta& Mayans,& Andrés&García,&Sergio&Morancho,&Cristina&Molas& EEBE.&Departament&d’Enginyeria&Gra0 fica&i&de&Disseny&(UPC)& Abstract Virtual Reality (VR) is having an increasing influence in fields such as education, marketing or scientific research. Thus, biomedical and health applications are also improving due to advances in Virtual Reality. The aim of this project is to develop computational applications focused on the rehabilitation of different parts of the body. These applications assist the motor and cognitive rehabilitation of post-stroke subjects. To this end, indications provided by the specialists of our collaboration center, the Functional Diversity Association of Osona (ADFO), have been taken into account. This association is planning to use the developed applications in their rehabilitation processes in order to make them less monotonous and more interactive. The telerehabilitation system consists of a depth camera to track the user’s movements and a set of computational applications. There are two types of applications: a set of rehabilitation exercises of individual movements and a set of serious games. The user will combine sessions with both sets of applications during the rehabilitation process. The test of applications with post-stroke subjects will be performed in the near future. Some results have been obtained testing the system on healthy people, and based on those results, corrections have been applied. This project aims to provide more playful rehabilitation exercises and, therefore, a more enjoyable and effective rehabilitation process. 159 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) 1. Introduction The estimated number of stroke-affected people per year in Spain is 110.000 (Agencia EFE, 2021). Around 75% of affected people by stroke are over 65 years (although the age of affected people is decreasing). Of these, 41.5% will suffer moderate or severe physical damages, and during the first year, 16% will die (Neural, 2021). Life expectancy after a stroke depends on many factors. The main ones are the age and the severity of the injury. Other factors, such as chronic diseases, will also play a role. Most of the motor and cognitive recovery can be achieved during the first six months after the injury (Chamelian, 2004; Laver et al., 2017). However, after this period, active rehabilitation maintenance is also crucial to avoid losing recovery and maintain patient independence. Therefore, carrying a rehabilitation program according to the patient’s needs will be essential. Here is where technology can help to make the patient’s lives and the work of clinicians easier. Virtual Reality (VR) applications have been developed in different fields of medicine: post-stroke rehabilitation (Laver et al., 2017; Howard, 2017; Gibbons et al., 2016), pediatric rehabilitation (Olivieri et al., 2018), physical rehabilitation (Borrego et al., 2016), cognitive training (Hill et al., 2017), neurocognitive diagnosis (Zygouris et al., 2017), traumatic brain injury rehabilitation (Aida et al., 2018), mental health (Pla-Sanjuanelo et al., 2019), pain management (Dascal et al., 2017; Pourmand et al., 2017). Furthermore, VR and Augmented Reality (AR) are used in surgery procedures (Robison et al., 2011; Aïm et al., 2016) and other medical training programs (Gout et al., 2020). The main benefits of VR applications over other treatments are their non-invasiveness, low cost, the facility to engage and motivate the user based on the gamification, the easy portability for home-based treatments, and the facility of providing feedback to patients and reporting data to therapists (Gibbons et al., 2016; Domínguez-Téllez et al., 2020). Although there is no unanimity regarding the effectiveness of their use, there is a willingness to continue evaluating and generalizing their utilization (Rose et al., 160 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) 2018). There is a consensus that virtual environments stimulate enjoyment and motivation in rehabilitation tasks (Neural, 2021). According to Pietrzak et al. (2014), videogames based on VR systems can be incorporated into post-stroke rehabilitation to make the treatments easier for both clinical-based and home-based treatments. Moreover, therapists could adapt individualized VR games based on the patient’s clinical needs by modifying the type, accuracy, number, frequency, duration and difficulty level. The computational application can also provide personalized and positive feedback (Lin et al., 2013) to enhance the well-being of healthy and injured adults (Montana et al., 2020). Traditional rehabilitation is often associated with boring processes and repetitive exercises, which can reduce patients’ motivation over time. Additionally, it requires (at least) one therapist to treat each patient individually, increasing the need for resources and, therefore, the staff and the healthcare system’s costs. These visits also involve unnecessary movements by the injured subject and the familiars. Many researchers combine the principles of VR with other technologies to treat chronic stroke survivors with any impairment (Levin et al., 2012). Over time, VR has shown to improve patients’ ability to imagine and move the rehabilitation field in different directions. VR is compatible with traditional therapy and is low-cost. It offers the possibility to experience psychological states of immersion and involvement and gives rise to a sense of presence, even in physically impossible situations (Blake et al., 2005). This feature allows personalizing the experience and improving the system’s therapeutic function, providing graded and quantified rehabilitation activities that can be individualized (Yates et al., 2016). This project aims to develop and validate a set of computational applications based on non-immersive VR. With only a depth camera (Intel Real Sense 415, Intel, California, USA) and a regular computer, the user can follow the rehabilitation treatment from home while feeling that he is playing and entertaining himself. We have developed a set of basic 161 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) movement exercises and a set of serious games. In the first ones, the user has to repeat the movements of a virtual avatar. He also can see his movements in a virtual mirror. In the serious games, the user has to achieve a certain objective in daily life situations, like catching balls as a goalkeeper, cooking a pizza, or cleaning a bathroom. 2. Methodology This section describes the followed methodology and the technological resources used during the project’s development. Overall, the project has been developed in Unity to develop the computational environment. The Software Development Kit (SDK) of Nuitrack has been used to treat the data coming from the Real Sense 415, and Navicat has been used to create the database. Unity allows the creation of both 2D and 3D games as well as virtual reality games and augmented reality, in addition to having other functionalities such as the possibility of including sounds or having a physical engine that simulates the laws of physics. The “scene” is one of the most important windows in the Unity editor because it contains all the video game objects. It is where the developer can move and manipulate all the objects. A videogame can contain several scenes that can be modelled according to the needs. For example, each video game level can be considered a scene. One important aspect of the game window is the view from where the user will perceive the scene. Each scene can have one or more virtual cameras placed around the scene, to be visualized from different angles. All objects contained (or potentially to be contained) in the scenes are listed in a hierarchy window, which allows to identify and link the objects easily. Unity uses different elements to create a scene, some components called assets are prefabricated. Assets can be created from Unity elements or imported from external files. Unity has a web library with free and paid assets, called Asset Store, accessible from the program itself or from 162 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) the Internet. Most objects used in this project were freely obtained from the web library. Scripting is an essential ingredient in all video games, from the simplest to the most complex. They define the behaviour of the components in the scene. They are used to respond to the player’s input, create graphic effects, control the behaviour of the components in a scene, etc. Unity offers the possibility of developing code in JavaScript, C++ or C#, the latter being the most used by the video game development community (due to the large number of functions that facilitate programming). The entire project has been developed in C#, and compiled in VisualStudio within Unity. In this framework, an Intel Real Sense D415 camera (Figure 1) is capturing the movement of the user, and Nuitrack SDK (also C# based) is used to obtain the skeleton of the subject. This camera has a depth field of view (FOV) of 64º x 41º (horizontal by vertical) with a resolution of 1280 x 720 pixels. Its red, green, blue (RGB) FOV is the same, 64º x 41º, and the RGB resolution is up to 1920 x 1080 pixels. The depth frame rate could be up to 90 fps. Nuitrask SDK tracks a full-body skeleton with 19 body landmarks, though in this project 17 were used (hand position was neglected, the arm segment ends with the wrist point). The skeleton is captured at each frame. Nuitrack can report the angles needed to orient the segments in the space. However, angles indicated by the therapists were calculated 163 Figura 1. Intel® Depth Sensor Camera RealSense™ D415 (Intel, 2021) Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) (shoulder flexion, vertical and horizontal shoulder adduction, elbow flexion, hip and knee flexion). Navicat Premium is a database development tool that allows simultaneously connecting to MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases from a single application. It can be used to quickly and easily build and maintain remote databases. In this project, Navicat has been used to register the users and record scalar data for each session, such as the number of repetitions of the exercise, the duration, the range of the movement, or a quantified indicator of the performance. Apart from the scalar data, the positions of the skeleton’s points and the angles are exported to a server in ASCII format. 3. Applications design Six exercises of simple joint movements and seven serious games have been integrated under a single Unity executable application. The user has to login either as a therapist to register subjects or as a subject to start using the application (Figure 2a). The six exercises (Figure 2b) consist of shoulder flexion, vertical and horizontal shoulder adduction, elbow flexion, body swing in the medial-lateral direction, and a neutral exercise. In the first five exercises, a virtual avatar guides the user with the movement to be performed. In turn, the user can see his movement reflected in a virtual mirror. The user can choose to perform these exercises by a certain time duration or by the number of repetitions. Figure 3 shows the environment with the virtual trainer avatar (left) and the virtual mirror avatar (right). The traffic light starts in red, while the virtual trainer performs some repetitions as example, and then when it turns green, the user can start performing the exercise. 164 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) The neutral exercise (Figure 2b) will only be used during the assessment of the intervention treatment. The physiotherapist will guide the user to perform some exercises and the “Neutral exercise” will store the trajectory of the keypoints of the body skeleton. These data will be processed a posteriori. 165 Figure 2. Main panels of the application. a) User login. b) Exercise selection. c) Serious game selection. Figure 3. Example of the shoulder flexion exercise. Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) As for the games (serious games, Figure 2c), there are seven, with different topics. Four of them are intended for the user to move upper limbs, one to move lower limbs, and two to perform medial-lateral swing. Clean the bathroom place the user in a bathroom and he/she has to clean the mirror using the arms (Figure 4a). Two games can be played, one where the user must follow a certain path and sweep some checkpoints. The other one consist of sweeping the whole mirror to remove the bath steam. The Kitchen game has three subgames based on the movement of the arms (Figure 4b). The user must cut some vegetables in slices (shoulder and elbow flexion), introduce some ingredients in a bowl (movement to bring the hand near the face) and introduce the pizza to the oven (shoulder and elbow movements). Clean the horse consists of sweeping a horse following some predefined geometric curves (Figure 4c), while moving the upper limbs. GoalKeeper place the user in a soccer field and needs to perform shoulder flexions to stop the balls (Figure 4d). The Imbalance game consists of performing medial-lateral movements with the whole body with the purpose of collecting coins and avoid obstacles (Figure 4e). It has three subgames, with different environments (forest, road and under the sea). In PickApples, the user needs to perform the same movement as in the game Imbalance, but with the goal to collect apples that are falling from an apple tree (Figure 4f). In the Slingshot game, the user needs to perform hip and knee flexions to shot ballons (Figure 4g). At the beginning, the user is asked to record the maximum height of the knee, to define the height of the shots. There are different levels according the height of the ballons. 166 Recerca i Tecnologia en Enginyeria Gràfica i Disseny a la UPC (Volum 2) 4. Conclusions and future lines This project aims to implement non-immersive virtual reality in the rehabilitation treatment of post-stroke subjects to maintain and improve their motor skills. To this end, several games and exercises have been developed under a single computational application in collaboration with 167 Figure 4. Seven serious games of the application. a) Clean the bathroom, b) Kitchen, c) Clean the horse, d) Goalkeeper, e) Imbalance, f) PickApples, g) Slingshot.