A new approach to study gait impairments in Parkinson’s disease based on mixed reality

October 2022 Beatriz Miranda A new approach to study gait impairments in Parkinson’s disease based on mixed reality Beatriz Maria Redondo Miranda A new approach to study gait impairments in Parkinson’s disease based on mixed reality October 2022 i Beatriz Maria Redondo Miranda A new approach to study gait impairments in Parkinson’s disease based on mixed reality Master dissertation Master Degree in Biomedical Engineering Medical Electronics Dissertation supervised by Professor Doctor Cristina Manuela Peixoto dos Santos October 2022 DIREITOS DE AUTOR E CONDIÇÕES DE UTILIZAÇÃO DO TRABALHO POR TERCEIROS Este é um trabalho académico que pode ser utilizado por terceiros desde que respeitadas as regras e boas práticas internacionalmente aceites, no que concerne aos direitos de autor e direitos conexos. Assim, o presente trabalho pode ser utilizado nos termos previstos na licença abaixo indicada. Caso o utilizador necessite de permissão para poder fazer um uso do trabalho em condições não previstas no licenciamento indicado, deverá contactar o autor, através do RepositóriUM da Universidade do Minho. Atribuição-NãoComercial-SemDerivações CC BY-NC-ND https://creativecommons.org/licenses/by-nc-nd/4.0/ ii AGRADECIMENTOS Esta dissertação, desenvolvida durante o último ano, foi o resultado de muito trabalho e esforço que nunca poderia ter acontecido sem a contribuição direta e indireta de muitas pessoas que me são muito queridas, restando-me agradecer-lhes. Em primeiro lugar, quero agradecer à minha orientadora, Professora Cristina, pela oportunidade que me deu em trabalhar neste projeto, por toda a motivação e orientação que foi dando, e ainda por todas as reuniões de trinta minutos que se prolongavam por duas horas, porque nem tudo é trabalho. Admiroa como profissional, mulher e mãe. Estou extremamente agradecida por toda a sua dedicação e suporte. De seguida, a Helena. Pessoa que mais me acompanhou durante este ano e cérebro do projeto +sense. Quero agradecer-lhe por todas as horas em chamada mesmo após termos passado o dia no laboratório, por me ter ajudado na recolha de dados no hospital, por me ensinar literalmente tudo, até receitas de bolo da caneca. A paciência, ajuda, dedicação e motivação mostradas não são mensuráveis. À minha parceira de mestrado Marta, o 4º ano foi uma batalha vencida ao lado dela. Agradeço pelas incontáveis noites a trabalhar e a cantar e as terças-feiras no laboratório. A todos os meus amigos da universidade, agradeço não só a companhia nas aulas mais dolorosas como todas as brincadeiras, saídas à noite e por me terem dado a conhecer mais deles e das terras deles. Agradeço também às minhas colegas de casa por ouvirem todos os meus dramas e à melhor pessoa que a universidade me deu, a minha afilhada, Maria Pimenta, por ser tão disponível e tão amiga. A todos os meus “amigos de ponte”, Bruna, Guida, Jucas, Caroli, Inês, Nelson, Gina, Luces, Jaime, obrigada por me acompanharem desde o secundário, por todos os verões, por todos os cafés de sábado à noite e por serem sempre um porto seguro. Apesar de nos conhecermos há muitos anos, ensinam-me sempre uma coisa nova todas as semanas, nem que seja os novos sabores da Água das Pedras. Não menos importante, tenho de agradecer ao Spotify, às suas playlists e artistas, por terem sido os meus melhores amigos calmantes durante este período. A música sempre será um pedaço de mim, mesmo que tenha seguido o mundo da engenharia. Por fim agradeço à minha família, aos meus pais e também à Lua, por serem o meu maior suporte financeiro e emocional, por me terem dado as melhores condições que podia pedir, pelo interesse que mostram pela minha área mesmo que não percebam sempre. E também ao meu irmão por me emprestar equipamentos e por me ajudar quando a tecnologia não quer ser minha aliada. Muito obrigada a todos. Bia iii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the University of Minho. iv ABSTRACT Parkinson’s disease (PD) is the second most common neurodegenerative disorder after Alzheimer's disease. PD onset is at 55 years-old on average, and its incidence increases with age. This disease results from dopamine-producing neurons degeneration in the basal ganglia and is characterized by various motor symptoms such as freezing of gait, bradykinesia, hypokinesia, akinesia, and rigidity, which negatively impact patients’ quality of life. To monitor and improve these PD-related gait disabilities, several technology-based methods have emerged in the last decades. However, these solutions still require more customization to patients’ daily living tasks in order to provide more objective, reliable, and long-term data about patients’ motor conditions in home-related contexts. Providing this quantitative data to physicians will ensure more personalised and better treatments. Also, motor rehabilitation sessions fostered by assistance devices require the inclusion of quotidian tasks to train patients for their daily motor challenges. One of the most promising technology-based methods is virtual, augmented, and mixed reality (VR/AR/MR), which immerse patients in virtual environments and provide sensory stimuli (cues) to assist with these disabilities. However, further research is needed to improve and conceptualize efficient and patient-centred VR/AR/MR approaches and increase their clinical evidence. Bearing this in mind, the main goal of this dissertation was to design, develop, test, and validate virtual environments to assess and train PD-related gait impairments using mixed reality smart glasses, integrated with another high-technological motion tracking device. Using specific virtual environments that trigger PD-related gait impairments (turning, doorways, and narrow spaces), it is hypothesized that patients can be assessed and trained in their daily challenges related to walking. Also, this tool integrates on-demand visual cues to provide visual biofeedback and foster motor training. This solution was validated with end-users to test the identified hypothesis. The results showed that, in fact, mixed reality has the potential to recreate real-life environments that often provoke PD-related gait disabilities, by placing virtual objects on top of the real world. On the contrary, biofeedback strategies did not significantly improve the patients’ motor performance. The user experience evaluation showed that participants enjoyed participating in the activity and felt that this tool can help their motor performance. Keywords: Parkinson’s disease; Virtual reality; Augmented reality; Mixed Reality; Rehabilitation; Gait disabilities; Sensory cueing; Biofeedback. v RESUMO A doença de Parkinson (DP) é a segunda doença neurodegenerativa mais comum depois da doença de Alzheimer. O início da DP ocorre, em média, aos 55 anos de idade, e a sua incidência aumenta com a idade. Esta doença resulta da degeneração dos neurónios produtores de dopamina nos gânglios basais e é caracterizada por vários sintomas motores como o congelamento da marcha, bradicinesia, hipocinesia, acinesia, e rigidez, que afetam negativamente a qualidade de vida dos pacientes. Nas últimas décadas surgiram métodos tecnológicos para monitorizar e treinar estas desabilidades da marcha. No entanto, estas soluções ainda requerem uma maior personalização relativamente às tarefas diárias dos pacientes, a fim de fornecer dados mais objetivos, fiáveis e de longo prazo sobre o seu desempenho motor em contextos do dia-a-dia. Através do fornecimento destes dados quantitativos aos médicos, serão assegurados tratamentos mais personalizados. Além disso, as sessões de reabilitação motora, promovidas por dispositivos de assistência, requerem a inclusão de tarefas quotidianas para treinar os pacientes para os seus desafios diários. Um dos métodos tecnológicos mais promissores é a realidade virtual, aumentada e mista (RV/RA/RM), que imergem os pacientes em ambientes virtuais e fornecem estímulos sensoriais para ajudar nestas desabilidades. Contudo, é necessária mais investigação para melhorar e conceptualizar abordagens RV/RA/RM eficientes e centradas no paciente e ainda aumentar as suas evidências clínicas. Tendo isto em mente, o principal objetivo desta dissertação foi conceber, desenvolver, testar e validar ambientes virtuais para avaliar e treinar as incapacidades de marcha relacionadas com a DP usando óculos inteligentes de realidade mista, integrados com outro dispositivo de rastreio de movimento. Utilizando ambientes virtuais específicos que desencadeiam desabilidades da marcha (rodar, portas e espaços estreitos), é possível testar hipóteses de que os pacientes possam ser avaliados e treinados nos seus desafios diários. Além disso, esta ferramenta integra pistas visuais para fornecer biofeedback visual e fomentar a reabilitação motora. Esta solução foi validada com utilizadores finais de forma a testar as hipóteses identificadas. Os resultados mostraram que, de facto, a realidade mista tem o potencial de recriar ambientes da vida real que muitas vezes provocam deficiências de marcha relacionadas à DP. Pelo contrário, as estratégias de biofeedback não provocaram melhorias significativas no desempenho motor dos pacientes. A avaliação feita pelos pacientes mostrou que estes gostaram de participar nos testes e sentiram que esta ferramenta pode auxiliar no seu desempenho motor. Palavras-Chave: Doença de Parkinson; Realidade virtual; Realidade aumentada; Realidade mista; Reabilitação; Desabilidades motoras; Pistas sensoriais; Biofeedback. vi Table of Contents 1 Introduction .............................................................................................................. 1 1.1 Motivation ................................................................................................................... 2 1.2 Problem statement ...................................................................................................... 4 1.3 Goals ........................................................................................................................... 5 1.4 Research Questions ...................................................................................................... 6 1.5 Contributions to Knowledge ......................................................................................... 6 1.6 Dissertation Structure .................................................................................................. 7 2 Literature Review ...................................................................................................... 8 2.1 Introductory Insight...................................................................................................... 9 2.2 Methods .................................................................................................................... 10 2.2.1 Data sources, search strategy and studies selection ....................................................... 10 2.3 Results ....................................................................................................................... 10 2.3.1 General Results ................................................................................................................ 10 2.3.2 VR/AR/MR in PD .............................................................................................................. 11 2.3.3 Technology supporting VR/AR/MR-based approaches in PD .......................................... 13 2.3.4 Validation methodology highlights: participants, criteria study, setting, protocols, Schedule, metrics.......................................................................................................................... 16 2.4 Discussion .................................................................................................................. 27 2.4.1 How have the VR/AR/MR-based approaches been applied in PD to help patients mitigate gait disabilities? .............................................................................................................. 27 2.4.2 Which technologies have been used to support VR/AR/MR-based approaches in PD? . 27 2.4.3 How have the VR/AR/MR-based approaches been clinically validated in PD? ............... 28 2.5 Conclusions and Future directions .............................................................................. 29 3 Solution Overview ................................................................................................... 32 3.1 Problem description ................................................................................................... 33 3.2 +sense ....................................................................................................................... 33 3.3 +sImmersive ............................................................................................................... 34 3.3.1 Mixed reality smart glasses: Microsoft HoloLens 2 ......................................................... 35 3.3.2 Motion tracking system: Xsens MVN Awinda ................................................................. 37 3.4 Conclusions ................................................................................................................ 39 4 Solution description ................................................................................................ 40 xiii M MMSE Mini-Mental State Examination MoCA Montreal Cognitive Assessment MiniBESTest Mini-Balance Evaluation Systems Test MR Mixed Reality M1 Monitoring test 1 - dice M2 Monitoring test 2 - door M3 Monitoring test 3 – narrow spaces P PD Parkinson’s Disease PIGD Postural Instability and Gait Disorder PTF Percentage of time frozen PC Personal computer PD-f Parkinson’s Disease patients with freezing of gait (freezers) PDQL Parkinson’s Disease Quality of Life questionnaire Q QoL Quality of Life S SI Semi immersive SSQ Simulator Sickness Questionnaire SUS System Usability Scale SAC Stress Arousal Checklist T T Training TUG Timed Up and Go test TC1 Control test 1 TC2 Control test 2 T1 Training test 1 – dice with arrows T2 Training test 2 – door with footprints T3 Training test 3 – narrow spaces with footprints U xiv UPDRS Unified Parkinson’s Disease Rating Scale UPSRS-III Unified Parkinson’s Disease Rating Scale part III V VR Virtual Reality VGoT Videogame-oriented training 1 INTRODUCTION 2 This dissertation presents the work carried out over the past year, integrated in the scope of the Master Degree in Biomedical Engineering at the Biomedical Robotic Devices Lab (BiRDLAB) included in the Centre of MicroElectroMechanical Systems (CMEMS), a research centre of the Department of Industrial Electronics (DEI) of University of Minho. The project main goal was to develop and validate a mixed reality (MR) tool for the assessment and training of gait disabilities in Parkinson’s disease. This solution was developed to bring a new paradigm shift. Thus, by triggering PD-related gait impairments, patients can be assessed and trained in everyday situations. The potential of mixed reality, integrated with a motion tracking system, to mimic everyday environments, was tested, aiming a more objective medical assessment, and enhanced and motivational rehabilitation exercises. All the steps performed to achieve this solution are detailed in this document. 1.1 MOTIVATION Parkinson’s disease (PD) is the second most common neurodegenerative disorder after Alzheimer's disease [1]. It is believed that PD physiopathology lies in the loss of dopamine-generating neurons in the basal ganglia, which is related to human movement control. The first dyskinesias appear when there is a deficiency in dopamine release by these cells [1], [2]. Its average onset is at 55 years-old and its incidence increases with age [1]. This disease is characterized by several symptoms with gait impairments being the most common and disabling ones. Motor symptoms include freezing of gait (FoG), bradykinesia (movement slowness), hypokinesia (reduced movement amplitude), akinesia (problems initiating movement), festination (tendency to speed up when performing repetitive movements), rigidity, postural instability, and reduced movement automaticity, all of them diminishing patients’ quality of life [1]–[7]. Usually, patients start by reducing their walking velocity, taking shorter steps, and presenting some gait asymmetries and eventually suffering motor freezing events. FoG is defined as the “ sudden inability to continue walking despite the intent to maintain locomotion” and “it is episodic and variable by nature ”, being one of the most debilitating and difficult impairments to assess [3], [4], [7]. FoG events are typically triggered by specific situations, such as initiation of walking, turning during steady-state walking, facing objects, stress, distraction, and nearing doorways. Even if these environments did not trigger a complete moment of gait blockage, they contributed to a decrease in the step length and velocity, occurring festination and akinesias. Moreover, these gait disabilities can be aggravated by dual-task attentional requirements [2], [3], [6]–[8]. 3 Motion tracking systems make it possible to monitor patients’ motor function using wearable sensors [9], while electromyography systems acquire electrical muscle activity, reflecting motor fluctuations [10]. Both systems are at the forefront of motor function’s monitoring, enabling to gather data with low-cost, portable, and miniaturized sensors. It has been possible to monitor kinematic and electromyography-driven information about patients’ motor conditions, such as their gait spatiotemporal parameters [2], [11], [12], FoG events duration and occurrence [2], [13]–[15], muscles activity [10], or postural changes [11], [12]. Indeed, this information represents trivial data for physicians to monitor, over time, the motor state of their patients, controlling the progression of the disease, especially if they can access these data collected on patients’ home environments, during their daily tasks. This would result in greater supervision of the development of the disease and would allow treatments to be personalised to the patient. Despite the continuous progress of these technological solutions in the continuous monitoring of PD-associated gait disabilities, it is still difficult to feasibly assess and train patients for their common daily tasks. Several researchers have dedicated themselves to the study of the neurological origin of these impairments in order to customize treatment methodologies for PD-related gait disabilities [16]. Despite the scientific advances, different hypotheses are still pointed out. It is only known that there may be a failure in the activity of nerve messages between the central nervous system and the efferent muscles, responsible for movement, that can cause motor symptoms in PD [17]– [19]. To overcome these motor symptoms, researchers pointed out cueing-based interventions [5]. These strategies involve the use of external temporal or spatial stimuli to facilitate movement, in the form of visual, auditory and vibrotactile cues. These cues contribution consists in bypassing faults in nervous messages that may be at the origin of gait impairments [17]–[19]. Indeed, these sensory cueing strategies are integrated into biofeedback devices, which have already been explored in the PD field. These systems make use of wearable technology to provide sensory acquisition and trigger a cue information (biofeedback). They can detect a decrease in cadence or a change of the lower leg muscle activity, and through the detection of such motor behaviors deliver sensory cues [8], [17]. Thus, these interventions could lead to a change in postural control, step pattern, and unfreeze gait freezing events, prevent falls, and consequently could promote less variability in gait and a more goal-oriented gait. Further, wearable systems allow their integration into patients’ everyday tasks, ensuring greater freedom of movement and comfort [9]. However, to the best of knowledge, the effectiveness of these technologies for rehabilitation has seldom been investigated and validated in real-life situations. Thus, the use of virtual environments to immerse patients in those situations could potentiate 4 the biofeedback interventions. Further, these studies did not follow a patient-centred approach, some did not use fully wearable systems and did not include modular systems, meaning they are not easy to integrate with other technologies [2], [17]. Lastly, patients and physicians were rarely included in the development phase and there were very few accurate and objective evaluation metrics, plus it was unusual to use functional movement training. Virtual, augmented, and mixed reality emerged in the last decades as a promising strategy to allow patients’ immersion in customized virtual environments. In fact, these personalised virtual and interactive environments allow patients to be placed in situations where they can perform daily tasks, obtaining more feasible and natural motion data. When a modular development architecture is used, this VR/AR/MR equipment may be integrated with other monitoring and actuation devices, fostering patients’ motor assessment and training. In this sense, in order to overcome the limitations encountered, this dissertation aims to explore the use of mixed reality (MR) in the assessment and training of PD patients during their motor tasks. By developing a robust system of MR integrated with a high technological motion tracking device, capable of assessing PD-related gait disabilities, it is expected to show how immersive, interactive MR technology can offer a new methodological framework for monitoring and training gait-related behaviours in PD. 1.2 PROBLEM STATEMENT More reliable assessment and consistent training geared towards daily tasks are needed, and this can be achieved by immersing PD patients in virtual environments to improve their motor function assessment and training. It is expected to (i) explore the use of MR to develop and design virtual environments closer to patients’ daily reality; (ii) develop a modular architecture capable of integrating the MR approach with a motor assessment device; (iii) investigate the potential of sensory cues in improving gait impairments through augmentative cues. To address these problem statements, it is crucial to follow a user-oriented approach capable of developing a motor assessment and training strategy closer to patients’ daily needs. This dissertation will adopt a systematic approach to answer these key constraints. 5 1.3 GOALS The ultimate goal of this dissertation was to design, develop, test, and validate three different virtual environments, which recreate everyday situations, to assess and train PD-related gait disabilities using MR smart glasses and a motion tracking system. One of the most distinguishable features of this dissertation is its multi-disciplinary nature spanning from neurosciences to algorithms for MR, modular architectures, wearable sensors, and biofeedback strategies. Thus, this work required dealing with existing and front-end hardware, designing virtual environments, gait analysis and segmentation, validating protocols with end users, and data analysis. To reach this main goal, the following step-goals and Key Performance Indicators (KPI) needed to be defined and achieved: Goal 1: Gather knowledge about VR/AR/MR strategies used in PD for motor training and assessment, through literature reviews, to answer the following questions: (i) “How have the VR/AR/MR-based approaches been applied in PD to help patients mitigate gait disabilities?”; (ii) “Which technologies have been used to support VR/AR/MR-based approaches in PD?”; and (iii) “How have the VR/AR/MR-based approaches been clinically validated in PD?”. This goal relates to KPI 1: summarising the literature through at least twelve articles; what are the most common gait disabilities in PD; what are the real-world situations that most cause PD-related gait disabilities. Chapter 2 presents these surveys. Goal 2: Implementation of a modular, user-customised, technological solution based on mixed reality to immerse patients in scenarios that can trigger PD-related gait disabilities. Based on the literature review, virtual environments that evoke PD-related gait disabilities and that are customisable according to the users' height will be developed. This will address the limitations identified in VR/AR/MR-based approaches in PD. This goal relates to KPI 2: development of three virtual environments that represent situations that typically cause PD-related gait disabilities. Chapter 3 and Chapter 4 describe the materials and procedures of this solution. Goal 3: Implementation of a modular, user-customised, technological solution based on mixed reality integrated with another high-technological motion tracking device to help patients overcome PD-related gait disabilities. Outcomes cover the integration of a motion tracking system with MR technologies based on combined visual sensory cues, motion analysis and augmented reality. This goal relates to KPI 3: development of on-demand visual biofeedback strategies integrated in HoloLens 2; development of a real-time initial and final contact detection algorithm with a performance higher than 96% for accuracy. Chapter 3 and Chapter 4 detail the materials, methods and algorithms used and developed to achieve this solution. 6 Goal 4: Validation of the proposed MR strategy with end-users. It is intended to collect and analyse data to assess the usability, efficiency, and acceptability of implemented strategies (user-centred approach). This goal relates to KPI 4: validation of the solutions with at least ten end-users; statistically significant differences in spatiotemporal parameters between control and monitoring tests, and later between monitoring and training tests; SSQ score lower than sixteen points; IMI score greater than five points. Chapter 5 will reveal the obtained results. 1.4 RESEARCH QUESTIONS Considering the ultimate goal of this dissertation and the step-goals presented, relevant research questions (RQs) were identified, as follows: RQ 1: How have the VR/AR/MR-based approaches and technologies been applied to support PD patients and how have they been clinically validated? This question relates to Goal 1 and is answered in Chapter 2. RQ2: How to implement a modular, user-customised, mixed reality-based technology solution that immerses patients in environments that (1) cause PD-related gait impairments; and that (2) help overcome these impairments with the aid of a motion tracking system and biofeedback strategies? This issue considers Goal 2 and Goal 3. The answer is developed throughout Chapter 3 and Chapter 4. RQ3: How does the implemented modular technological solution, based on mixed reality integrated with a motion tracking system and with biofeedback strategies, affect the motor performance of PD patients during assessment and training? This question is linked to Goal 4 and is answered in Chapter 5. The presented RQs are summarized and answered in Chapter 6. 1.5 CONTRIBUTIONS TO KNOWLEDGE The main contributions of this dissertation to knowledge are: • Review on VR/AR/MR-based approaches currently deployed in PD to train and assess gait disabilities. • Development of three virtual environments and virtual tasks that cause gait impairments, customisable to each participant. 7 • Implementation and validation of an algorithm for detecting initial and final contacts, in real time. Also, in this scope, an algorithm to estimate spatiotemporal metrics was implemented, based on gait segmentation. • Development of two visual biofeedback strategies, customisable to each participant, to help overcome PD-related gait impairments. It is expected that the developed work will lead to the elaboration of a journal article. During this period, I had the privilege of guiding two students of the Integrated Masters in Electronic Engineering, in a project of the curricular unit "Projeto Integrador". In addition, I applied for a grant from the "Verão com Ciência" programme of the Fundação para a Ciência e Tecnologia (FCT). 1.6 DISSERTATION STRUCTURE This manuscript is organized into six chapters, as follows. Chapter 1 presents the motivation, problem statement and the ultimate goals of this work. Chapter 2 outlines a comprehensive review of current literature about VR/AR/MR-based approaches to study PD-related gait disabilities. The VR/AR/MR-based approaches are presented and discussed, regarding the VR/AR/MR technology, embedded sensors, virtual tasks, and clinical outcomes. The chapter finishes with a summary of the findings. Chapter 3 addresses the overview of the solution. It starts by describing the problem. Then, project +sense and its modules are presented, as well as a description of the hardware included in the strategy, mentioning its need and technical characteristics. Chapter 4 outlines the solution description. Firstly, an introductory insight is presented, describing the setup to be used. Secondly, the user-centred design of the solution is presented, identifying the virtual tasks and environments designed. Thirdly, the integration of the sensory system is described, starting by explaining the real-time initial and final contacts detection algorithm, up to the estimation of spatiotemporal metrics, performed offline. Finally, the integration of the biofeedback strategies is explained. Chapter 5 presents the validation protocol, results, and a critical discussion of the solution. Furthermore, it presents their limitations and possible explanations, along with research suggestions and improvements. Chapter 6 concludes the dissertation, while providing a brief analysis of the project and its results, along with future research insights. 2 LITERATURE REVIEW 15 It was observed that the VR/AR/MR equipment used by the selected studies were the Oculus Rift DK2 [2], [17], [28], HTC Vive [11], [12], [20], [25], Google Glass [13], [23], HoloLens [15], [18], and HTC Vive Pro [22], [26]. Furthermore, in [21] a micro display was attached to the eyeglasses frame and in [14] a prototype of custom-made smart glasses was designed. Finally, in [24] and [27] smart glasses were not worn, on the contrary, a computer assisted virtual reality environment (CAREN) and a C-Mill VR+ treadmill was used, respectively. The virtual environment immersion ranged from fully immersive [2], [11]–[15], [17], [18], [20]–[26], [28] to semi-immersive [27], being frequently used head-mounted displays. Virtual tasks included motor activities, like climbing stairs [14], turning [14], [15], and walking straight [11], [13], [14], [18], [21], [24], [27], [28], ( i.e., on a hallway as in [12], [17], [20]), or specific contexts that could trigger PD-related gait disabilities, such as crossing virtual [2], [12], [20] or real doors [13]. Additionally, when VR/AR/MR was applied for motor training strategies, a cue-oriented game was used in [22], while in [23]–[26] patients followed the tasks indicated on the virtual game, namely, dance [23], navigate a virtual boat [24], drive a ball to the finish line [24], smash flying objects [24], complete virtual words [25] or play a box game [26]. An overview of some virtual tasks and environments developed are shown in Figure 2-3. Regarding the acquisition module, three systems were found: InterSense IS900 [2], [17], Qualysis [20], IMU [13]–[15], accelerometer (G-Sensor) [24] and Vive trackers [22], [25], with none of them being built in. IMUs were placed in both full body [14], [15] and lower body [11], [13] configurations. With respect to the actuation module, it was identified the type and which device were used, and whether it was built in. All systems which had an actuation module used built-in actuators, such as augmentative visual cues or earphones [13]–[15], [17], [18], [21], [22], [24]. However, [15] also had a non-built-in actuator, namely, a speaker for auditory cueing. Furthermore, [24] used four types of actuators (visual, audio, vestibular and tactile), [13] used three types of actuators (audio, flashing light and optic flow), [15] used two types of actuators (visual and audio) whereas [14], [17], [18], [21], [22] only used one type of actuator, visual. Besides, visual cues were used more than auditory cues. 16 2.3.4 VALIDATION METHODOLOGY HIGHLIGHTS: PARTICIPANTS, CRITERIA STUDY, SETTING, PROTOCOLS, SCHEDULE, METRICS Table 2-3 summarizes the validation methodology of the selected studies. It highlights the participation and evaluation of PD patients, inclusion and exclusion criteria for participants selection, setting, experimental protocols, schedule, and the research evaluation metrics. Unified Parkinson’s Disease Rating Scale part III (UPDRS-III) [2], [12]–[15], [17], [18], [20], [23]– [25], [27], [28] and Hoehn and Yahr scale (H&Y) [11], [12], [14], [15], [18], [20], [21], [23]–[25], [27], [28] were the most commonly used rating scales for symptoms of PD. Along with UPDRS-III, Postural Instability and Gait Disorder sub-score (PIGD) [14], Activities-specific Balance Confidence (ABC) scale [11], [12], [22], [23], [28], Mini-Balance Evaluation Systems Test (Mini-BESTest) [11], [12], [20], [25], [28] and UPDRS-II [24] were used to reflect the evolution of motor function. To indicate the patients’ cognitive and mental state, the Mini-Mental State Examination (MMSE) scale was used in [11], [14], [15], [27], Montreal Cognitive Assessment (MoCA) in [12], [18], [20], [23], [28] and Frontal Assessment Battery (FAB) scale in [13]–[15]. FOG-questionnaire (FOG-Q) was used in [2], [12]–[15], [17], [18], [20]. To assess simulator sickness symptoms, the Simulator Sickness Questionnaire (SSQ) was used [25]. Figure 2-3 - Overview of some virtual tasks and environments taken from [2], [3], [12], [13], [19], [20], [22]–[26]. 17 Table 2-3 – Clinical Highlights of the developed VR/AR/MR technologies to PD patients, over the last ten years Goal Paper Participants Criteria Study Setting Protocol Schedule Metrics N Scales Inclusion Exclusion A [2] 10 - UPDRS-III; - FOG-Q; - - Laboratory Walk under 3 different virtual conditions: (1) no door; (2) narrow doorway; (3) standard doorway. Single visit: (1) familiarisation phase; (2) 18 trials (6x each condition). Total: 20min - step cadence (mean and CV); - step velocity (mean and CV); - step length (mean and CV); - duration of FoG episodes (mean and SD); - % trials with a FoG episode [11] 10 - UPDRS; - MiniBESTest; - ABC scale; - H&Y; - MMSE; - Diagnosis of PD; - - H&Y≤3; - MMSE > 24/30; - No other pathology interacting with gait or causing dizziness; - No uncorrected visual deficiency; - Ability to walk 512 consecutive strides (±10– 15 min); - Laboratory Walk in a randomized order in 3 conditions: (1) Overground Walking; (2) Treadmill Walking (3) immersive Virtual Reality on Treadmill Walking Single visit - speed; - step length; - cadence; - SSQ [12] 10 - MDSUPDRS-III; - NFoGQ; - MoCA; - Mini-BEST; - ABC Scale; - H&Y; - Diagnosis of PD; - Self-reported FoG; - Self-reported ability to walk 400m without assistance from a device or another person; - No diagnosis of dementia; - No uncorrected vision or hearing problems; - Laboratory Walk in 5 environments: (1) Physical laboratory without VR; (2) virtual laboratory without obstacles; (3) virtual doorway; (4) virtual hallway; (5) virtual street scene with crowds Single visit - gait speed; - step length (mean and CV); - step width; - step time; - step time asymmetry; - festination; - SSQ [20] 12 - MoCA; - NFOGQ; - Mini-BEST; - Diagnosis of PD without dementia; - Laboratory Walk under 4 conditions: (1) physical laboratory; (2) virtual laboratory; (3) virtual doorway; Single visit - kinematic variables; - gait speed; - step length (mean and CV); - step time; 18 - MDSUPDRS-III; - H&Y; - Self-reported or clinician-observed FoG; - Ability to walk 400 m without assistance from a device or another person; - No uncorrected vision or hearing deficits; (4) virtual hallway. - step time asymmetry; - step width; - DLS; - festination; - SSQ CoA [13] 12 “end-ofdose” - UPDRS-III; - NFOGQ; - FAB; - Presence of FoG more than twice per day; - Able to walk 20m over a flat surface without walking aids; - Significant cognitive impairments; - Comorbidities that impaired gait, or visual impairments; Laboratory Walk on 4 different walking courses in combination with 4 cueing conditions: (1) metronome; (2) flashing light; (3) optic flow; (4) no cue. Single visit: (1) familiarization phase; (2) 16 different cue-course combinations (2 trials each). Total: 2.5h - no. of FoG episodes; - duration of FoG episodes; - stride length (mean and SD); - speed; - cadence (mean and SD); - interview (user experience) [18] 24 “on state” - UPDRS-III; - H&Y; - MoCA; - NFOGQ; - Older than 18 years; - Diagnosis of PD; - Experience FOG in the dopaminergic “ON” state; - Additional neurological diseases and/or orthopedic problems; - Inability to walk independently; Home/Laboratory (1) HOME: walk a freezing provoking route multiple times with and without wearing the HoloLens (without Holocue); (2) LABORATORY: familiarize participants to walking with the (on-demand) holographic cues; (3) HOME: equal to session 1 but wearing the HoloLens with and without the Holocue 3 sessions of 1.5h, one week apart - no. of FoG episodes; - average duration of FoG episodes; - total duration of FoG episodes; - % time frozen (PTF) CoT [21] 20 - H&Y - - considerable visual deficit not compensated by correction; - ocular movement dysfunction; - gait disturbances due to neuromuscular diseases; Laboratory Walk a straight track of 10 m: (1) baseline; (2) online display off; (3) online display on; (4) residual effects; (5) examination; Single visit - speed; - stride length [14] 25 “end-ofdose” - UPDRS-III; - UPDRSPIGD; - Diagnosis of PD; - Older than 18 years old; - History of stroke; - Psychiatric disease; Laboratory Walk on 3 different walking courses in combination with 5 cue conditions: Single visit: - no. of FOG episodes; - % freezing time; - stride length (mean and SD); 19 - H&Y; - NFOGQ; - MMSE; - FAB; - Presence of FoG more than once per day; - Severe uncorrected visual or hearing impairments; - Comorbidity limiting ambulation; - Inability to walk unaided; - Deep brain stimulator or apomorphine pump; - Jejunal levodopa gel infusion; - MMSE score < 24; (1) augmented visual cue bars; (2) augmented visual cue staircases; (3) conventional 3D transverse bars on the floor; (4) metronome; (5) no cueing. 2 sessions separated by 30min break Total: 2.5h-3h - cycle time (mean and SD); - cadence; - speed; - interview (user experience) [17] 12 - UPDRS-III; - FOG-Q; - - Laboratory Walk using visual cues: (1) 2 spatial conditions: 115% and 130% of an individual’s baseline step length and; (2) 3 different temporal conditions: spatial only condition, 100 and 125% baseline step cadence. Single visit: (1) familiarisation phase; (2) 6 different cueing conditions (8x each condition). Total: 40min - Step length (mean and CV); - Step cadence (mean and CV); - Step velocity (mean and CV); (at baseline and post intervention) [15] 16 “end-ofdose” - UPDRS-III; - H&Y; - MMSE; - NFOGQ; - FAB; - Diagnosis of PD; - Presence of FoG more than twice per day; - MMSE score < 24; - FAB score < 13; - Comorbidity causing severe gait impairments; - Severe bilateral visual or auditory impairments; - Inability to perform a 180◦ turn unaided; Laboratory Perform a series of 180º turns under: (1) an experimental condition with AR visual cues and; (2) two control conditions: auditory cues and no cues. Single visit: (1) 1 training session: 3 blocks (15 trials each); (2) 2 experimental sessions: 3 blocks (15 trials each). - PTF; - no. of FoG episodes; - duration of FOG episodes; - cadence; - peak velocity; - stride time (mean and CV); - step height (mean and CV); - max head-pelvis separation; - time to max head-pelvis separation; - max medial CoM deviation; - turn time; - interview (user experience) [22] 5 - - Diagnosis of PD; - H&Y I-III; - Able to walk independently; - Conditions that could have affected exercise function; Laboratory Play the game “Treasure Island Adventure” with and without obstacles in combination with 3 levels: 35, 40, and 45cm between the visual cues 1 session per week of 30min, over 3 weeks - BBS; - ABC; - Step distance; - Leg raising; (at baseline and post intervention) 20 VgoT [23] 7 “ON” - UPDRS-III; - H&Y; - MoCA; - Diagnosis of PD; - H&Y > III; - MDS-UPDRS-III > 57; - Unable to wear or operate Google Glass; - Dementia; Home Complete at least 3 modules of MTG per day Every day for 3 weeks - Mini-BESTest; - one-leg stance; - TUG; - dual-task; - ABC scale; - BDI; - PDQL; - interview [24] 22 - UPDRS-II; - UPDRS-III; - H&Y; - MiniBESTest; - MMSE; - Diagnosis of PD; - H&Y≤3; - MMSE ≥ 24 - age > 85 years; - presence of severe medical and psychiatric illness potentially interfering with the VR training Laboratory Complete four scenarios: (1) Navigate a virtual boat through a slalom course; (2) walk across the board; (3) drive a red ball, moving the oad up to the finish line; (4) swat at flying objects that emerge along the path 20 conventional physiotherapy sessions + 3month rest + 20 sessios of CAREN training - BBS; - TUG; - UPDRS-II; - UPDRS-III; - FES-I; - H&Y; - 10MWT; - stride length; - cycle time; - stance phase/time; - swing phase/time; - percentage of singleand double-limb support; - speed; - cadence; - step length; - step width [25] 9 “ON” - UPDRS-III; - H&Y; - MiniBESTest; - SSQ; - Diagnosis of PD; - No motor fluctuations; - H&Y I-III; - Older than 18 years old; - Walking independently; - Stable medication; - Uncontrolled, involuntary movements (dyskinesia); - Musculoskeletal injuries; - Pain that limited movement; Laboratory Complete a puzzle that consisted of a word with missing letters located at eye level in the virtual environment 3 sessions of 30min each, over 1 week Total: 1h30 - SSQ; - ITC-SOPI; - IMI; - SUS; [26] 4 - - H&Y II; - Inability to correctly respond to the assessment protocol; - Presence of cardiovascular, pulmonary, or Laboratory Play the game BOX VR 2 sessions, 2 weeks apart; 1st session: (1) familiarization phase with Steam VR Home (9min); - SUS; - SSQ; - GEQ-post game; - interview (user experience) 21 musculoskeletal condition; - Presence of severe visual loss; - Vertigo, epilepsy, and psychosis; (2) training: game gym (3min); 2nd session: (1) familiarization phase with TheBlue; (2) training: game gym. [27] 29 PIGD + 23 nonPIGD 2h after medication - UPDRS-III; - H&Y; - MMSE; - Diagnosis of primary PD; - <75 years old - H&Y stage I–III (“on” period); - MMSE>24 (>20 for those with only primary school education); - Serious complications or comorbidities; - Special treatment required for other comorbidities; - Deep brain stimulation or in vivo implants; - Atypical or secondary PD; - Comorbidities affect walking; - severe cognitive, visual, and hearing impairment; - Using a psychotropic substance; Laboratory Complete 5 modules of C-Mill training in each training session 1 session of 30min per day for 7 days: (1) familiarization phase; (2) modules of CMill training. - 10-meter walking test; - TUG test; - BBS; - Posture sway; - Gait adaptability; - Borg 6-20 Questionnaire; - perceived risk of falling; - PDQL T [28] 11 “ON” - UPDRS-III; - H&Y; - MoCA; - ABC; - MiniBESTest; - Able to walk for 30 min on a treadmill; - 19<MoCA<30; - No other neurological disorders; - Laboratory Walk for 20 min on a treadmill while viewing a virtual city scene Single visit Total: 20min - CoP excursion; - SSQ; - SAC; (at baseline and post intervention) [Ref.]: study reference; A: assessment; CoA: cue-oriented assistance; CoT: cue-oriented training; VGoT: videogame-oriented training; T: training; UPDRS-III: Unified Parkinson’s Disease Rating Scale part III; FOG-Q: Freezinf of Gait questionnaire; FAB: Frontal Assessment Battery; H&Y: Hoehn and Yahr scale; MMSE: Mini-Mental State Examination; UPDRS-PIGD: Unified Parkinson's Disease Rating Scale - Postural Instability and Gait Disorder ; Mini-BesTest: Mini-Balance Evaluation Systems Test; SSQ: Simulator Sickness Questionnaire; MoCA: Montreal Cognitive Assessment; ABC: Activities-specific Balance Confidence scale; CV: coefficient of variation; SD: standard deviation; PTF: percentage time freezing; DLS: double limb support; CoM: Centre of Mass; BBS: Berg Balance Scale; ITC-SOPI: Independent Television Commission Sense of Presence Inventory; IMI: Intrinsic Motivation Inventory; SUS: System Usability Scale; GEQ-post game: Game experience questionnaire-post game; TUG: Timed Up and go Test; BDI: Beck Depression Inventory; PDQL: Parkinson's Disease Quality of Life questionnaire; CoP: Centre of Pressure; SAC: Stress Arousal Checklist. 22 Inclusion criteria included the diagnosis of PD [11], [12], [14], [15], [18], [20], [22]–[25], [27], ability to walk independently [11]–[14], [18], [20], [22], [25], motor fluctuations absence [13]–[15], [25], musculoskeletal injuries absence [21], [25], lack of other neurological disorders [14], [18], [24], [28], lack of severe bilateral visual or auditory impairments [11]–[15], [20], [21], [26], [27], ability to perform a 180º turns unaided [15], lack of cognitive impairments [13], [23], lack of deep brain stimulation [27] or apomorphine pump and jejunal levodopa gel infusion [14], presence of stable medication [25]. For the studies that evaluated FoG, the presence of this symptom was also an inclusion criteria [12]–[15], [18], [20]. Additionally, specific scores of PD scales were used to include participants: 19<MoCA>30 in [28]; MMSE>24 in [11], [14], [15], [24], [27]; FAB score>13 in [15]; H&Y stage I-III in [22], [25] , H&Y stage I-III while on medication in [27]; H&Y stage II in [26]; H&Y stage<III in [11], [23], [24]. Moreover, some studies used clinical characteristics as inclusion criteria, namely, age: older than 18 years old in [14], [18], [25]; younger than 75 years old [27] and younger than 85 years old [24]. Regarding the validation scenarios, all [2], [11]–[15], [17], [20]–[22], [24]–[28] articles conducted an intervention in a laboratory setting apart from [18], [23] which followed a home-based approach. Those articles that evaluated FoG used the following metrics: duration of FoG episodes [2], [13], [15], [18], percentage of trials with a FoG episode [2], number of FoG episodes [13]–[15], [18] and percentage of freezing time [14], [15], [18]. Moreover, gait-related metrics were used, such as step cadence (mean [2], [11], [13]–[15], [17], [24], coefficient of variation (CV) [2], [17] and standard deviation (SD) [11], [13]), step velocity (mean [2], [11]–[14], [17], [20], [21], [24], CV [2], [17] and SD [11], [24]), step length (mean and CV [2], [11], [12], [17], [20], [24]), stride length (mean and SD [13], [14], [21], [24]), peak velocity [15], step time (mean and asymmetry [12], [20]), stride time (mean and CV [15]), step width (mean [12], [20], [24] and SD [24]), festination [12], [20], kinematic variables [20], double limb support (DLS) [20], step height (mean and CV [15]), maximum head pelvis separation [15], time to maximum head-pelvis separation [15], maximum medial centre of mass (CoM) deviation [15], turn time [15], cycle time (mean and SD [14], [24]), step distance [22], leg raising [22], stance and swing phase [24], percentage of single limb support [24], posture sway [27], gait adaptability [27], 10-meter walking test (10MWT) [24], [27] and centre of pressure (CoP) excursion [28], in order to analyse gait performance. In terms of balance analysis, Berg Balance Scale (BBS) [22], [24], [27], ABC scale [22], [23], Timed Up and Go Test (TUG) [24], [27] and dual-task [23], one-leg stance [23], Mini-BESTest [23], Falls Efficacy Scale International (FES-I) [24] and perceived risk of falling [27] were assessed. Furthermore, Simulator Sickness Questionnaire (SSQ) [11], [12], [20], [24]–[26], [28] was conducted to evaluate simulator 23 sickness symptoms, Independent Television Commission Sense of Presence Inventory (ITC-SOPI) [25] to check perceived sense of presence, Intrinsic Motivation Inventory (IMI) [25] to score levels of motivation, System Usability Scale (SUS) [25], [26] to evaluate system overall usability, Parkinson's Disease Quality of Life Questionnaire (PDQL) [23], [27] to evaluate quality of life (QoL), Beck Depression Inventory (BDI) [23] to assess depressive disorder status, Borg 6-20 Questionnaire [27] to check participants’ perceived exertion and fatigue and Stress Arousal Checklist (SAC) [28] to assess stress. Finally, in [13]–[15], [23], [26] an interview was conducted on user experience, and in [26] a game experience questionnaire-post game (GEQ-post game) was also undertaken. Gómez-Jordana et al. [2] proposed a study to assess if the presence of virtual doorways in a virtual environment could induce FoG the same way real doorways do. For experimental protocols, there were three groups, a group of healthy participants as a control group, a group of PD patients without FoG and a group of PD patients with FoG, named as freezers (PD-f). All groups walked along a hallway under three different virtual conditions (no door, narrow doorway (100% of shoulder width) and standard doorway (125% of shoulder width)). The presence of virtual doors resulted in a reduction on step length and velocity and an increase on gait variability, with the worst values occurring for PD-f. The narrow door was the one that provoked the most FoG. Lheureux et al. [11] aimed to assess the effects of adding an optic flow displayed through an immersive virtual reality headset during treadmill walking on gait. PD patients were instructed to walk in a randomized order in 3 conditions: (i) overground walking; (ii) treadmill walking; and (iii) immersive virtual reality on treadmill walking. As a result, a greater step length and lower cadence were obtained. SSQ was similar between the (ii) and (iii) conditions. Yamagami et al. [12] intended to investigate whether virtual environments that replicate FoGprovoking situations would exacerbate gait impairments associated with FoG compared to unobstructed VR and physical laboratory environments. Participants performed a series of walking tasks on five different environments (physical laboratory without VR; virtual laboratory without obstacles; virtual doorway; virtual hallway; virtual street scene with crowds). The results showed that FoG-provoking VR environments could exacerbate gait impairments that are related to FoG. Besharat et al. [20] aimed to examine the effects of virtual doorways and hallways on gait kinematics among people with PD and FoG. Participants performed a series of walking tasks on four different conditions (physical laboratory; virtual laboratory; virtual doorway; virtual hallway). As a result, kinematic changes commonly associated with FoG episodes were obtained. 24 Zhao et al. [13] intended to evaluate rhythmic visual and auditory cueing in a laboratory setting. Participants performed a series of walking tasks on four different walking courses (wide turn, narrow turn, full turn, and doorway) in combination with three cues (metronome, flashing light and optic flow). A more stable gait pattern with the aid of these cues was obtained but FoG did not diminish significantly. The metronome was more effective than rhythmic visual cues and preferred by more participants. Geerse et al. [18] explored unfamiliarity and habituation effects associated with wearing the HoloLens on FoG and evaluated the potential immediate effect of Holocue on alleviating FoG in the home environment. Patients performed three sessions of 1.5h, scheduled one week apart. In the first session, participants walked a freezing provoking route multiple times with and without wearing the HoloLens (without Holocue), in their homes. Session 2 took place in a laboratory and consisted of individually customise the cues of the Holocue application in terms of intercue distance and preferred type of cues and familiarize participants to walking with the holographic cues. Finally, the last session took place again at the patients’ home. Participants walked the same route with the same conditions as in session 1, while wearing the HoloLens with and without the Holocue application. Wearing the HoloLens (without Holocue) did significantly increase the number and duration of FOG episodes, but this unfamiliarity effect disappeared with habituation over sessions. Holocue had overall no immediate effect on FOG, although objective and subjective benefits were observed for some individuals, most notably those with long and/or many FOG episodes. Badarny et al. [21] studied the effects of visual feedback cues on gait. The virtual environment consisted of a virtual tiled floor in a checkerboard arrangement. The experimental protocol was divided into 5 phases: (i) walking without the device; (ii) walking with the device placed on but with the display turned off; (iii) walking with the display turned on; (iv) walking without the device after a 15-minute break; and (v) re-evaluation of baseline performance without the device one week after the first examination. The results suggested that wearing the device turned off resulted in a negligible effect of about 2%. With the display turned on, 56% of the patients improved their gait speed or stride length or both. After removing the device, 68% of the patients showed over 20% improvement in either gait speed or stride length or both. One week later, 36% of the patients showed over 20% improvement in baseline performance with respect to the previous test. Janssen et al. [14] investigated the usability of 3D augmented reality cues compared to conventional 3D transverse bars on the floor and auditory cueing, in reducing FoG and improving gait parameters. Patients were presented to three walking courses (walking straight, stop and start and turning) with five cue conditions (two experimental conditions: AR visual cues bars, AR visual cues staircase; and three 31 Therefore, a systematic approach was followed to identify the requirements of the system, from the point of view of the user and the technologies, considering the limitations identified in the literature review, allowing to move on to the next tasks of the dissertation. Table 2-4 - Identified limitations of current VR/AR/MR-based approaches and guidelines for their mitigation Limitations Guidelines to be followed Technological Smart glasses characteristics: heavy, uncomfortable, monocular and with a narrow field of view Smart glasses should be more lightweight, comfortable, with a user-friendly design, binocular and with an adequate eye calibration and field of view Explore the use of mixed reality Unknown monitoring systems’ contribution Explore the use and potential of other integrated monitoring systems specified for different gait impairments Unclear correlation between virtual environments and tasks and better assessment and training Study which are the best virtual environments and virtual tasks to motor assessment and training Validation Failure to carry out usability, safety, and feasibility questionnaires Assessment of the VR/AR/MR-approach should include usability, safety and feasibility questionnaires and more objective tests Implementation of a suitable familiarization phase with VR/AR/MR equipment Short-term interventions and followup absence VR/AR/MR should incorporate treatment protocols of several sessions per week, for several weeks with longer follow-up intervals Control group absence VR/AR/MR should integrate a control group to study the effects of it Control condition absence VR/AR/MR should integrate a control condition to distinguish distraction by the smart glasses 3 SOLUTION OVERVIEW 33 The following chapter specifies the materials and methods used to develop the proposed strategy and to acquire and process all the data required. This includes i) an overview of the solution found, starting by summarising the problem raised; ii) a presentation of the project in which this dissertation is integrated; and iii) the respective project module to which this dissertation has contributed; and iv) an introduction to the devices and systems used. 3.1 PROBLEM DESCRIPTION From the literature review it was concluded that VR/AR/MR strategies have the potential to not only immerse patients in environments that recreate daily situations which may trigger PD-related gait disabilities but also to integrate biofeedback strategies to help patients emerge from these disabilities. However, some limitations were identified, such as the fact that the smart glasses were heavy, no usability, safety or feasibility questionnaires were used, and the lack of control conditions in the protocols, preventing a clear discussion of the results obtained. Thus, a new strategy must include patients with Parkinson’s disease as target audience and will be implemented based on MR technology, i.e., the combination of real-world immersion and virtual objects interaction, integrated with a motion tracking system. Patients’ motor performance will be recorded and assessed by the motion tracking system, which should present a real-time synchronization with the MR technology. Further, the analysis of users’ motion will make it possible to provide on-demand visual cues, following a visual biofeedback strategy. In this sense, this dissertation expects to (i) develop virtual environments that lead to gait impairments and (ii) integrate a biofeedback strategy that enables patients to overcome these episodes. Therefore, three different virtual environments were developed, in which patients were immersed and encouraged to perform motor tasks, that corresponded to three situations that typically cause PD gait impairments (turning, crossing doors, narrow spaces). In addition, a biofeedback strategy based on visual cues was proposed to improve patients’ motor performance in the same virtual environments. This solution is integrated in the +sImmersive module of the +sense project, which is presented in the next section. 3.2 +SENSE This dissertation is integrated and intended to contribute to the +sense project. The project aims to improve patients’ quality of life, promoting less dependence on third parties by improving their mobility 34 and motor autonomy. In this sense, +sense offers front-end high-tech solutions based on wearable biofeedback devices which rely on acquisition, interpretation, and feedback of patients’ sensorimotor information. Currently, +sense is divided into four modules, as shown in Figure 3-1: (1) +sBiofeedback; (2) +sMotion; (3) +sC-support and (4) +sImmersive. The development of this dissertation contributed to the fourth module. 3.3 +SIMMERSIVE This module brings a new paradigm shift by using mixed reality approaches, integrated with a motion tracking device and biofeedback strategies, as a complementary tool for motor monitoring and training of PD-related gait disabilities. The three virtual environments allow the immersion of the user in everyday situations, having to perform daily motor tasks, in order to achieve a more reliable motor assessment and rehabilitation. Thus, this dissertation brings a step forward in the knowledge of how mixed reality-based motor assessment and training can be applied in Parkinson's disease. The +sImmersive considers the multifactorial nature of PD and innovates by contributing with a patient-centred approach. Bearing this in mind, two devices were used: (1) mixed reality smart glasses; and (2) motion tracking system, explained in detail in the following sections. Figure 3-1 - +sense modules. 35 3.3.1 MIXED REALITY SMART GLASSES: MICROSOFT HOLOLENS 2 In order to implement the mixed reality strategy, the Microsoft HoloLens 2 was used, a fully immersive, portable, and wearable commercial setup device of augmented/mixed reality, Figure 3-2. It consists of an AR/MR headset and a USB Type-C cable, which allows to charge the smart glasses and connect them to other devices such as computers. Some of the HoloLens 2 system specifications are mentioned in Table 3-1 [30]. HoloLens 2 presents see-through holographic lenses, enabling to see the real world, never losing the sense of reality. Moreover, they have several sensors that allow head and eye tracking, making it possible for the glasses to always know where the user is in space. These smart glasses have built-in speakers and microphone. In this sense, beyond the visual feedback, they can also provide auditory feedback to users. One of the biggest strengths is that they can understand the human and the environment through hand tracking, eye tracking, voice, 6DoF tracking and spatial mapping, making these glasses user-friendly. In addition, they only need a USB Type-C cable to connect to a computer, they are lightweight (556g), one can wear glasses under them, and their battery lasts up to 3 hours of active use. This vast range of features of HoloLens 2 motivated its selection, as it was intended to use mixed reality smart glasses that allow holograms to be placed in real space, while still seeing the real world. Figure 3-2 - Microsoft HoloLens 2. 36 Table 3-1 - Specifications of HoloLens 2 smart glasses [30] HoloLens 2 Technical Specifications Display Optics See-through holographic lenses (waveguides) Resolution 2k 3:2 light engines Holographic density 2.5k radiants (light points per radiant) Eye-based rendering Display optimization for 3D eye position Sensors Head tracking 4 visible light cameras Eye tracking 2 IR cameras Depth 1-MP time-of-flight (ToF) depth sensor IMU Accelerometer, gyroscope, magnetometer Camera 8-MP stills, 10800p30 video Audio and speech Microphone array 5 channels Speakers Built-in spatial sound Human understanding Hand tracking Two-handed fully articulated model, direct manipulation Eye tracking Real-time tracking Voice Command and control on-device; natural language with internet connectivity Windows Hello Enterprise-grade security with iris recognition Environment understanding 6DoF tracking World-scale positional tracking Spatial Mapping Real-time environment mesh Mixed Reality Capture Mixed hologram and physical environment photos and video Compute and connectivity SoC Qualcomm Snapdragon 850 Compute Platform HPU Second-generation custom-built holographic processing unit Memory 4-GB LPDDR4x system DRAM Storage 64-GB UFS 2.1 Wi-Fi Wi-Fi: Wi-Fi 5 (802.11ac 2x2) Bluetooth 5 USB USB Type-C Fit Single size Yes Fits over glasses Yes Weight 566g Software Windows Holographic Operating System Microsoft Edge Dynamics 365 Remote Assist Dynamics 365 Guides 3D Viewer Power Battery life 2–3 hours of active use Charging USB-PD for fast charging Cooling Passive (no fans) 37 HoloLens 2 has some recommended system requirements (Table 3-2)[31] that the host computer must meet to properly enjoy the experience. A computer TUF Gaming with a NVIDIA GeForce GTX 1060 GPU was used to run and connect the software needed to build the mixed reality tool. According to Table 3-2, the computer TUF Gaming comprises all the minimum and recommended requirements to use HoloLens 2 system. Table 3-2 - Comparison of recommended system requirements for using HoloLens 2 [31] and the specifications of the used computer (TUF Gaming FX505GM_FX505GM) Component Recommended system requirements TUF Gaming FX505GM_FX505GM CPU 64-bit with 4 cores or equivalent Intel® CoreTM i7-8750H CPU 2.20GHz GPU DirectX 11.0 or later WDDM 1.2 driver or later NVIDIA GeForce GTX 1060 RAM 8 GB or more 32 GB Operating system 64-bit Windows 10 Pro, Enterprise, or Education (Hyper-V support) Windows 11 Home 22H2 3.3.2 MOTION TRACKING SYSTEM: XSENS MVN AWINDA The IMU-based motion capture system relies on MVN Awinda (Xsens, Enschede, The Netherlands) [32], [33] given its reliability for body motion analysis in free-living conditions. The lower body configuration (Figure 3-3a) comprises a total of 7 wearable Wireless Motion Trackers (MTw) sensors (Figure 3-3b) which are placed on the body through adjustable straps (Figure 3-3c). This system collects the lower-body kinematic data that will be used to study the participants’ motor performance and act accordingly. Furthermore, this system was used to communicate with Unity software, providing information about the occurrence of gait initial or final contact (IC/FC) so that the biofeedback could act in HoloLens 2. Figure 3-3 – Xsens MVN Awinda components. (a) lower body configuration; (b) MVN Motion Tracker (MTw); (c) MVN Awinda straps; (d) MVN Awinda station. 38 The MTw sensors have embedded accelerometers, gyroscopes and magnetometers that provide 3D acceleration, 3D angular velocity and 3D magnetic field, respectively [32], [33]. These measurements become particularly interesting for position and orientation estimation of human body segments. Thus, it was possible to develop an algorithm (Section 4.3) for detecting initial and final contacts, according to the data coming from Xsens, namely the angular velocity in y and the linear velocity in z of the foot sensors. Data from the MTw sensors are wirelessly transmitted and synchronised by the Awinda Station (Figure 3-3d). During the data acquisition sessions, it was used the MVN Analyze Pro 2021.2, an easy-to-use software for real-time viewing and recording, which allows the export of motion capture data to third party applications [32]. Furthermore, this software has a streaming feature which enables computers to stream the captured data over a network to other client computer, in real-time, Figure 3-4. This real-time network streaming protocol is based on User Datagram Protocol (UDP). The UDP Protocol is unidirectional, is stateless and does not require the receiver to answer incoming packets, which allows greater speed. Upon this, Xsens has developed plug-ins, available for Unity3D, for free at asset store, for usage with third party tools as a client application, allowing to receive motion capture data in real-time. The data content in the datagram is defined by the specific protocol set. Each datagram starts with a 24byte header followed by a variable number of bytes for each body segment, depending on the selected data protocol. All data is sent in ‘network byte order’, which corresponds to big-endian notation. The header contains the type of the data and some identification information, so the receiving end can apply it to the right target [34]. Thus, a new session was created for each “equipped” volunteer and anthropometric data was measured and registered to build the person’s biomechanical model. After, a calibration method is performed to align the MTw sensors with the user’s body segments by the “Npose + Walk” task. When a successful calibration is achieved and the “stream” feature is on, as well as the "Linear Segment Kinematics", “Angular Segment Kinematics” and “Time code” datagrams selected, it is finally possible to start recording a real-time session according to the defined protocol. 39 3.4 CONCLUSIONS After an extensive literature review about the currently VR/AR/MR-based approaches used in PD, it was noticed that mixed reality may be the best technology to be used with individuals with Parkinson's disease, as it allows virtual and interactive objects to be added to the real world, without ever losing the sense of reality. Thus, this dissertation aims to explore this technology not only for the assessment but also for the training of PD-related disabilities. To this end, this dissertation is inserted in the +sense project, contributing to the +sImmersive project module and makes use of two high-tech equipment, namely the HoloLens 2 mixed reality smart glasses and the Xsens motion tracking system. Figure 3-4 – Real-time streaming feature in MVN Analyze Pro. 4 SOLUTION DESCRIPTION 47 The data acquisition protocol consisted of performing two different trials, three times each, which consisted of walking in a straight line along 10 meters: (1) with the smart glasses OFF; and (2) with the smart glasses ON, showing a virtual scenario (scenario 3, narrow spaces). In total each patient performed 6 trials. After the experimental protocol was completed, the acquired data was analysed in offline. To do this, the trials were exported in MVN Analyze Pro and then the exported trials (mvnx files) were loaded into MATLAB. Afterwards, the real-time IC/FC detections were compared with the Xsens foot contact signals (ground truth), as depicted in Figure 4-2. Thus, (1) The angular velocity signal in the y-direction from both feet was compared with the respective foot contact signal to assess the performance of real-time identification of IC; (2) The velocity signal in the z-direction from both feet was compared with the respective foot contact signal to assess the performance of real-time identification of FC. Detected gait events were evaluated considering their accuracy (Equation 4-1), precision (Equation 4-2), sensitivity (Equation 4-3), and specificity (Equation 4-4). These metrics portray the performance of the developed algorithm. True positives (TP) corresponded to the gait events correctly identified, true negatives (TN) represented gait events that the algorithm correctly detected as a non-event, false positives (FP) corresponded to gait events not correctly identified and false negatives (FN) the events that should had been detected. Furthermore, advance and delayed detections were also assessed based on their percentage of occurrence and duration. Advance and delayed detections were considered from the TP detections. Xsens HoloLens 2 Figure 4-4 –Representation of the devices used in the verification tests. 48 Equation 4-1 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 (%)=𝑇𝑃+𝑇𝑁 𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁 Equation 4-2 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 (%)= 𝑇𝑃 𝑇𝑃+𝐹𝑃 Equation 4-3 𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 (%)= 𝑇𝑃 𝑇𝑃+𝐹𝑁 Equation 4-4 𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 (%)= 𝑇𝑁 𝑇𝑁+𝐹𝑃 4.3.2.2 RESULTS AND DISCUSSION Table 4-4 presents the performance of the real-time IC and FC detection algorithm. It shows the accuracy, precision, sensitivity, specificity and, advance and delayed detections (by means of their percentage of occurrence and duration). Table 4-4 – Results of the verification tests Metric Mean (±SD) Accuracy (%) 98.93 (± 1.38) Precision (%) 100.00 (± 0.00) Sensitivity (%) 97.87 (± 2.75) Specificity (%) 100.00 (± 0.00) Delays (freq %)) 0.27 (± 0.52) Delays (time (s)) 0.01(± 0.02) Advances (freq %)) 0.19 (± 0.37) Advances (time (s)) 0.02 (± 0.03) The proposed algorithm showed to be significantly accurate (mean of 98.93%), sensitive (mean of 97.87%), precise (100%), and specific (100%) for the tests performed, meaning that the developed algorithm is able to detect, without much error, the initial and final contacts, presenting sufficient capacity to integrate the biofeedback strategy, having reached the KPI3, which defined 96% as the percentage of accuracy to be met, that is, in a space of 10 meters where 20 steps are taken, the algorithm detects 19 IC/FCs. However, an adjustment to the thresholds was subsequently made using existing data from the 49 project database for 9 PD patients. Nevertheless, further modifications may have to be made in the future considering the heterogeneity of PD and the intraand inter-subject variability. 4.3.2.3 CONCLUSIONS The proposed real-time IC and FC detection algorithm has shown to be accurate, sensitive, precise, and specific. The adaptability introduced in the IC and FC detection ensures greater robustness of the system in the eventual occurrence of perturbations. These aspects make this algorithm suitable to be integrated with an actuation system, i.e., with a biofeedback strategy. However, there are some future challenges such as the need to validate this algorithm with (1) data collected from PD patients; (2) data collected from PD patients at various stages of the disease; and (3) data collected over time. 4.3.3 SPATIOTEMPORAL METRICS ESTIMATION In order to assess whether the immersive virtual environments were able to trigger PD-related gait disabilities and whether the visual biofeedback was able to help patients overcome these impairments, spatiotemporal metrics were estimated using the motion data captured by Xsens. Thus, a code was developed in MATLAB for this estimation. Table 4-5 presents the calculated spatiotemporal parameters, as well as the definition and formula of each and the units of measurement. Furthermore, the variability (SD) and asymmetry (AS) of these metrics were also calculated. 50 Table 4-5 - Spatiotemporal parameters: description, formula and units [39] Spatiotemporal parameter Definition Formula Measured units Step duration Time between the contact of two consecutive limbs in ground 𝐼𝐶𝑖+1 − 𝐼𝐶𝑖 Seconds Stride duration Duration of one gait cycle, i.e., the interval between two sequential initial contacts on the ground by the same limb 𝐼𝐶𝑖+2 − 𝐼𝐶𝑖 Seconds Stance phase duration Duration of stance phase or ratio of stance phase duration with stride duration (𝐹𝐶𝑖+1 − 𝐼𝐶𝑖)×100 𝑆𝑡𝑟𝑖𝑑𝑒 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 Seconds or percentage Swing phase duration Duration of swing phase or ratio of swing phase time with stride duration (𝑆𝑡𝑟𝑖𝑑𝑒𝑇𝑖𝑚𝑒𝑖+1 −𝑆𝑡𝑎𝑛𝑐𝑒𝑇𝑖𝑚𝑒𝑖)×100 𝑆𝑡𝑟𝑖𝑑𝑒 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 Seconds or percentage Double support phase duration Interval of time of the double support phase or ratio of double support phase duration with stride duration (𝐼𝐶𝑖+1 −𝐹𝐶𝑖)×100 𝑆𝑡𝑟𝑖𝑑𝑒 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 Seconds or percentage Step length Distance that one part of the foot moves in front of the same part of the other foot during each step 2√2𝐿ℎ−ℎ2 ,ℎ= ∬ 𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑣𝑒𝑟𝑡𝑖𝑐𝑎𝑙 𝐼𝐶𝑖+1 𝐼𝐶𝑖 Meters Stride length Distance between two consecutive initial contacts on the ground by the same limb 𝑠𝑡𝑒𝑝 𝑙𝑒𝑛𝑔𝑡ℎ𝑖+ 𝑠𝑡𝑒𝑝 𝑙𝑒𝑛𝑔𝑡ℎ𝑖+1 Meters Velocity Distance covered by the whole body in a given time 𝑠𝑡𝑒𝑝 𝑙𝑒𝑛𝑔𝑡ℎ𝑖 𝑠𝑡𝑒𝑝 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑖 Meters per second Cadence Number of steps taken in a specific time 𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦𝑖×60 𝑠𝑡𝑒𝑝 𝑙𝑒𝑛𝑔𝑡ℎ𝑖 Steps per minute ROM Range of the signals 𝑚𝑎𝑥 (𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑥,𝑦,𝑧) −min(𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑥,𝑦,𝑧) Meters per second squared RMS Relates to the vibration levels of a signal 𝑟𝑚𝑠 (𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑥,𝑦,𝑧) Meters per second squared JERK First time derivative of acceleration 𝑑𝑖𝑓𝑓 (𝑎𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑥,𝑦,𝑧) Meters per second cubed ROM: range of motion; RMS: Root mean square. 51 Figure 4-5 is a representation of the gait cycle of a healthy subject, for easier interpretation of the concepts. When the participant is exposed to virtual environments intended to assess motor performance, temporal metrics, such as step duration, are expected to increase and spatial metrics, including step length and velocity, to decrease, as these patients tend to present a more cautious behaviour in performing these tasks, leading to slower and smaller steps [2], [12], [20] Conversely, when biofeedback is used, temporal metrics are expected to decrease, and spatial metrics are expected to increase, i.e., more stable gait pattern, faster, and bigger steps [1], [13]–[15], [17]. 4.4 BIOFEEDBACK INTEGRATION Regarding biofeedback strategies, two modalities were developed, one in open loop and the other in closed loop, which are explained below. 4.4.1 OPEN LOOP STRATEGY The biofeedback strategy for virtual scenario 1, corridor with dices, consisted of using visual cues that indicated the path to be taken by the hand with the dice, from the initial location of the dice to the corresponding coloured box. To this end, arrows were drawn in the air, in the colour of the corresponding Figure 4-5 - Gait cycle of a healthy subject. 52 dice, to help the user to turn. Arrows were chosen because they represent direction and movement, trying to facilitate the turning of the body. Each set of arrows displayed the colour relative to the playing dice so that the user would not be confused about which dice is carrying. This strategy is intended to make turning a more fluid and easier task, by providing visuospatial cueing information. Figure 4-6 presents the strategy mentioned. 4.4.2 CLOSED LOOP STRATEGY The closed-loop biofeedback strategy was achieved by the communication protocol between MVN Analyze Pro and Unity, using the real-time IC/FC detection algorithm (Section 4.3.1). This way, the strategy for virtual scenarios 2 (virtual door) and 3 (narrow spaces) consisted of presenting visual cues on the floor in front of the user in a closed loop. In this case, it was chosen to use footprints since they show relevance in the gait. The green was chosen since the eye is most sensitive to a yellowish-green colour under normal lighting conditions [35], [36] and, beyond that, green is related to “being right”, “proceeding”, unlike de colour red, as in traffic lights. Thus, when the user places the right foot on the floor, a right IC or heel strike is detected, and the system will place a left green footprint to serve as a spatial guideline, indicating where to place the left foot. On the contrary, once the user places the left foot on the floor, a left IC or heel strike is detected, and the right green footprint is displayed. Therefore, a more fluid and continuous gait is motivated. Figure 4-7 shows a green footprint of the right foot after detection of a heel strike from the left foot. Figure 4-6 - Biofeedback strategy for scenario 1. 53 4.5 CONCLUSIONS This chapter has presented the methods used to implement a modular, user-customised, mixed reality-based technology solution that (1) immerses patients in environments that cause PD-related gait impairments; and (2) immerses patients in environments that help overcome these impairments with the aid of HoloLens 2, Xsens and biofeedback strategies (RQ2). The virtual environments and tasks were defined, having developed three different environments that aimed to represent the real-life situations that most cause PD-gait disabilities in PD patients, namely (1) turning (scenario 1); (2) walking through doors (scenario 2); and (3) walking in narrow spaces (scenario 3). Regarding biofeedback strategies, arrows and footprints were added to the virtual environments, and it would be expected that the user would follow these visual cues. The footprints were provided in closed loop, i.e. as the user walks, more footprints will appear which are activated by the occurrence of ICs. Thus, when the user puts one foot on the ground, the HoloLens projects the footprint relative to the opposite foot in order to promote a more continuous and fluid gait. To make this possible, an algorithm was developed to detect ICs and FCs in real time, based on adaptive thresholds. Furthermore, a MATLAB code was developed for the estimation of spatiotemporal metrics in offline, so that it was possible to evaluate the motor performance after exposure to the virtual environments and after exposure to the biofeedback strategies in these patients. Figure 4-7 – Display of the right green footprint. 5 SOLUTION VALIDATION 55 This chapter describes the methodologies for validating the solution. Firstly, the protocol used in the validation of the mixed reality strategies with individuals with PD is presented. This validation protocol followed a pre-post experimental study design aiming to evaluate the subjects’ motor performance. The participants and their characteristics are specified, as well as the inclusion and exclusion criteria. Next, the materials used in the intervention are described, followed by the data acquisition methods and the study variables. The data processing conducted to achieve the intended outcomes measures is exposed, as well as the statistical analysis performed. Finally, the results obtained are presented, as well as a detailed discussion of them, answering RQ3. 5.1 INTRODUCTION 5.1.1 HYPOTHESIS, RESEARCH QUESTION AND STUDY DESIGN PD is currently incurable, so its treatment consists of applying interventions that slow down the rapid progression of the disease. The mixed reality strategies developed in this dissertation aim to bring the day-to-day reality of the patient closer to the medical appointment. In fact, it becomes critical that these patients are correctly and objectively assessed, since disease progression is very fast. In addition, it is intended to verify whether this strategy could complement rehabilitation sessions, through a more fun and disease-focused training, using visual cues, with the aim of improving their mobility and autonomy. In that sense, the research question to be answered is RQ3: “How does the implemented modular technological solution, based on mixed reality integrated with a motion tracking system and with biofeedback strategies, affect the motor performance of PD patients during assessment and training?” The study in question consisted of a cross-sectional study as an observation of a defined population was conducted at a single point in time. Exposure to the intervention and outcome were determined simultaneously [40]. 5.2 METHODOLOGY The validation protocol with pathological end-users was conducted in Hospital of Braga, with the collaboration of the physicians from 2CA-Braga, following the principles of the Declaration of Helsinki and the Oviedo Convention, in accordance with the ethical guidelines of the Ethics Committee in Life and Health Sciences (CEICVS 147/2021). All participants filled out an informed consent to participate in the current research. 56 5.2.1 PARTICIPANTS Eleven subjects (six females and five males) were recruited and accepted to participate in this data collection. A list of inclusion and exclusion criteria was outlined in order to select the participants. Participants were recruited if they had: I) diagnosis of PD according to the UK Parkinson’s Disease Society Brain bank criteria; II) presence of freezing of gait; III) Hoehn and Yahr stage between 1 and 4; IV) age between 45 and 85 years old; and V) able to walk without assistance. Exclusion criteria were: I) presence of comorbid disorders likely to affect gait, including stroke, orthopaedic disease, rheumatologic disease, other neurological and musculoskeletal disorders, cardiovascular and pulmonary diseases; II) significative cognitive impairment (MMSE<24); III) obvious motor impairments; IV) visual acuity deficits; V) audiometric deficits; VI) pain that may affect walking; and VII) inability to perform a 180º turn without assistance. Table 5-1 presents the participants’ detailed clinical characteristics and anthropometrics. Table 5-1 - Demographic information about the PD participants Participant ID Gender (M/F) Age (years) Body height (cm) Body mass (kg) Clinical State NFoG-Q UPDRS, Part III H&Y PD-f 01 F 74 160 62 ON 27 66 4 PD-f 02 F 48 166 65 ON 24 4 PD-f 03 F 67 164 73 ON 24 71 3 PD 04 F 59 155 71 ON 0 18 2 PD 05 M 75 168 80 ON 0 12 2 PD-f 06 M 71 163 85 ON 24 19 2 PD-f 07 M 65 172 60 ON 25 15 1 PD-f 08 F 70 163 77 ON 13 22 2 PD-f 09 F 47 169 64 ON 12 14 1 PD-f 10 M 83 160 75 ON 26 57 4 PD-f 11 M 84 162 62 ON 26 42 3 Mean (±STD) - 67.55 (± 11.71) 163.82 (± 4.55) 70.36 (± 7.96) - 18.28 (± 9.88) 31.86 (±22.03) 2.29 (±1.08) ID: identification; PD: individual with Parkinson's disease; PD-f: individual with Parkinson's disease and freezing of gait; M: male; F: female; NFoG-Q: New freezing of gait questionnaire; UPDRS-III: Unified Parkinson’s disease rating scale – part III; H&Y: Hoehn and Yahr scale; STD: standard deviation. 5.2.2 MATERIALS The materials used in the validation phase were the HoloLens 2 mixed reality smart glasses and the Xsens motion tracking system. In addition, a document recording the participants' demographic and clinical information was also used. HoloLens 2 was used to display the virtual environments and to provide the visual cues (arrows and green footprints). In turn, Xsens was used to stream inertial data to Unity at 63 5.2.6 STATISTICAL ANALYSIS The statistical analysis was performed through IBM SPSS software version 25.0 (for Windows) (IMP Corp, Armonk, NY, USA). Firstly, descriptive statistics were obtained to summarise the results (means and standard deviations) for each group and the data normality was assessed using ShapiroWilk test. The population was considered normally distributed if the significance value was higher than 0.05. This study presents paired samples as the samples are from the same participants, Table 5-8. In this sense, to compare two paired groups, paired t test was performed for the parametric metrics and the Wilcoxon signed-rank test was applied to variables where the assumption of normality was not verified. When more than two paired groups were to be compared, repeated-measures ANOVA was used for normal populations, on the contrary, the Friedman test was used for non-normal variables. All statistical tests were executed considering a confidence level of 95% (α = 0.05). The statistical tests were conducted to evaluate the following null hypothesis (H0): “there are statistically significant differences between interventions”. If p -value<0.05, the H0 is accepted. Table 5-8 - Parametric and non-parametric tests Type of data Goal Measurement (of normal populations) Order, result or measure (of non-normal populations) Compare two pared samples Paired t test Wilcoxon test Compare more than two pared samples Repeated-measures ANOVA Friedman test 5.3 RESULTS This subchapter aims to present the results from the validation protocol with PD patients. The results are divided into three sections. Firstly, the results concerning the motor assessment are presented, followed by the results related to the motor training. In addition, the results of freezing gait episodes (number and duration) per test are shown. Finally, the answers of the user’s experience evaluation tests are depicted. Eleven patients underwent the tests, with ten patients completing all tests. Patient 10 dropped out due to fatigue/no interest. Most participants successfully completed all the tests provided, in an average duration of 40 minutes. 64 5.3.1 IMMERSIVE VIRTUAL FRAMEWORK FOR MOTOR ASSESSMENT Firstly, the aim was to check whether wearing the HoloLens 2 OFF would have any influence on the motor function of these patients. To this end, the TC1 and TC2 tests were compared. Regardless of the outcome of the comparison of these tests, the TC1 was selected to be compared to the other tests for the purpose of results analysis. Next, it was aimed to study the potential of mixed reality to cause PD-gait disabilities by comparing the TC1 with the M2 and, later, the TC1 with the M3. Thus, the first step of the statistical analysis was to perform a descriptive analysis by test. The results of this step are shown in Table 5-10, for control tests 1 and 2 (TC1, TC2) and monitoring tests 2 and 3 (M2 and M3). Afterwards, the normality of the features was verified using the Shapiro-Wilk test. As most of the features did not show a normal distribution, the Wilcoxon test was chosen for comparison of two paired samples. The results of this test are also shown in Table 5-10, in which the difference of a certain feature between the several tests is considered significant if the significance value is smaller than 0.05 and these are in bold. Regarding control tests TC1 and TC2, only a few metrics, namely, step and stride length, velocity and AS swing time, presented statistically significant differences, since the p-value < 0.05, corroborating the null hypothesis. Afterwards, the comparison between TC1 and M2 showed statistically significant differences in the metrics step and stride duration, stance, swing and double support phase, velocity, SD step and stride duration, SD stance and swing phase, SD step length, SD cadence and AS stance and swing time, corroborating the null hypothesis. Finally, the comparison between TC1 and M3 showed statistically significant differences in almost all metrics, so the null hypothesis is corroborated. All these metrics are in bold. Throughout data acquisition, the researchers visually assessed the existence of gait freezing episodes. Later, during data processing, these episodes were excluded and their number, average and total duration were counted per test. The results are presented in Table 5-9. Figure 5-5 presents two QR codes showing videos of patient 7 performing tests M2 and M3. 65 Table 5-9 - Number, average and total duration os freezing of gait episodes in control tests and monitoring tests 2 and 3 Test Number of episodes Average duration (s) Total duration (s) TC1 0 0 0 TC2 0 0 0 M2 4 6.73 74 M3 0 0 0 Figure 5-5 – Videos of participant 7 performing the M2 and M3 tests. 66 Table 5-10 - Spatiotemporal metrics and their descriptive statistics, Shapiro-Wilk test and Wilcoxon test for control tests and scenarios 2 and 3 TC1 TC2 Wilcoxon Test (sig) (TC1-TC2) M2 Wilcoxon Test (sig) (TC1-M2) M3 Wilcoxon Test (sig) (TC1-M3) Metric Mean Std deviation ShapiroWilk (sig) Mean Std deviation ShapiroWilk (sig) Mean Std deviation ShapiroWilk (sig) Mean Std deviation ShapiroWilk (sig) Step duration 0.618 0.087 0.298 0.618 0.082 0.144 0.811 0.740 0.233 0.254 0.022 0.812 0.360 0.001 0.003 Stride duration 1.236 0.176 0.345 1.235 0.165 0.122 0.868 1.478 0.472 0.218 0.025 1.615 0.722 0.001 0.005 Stance phase 62.057 3.047 0.168 62.688 3.295 0.081 0.053 67.056 5.887 0.383 0.000 67.360 7.848 0.002 0.000 Swing phase 37.943 3.047 0.168 37.312 3.295 0.081 0.053 32.914 5.937 0.361 0.000 32.640 7.848 0.002 0.000 Double support phase 24.196 6.104 0.170 25.508 6.550 0.102 0.058 34.562 12.255 0.247 0.000 34.442 16.290 0.001 0.000 Step length 0.568 0.096 0.378 0.519 0.135 0.022 0.004 0.509 0.165 0.075 0.053 0.500 0.127 0.613 0.003 Stride length 1.143 0.199 0.265 1.046 0.257 0.033 0.005 1.019 0.344 0.108 0.053 0.994 0.258 0.596 0.004 Velocity 0.934 0.198 0.226 0.853 0.218 0.047 0.012 0.750 0.275 0.128 0.002 0.702 0.215 0.273 0.000 Cadence 99.395 13.881 0.149 99.988 13.399 0.035 0.744 95.866 27.003 0.399 0.396 89.104 21.928 0.042 0.006 SD step time 0.040 0.022 0.000 0.055 0.059 0.000 0.616 0.222 0.274 0.000 0.001 0.240 0.370 0.000 0.002 SD stride time 0.055 0.037 0.000 0.076 0.090 0.000 0.828 0.307 0.370 0.000 0.002 0.317 0.447 0.000 0.003 SD stance time 0.045 0.026 0.000 0.068 0.092 0.000 0.616 0.280 0.344 0.000 0.001 0.259 0.396 0.000 0.002 SD swing time 0.031 0.022 0.000 0.037 0.033 0.000 0.557 0.101 0.101 0.000 0.005 0.147 0.202 0.000 0.003 SD step length 0.139 0.067 0.049 0.119 0.052 0.043 0.184 0.180 0.102 0.043 0.039 0.159 0.105 0.001 0.420 SD stride length 0.182 0.121 0.002 0.134 0.089 0.000 0.145 0.252 0.166 0.072 0.133 0.211 0.171 0.002 0.528 SD velocity 0.229 0.119 0.008 0.191 0.084 0.087 0.170 0.247 0.126 0.259 0.396 0.204 0.125 0.004 0.231 SD cadence 6.170 2.530 0.002 7.453 5.207 0.001 0.500 19.854 19.948 0.000 0.004 13.895 15.022 0.000 0.039 AS step time 0.034 0.023 0.092 0.032 0.022 0.299 0.695 0.114 0.179 0.000 0.231 0.136 0.173 0.000 0.017 AS stride time 0.009 0.011 0.000 0.005 0.006 0.001 0.316 0.026 0.051 0.000 0.446 0.033 0.068 0.000 0.758 AS stance time 0.018 0.016 0.038 0.027 0.030 0.002 0.085 0.048 0.057 0.000 0.031 0.069 0.119 0.000 0.078 AS swing time 0.011 0.016 0.000 0.031 0.033 0.000 0.004 0.047 0.062 0.000 0.011 0.094 0.140 0.000 0.008 AS step length 0.112 0.079 0.053 0.121 0.092 0.044 0.777 0.102 0.071 0.080 0.845 0.109 0.077 0.058 0.983 AS stride length 0.043 0.043 0.002 0.026 0.031 0.001 0.085 0.038 0.040 0.003 0.586 0.039 0.043 0.002 0.472 AS velocity 0.176 0.130 0.023 0.184 0.132 0.111 0.349 0.159 0.090 0.103 0.913 0.173 0.104 0.158 0.616 AS cadence 5.546 3.910 0.241 5.768 3.701 0.176 0.557 7.072 8.293 0.000 0.845 9.726 7.091 0.006 0.071 67 5.3.2 IMMERSIVE VIRTUAL FRAMEWORK FOR MOTOR TRAINING This subchapter presents the results of the biofeedback contribution to motor performance, i.e., tests M1 with T1 (dice), M2 with T2 (door) and M3 with T3 (narrow spaces) were compared. Thus, a descriptive analysis per test was first performed and then the normality of the features was studied, using Shapiro-Wilk test. As most of the metrics did not present a normal distribution, the Wilcoxon test was used for comparison of two paired samples. The results of these tests are shown in Table 5-12, Table 5-13 and Table 5-14, in which the difference of a certain feature between the several tests is considered significant if the significance value is smaller than 0.05. Regarding scenario 1, no metrics showed statistically significant differences, rejecting the null hypothesis. In turn, scenario 2 showed that only the metrics step duration, stride duration, velocity, cadence, and SD velocity, presented statistically significant differences, since the p-value < 0.05, corroborating the null hypothesis. Finally, scenario 3 had two spatiotemporal metrics that showed statistically significant differences, namely step length and cadence. Thus, the null hypothesis is corroborated. All these metrics are in bold. Throughout data acquisition, the researchers visually assessed the existence of gait freezing episodes. Later, during data processing, these episodes were excluded and their number, average and total duration were counted per test. The results are presented in Table 5-11. Figure 5-6 presents six QR codes showing videos of patient 7 performing tests M1, T1, M2, T2, M3 and T3. Table 5-11 - Number, average and total duration os freezing of gait episodes in monitoring tests 1, 2, and 3 and training tests 1, 2 and 3 Test Number of episodes Average duration (s) Total duration (s) M1 0 0 0 T1 0 0 0 M2 4 6.73 74 T2 6 9.78 107.6 M3 0 0 0 T3 4 3.13 34.4 68 Table 5-12 - Spatiotemporal metrics and their descriptive statistics, Shapiro-Wilk test and Wilcoxon test for scenario 1 M1 T1 Wilcoxon Test (sig) Metric Mean Std deviation Shapiro-Wilk (sig) Mean Std deviation Shapiro-Wilk (sig) ROM X (+) 3.630 1.427 0.049 3.472 1.269 0.002 0.390 ROM Y (+) 3.616 1.279 0.050 3.241 0.851 0.266 0.372 ROM Z (+) 4.720 3.046 0.000 3.985 3.422 0.000 0.168 RMS X (+) 0.478 0.217 0.003 0.508 0.214 0.018 0.178 RMS Y (+) 0.480 0.180 0.195 0.491 0.149 0.279 0.615 RMS Z (+) 0.380 0.157 0.092 0.383 0.222 0.001 0.833 JERK X (-) 0.002 0.006 0.007 0.001 0.005 0.005 0.158 JERK Y (-) 0.002 0.004 0.013 0.000 0.003 0.502 0.123 JERK Z (-) 0.000 0.004 0.054 0.001 .003 0.041 0.661 Table 5-13 - Spatiotemporal metrics and their descriptive statistics, Shapiro-Wilk test and Wilcoxon test for scenario 2 M2 T2 Wilcoxon Test (sig) Metric Mean Std deviation Shapiro-Wilk (sig) Mean Std deviation Shapiro-Wilk (sig) Step duration 0.758 0.228 0.274 0.848 0.298 0.000 0.031 Stride duration 1.513 0.463 0.230 1.696 0.606 0.000 0.028 Stance phase 67.517 5.724 0.536 69.522 5.243 0.393 0.064 Swing phase 32.451 5.775 0.505 30.478 5.243 0.393 0.071 Double support phase 35.519 11.918 0.252 38.728 10.415 0.524 0.170 Step length 0.518 0.165 0.093 0.499 0.189 0.135 0.948 Stride length 1.038 0.345 0.135 0.991 0.408 0.045 0.777 Velocity 0.747 0.283 0.118 0.630 0.268 0.008 0.016 Cadence 93.068 25.000 0.319 79.108 15.118 0.282 0.002 SD step time 0.227 0.282 0.000 0.209 0.257 0.000 0.446 SD stride time 0.315 0.380 0.000 0.285 0.278 0.000 0.327 SD stance time 0.292 0.351 0.000 0.261 0.255 0.000 0.332 SD swing time 0.098 0.103 0.000 0.118 0.146 0.000 0.231 SD step length 0.182 0.104 0.070 0.168 0.092 0.006 0.647 SD stride length 0.254 0.171 0.087 0.210 0.135 0.004 0.586 SD velocity 0.246 0.130 0.181 0.197 0.127 0.000 0.028 SD cadence 18.870 20.107 0.000 13.568 7.317 0.003 0.557 AS step time 0.118 0.183 0.000 0.090 0.075 0.044 0.616 AS stride time 0.027 0.053 0.000 0.025 0.029 0.002 0.845 Figure 5-6 - Videos of participant 7 performing tests M1, T1, M2, T2, M3 and T3. 69 AS stance time 0.048 0.058 0.000 0.064 0.102 0.000 0.983 AS swing time 0.047 0.063 0.000 0.060 0.119 0.000 0.286 AS step length 0.104 0.072 0.145 0.134 0.100 0.133 0.184 AS stride length 0.040 0.040 0.005 0.039 0.044 0.000 0.647 AS velocity 0.158 0.092 0.077 0.159 0.113 0.241 0.586 AS cadence 7.044 8.547 0.000 7.023 4.985 0.032 0.913 Table 5-14 - Spatiotemporal metrics and their descriptive statistics, Shapiro-Wilk test and Wilcoxon test for scenario 3 M3 T3 Wilcoxon test (sig) Metric Mean Std deviation Shapiro-Wilk (sig) Mean Std deviation Shapiro-Wilk (sig) Step duration (-) 0.762 0.229 0.022 0.821 0.398 0.000 0.147 Stride duration (-) 1.512 0.449 0.023 1.627 0.746 0.000 0.147 Stance phase 67.644 7.198 0.001 66.531 3.850 0.223 0.520 Swing phase 32.356 7.198 0.001 33.469 3.850 0.223 0.520 Double support phase 35.295 15.152 0.001 33.065 8.002 0.232 0.314 Step length (+) 0.483 0.107 0.418 0.544 0.128 0.229 0.044 Stride length (+) 0.960 0.217 0.539 1.071 0.263 0.146 0.070 Velocity (+) 0.693 0.191 0.652 0.744 0.263 0.686 0.841 Cadence 89.330 16.968 0.027 83.140 19.546 0.290 0.024 SD step time 0.192 0.303 0.000 0.161 0.236 0.000 0.811 SD stride time 0.277 0.393 0.000 0.199 0.239 0.000 0.809 SD stance time 0.220 0.343 0.000 0.203 0.324 0.000 0.936 SD swing time 0.122 0.164 0.000 0.123 0.215 0.000 0.841 SD step length 0.144 0.098 0.000 0.162 0.106 0.001 0.421 SD stride length 0.184 0.163 0.000 0.212 0.178 0.000 0.314 SD velocity 0.190 0.108 0.003 0.209 0.135 0.001 0.904 SD cadence 13.360 15.084 0.000 9.254 4.855 0.087 0.445 AS step time 0.124 0.174 0.000 0.136 0.294 0.000 0.421 AS stride time 0.030 0.070 0.000 0.037 0.103 0.000 0.557 AS stance time 0.047 0.078 0.000 0.142 0.403 0.000 0.445 AS swing time 0.069 0.097 0.000 0.115 0.300 0.000 0.825 AS step length 0.118 0.073 0.111 0.111 0.104 0.013 0.398 AS stride length 0.027 0.028 0.005 0.049 0.077 0.000 0.227 AS velocity 0.173 0.103 0.273 0.140 0.107 0.008 0.122 AS cadence 8.772 6.882 0.000 7.226 6.754 0.008 0.277 5.3.3 SSQ AND IMI QUESTIONNAIRES At the end of data acquisition, participants completed the aforementioned acceptability questionnaires, SSQ and IMI (Appendix B – Subjective questionnaires). Table 5-15 presents the results of these questionnaires. 70 Table 5-15 - SSQ and IMI questionnaires results Participant ID SSQ IMI Interest/Enjoyment subscale Value/Usefulness subscale PD-f 01 4 5.67 5.33 PD-f 02 3 5.67 5.33 PD-f 03 4 7 6.33 PD 04 0 7 5.67 PD 05 0 7 7 PD-f 06 4 5.33 7 PD-f 07 0 7 7 PD-f 08 3 6.67 6.67 PD-f 09 1 7 7 PD-f 10 1 7 6.67 PD-f 11 1 7 6.33 Mean (±STD) 1.91 (± 1.62) 6.58 (± 0.64) 6.39 (± 0.63) 6.49 (± 0.64) 5.4 DISCUSSION This subsection discusses the results obtained in the previous subsection. The analysis is made for both “motor assessment” and “motor training” separately. In addition, a brief discussion is elaborated regarding the occurrence of FoG episodes and also the results of the acceptability questionnaires. 5.4.1 IMMERSIVE VIRTUAL FRAMEWORK FOR MOTOR ASSESSMENT Looking at the mean values of the metrics of the two control tests (TC1 and TC2), it can be seen that they hardly varied, with the exception of step and stride length and velocity. Actually, the referred metrics plus AS swing time presented statistically significant differences. Thus, it is concluded that the use of HoloLens influences motor performance in the mentioned metrics, even if they are switched off. This may be due to the presence of the HoloLens lenses that are not completely transparent, seeing some reflections. Virtual environment 2 (doors) showed an increase in the mean values of step and stride duration, and a decrease in step and stride length, velocity, and cadence. These metrics behaved as expected (just like [2]) with the exception of cadence, which should have increased. In fact, this virtual environment showed fourteen metrics out of twenty-five (step and stride duration, stance, swing and double support phase, velocity, SD step and stride duration, SD stance and swing phase, SD step length, SD cadence and AS stance and swing time) with statistically significant differences 71 between the control and monitoring tests. This means that the MR technology really disturbed the motor performance of the patients in the way that was expected, due to the presence of virtual objects and especially the virtual door that was able to recreate a real door. In turn, virtual environment 3 (narrow spaces) showed an increase in the mean values of step and stride duration and a decrease in step and stride length, velocity, and cadence. Actually, these metrics behaved as expected with the exception of cadence, which should have increased. Observing the results of the Wilcoxon test, sixteen out of twenty-five metrics showed statistically significant differences, demonstrating that the MR technology may in fact have triggered PD-gait related disabilities. This may be due to the presence of the various virtual objects that created a narrower corridor than the real corridor, acting as obstacles for the participant, making him take smaller and slower steps. With regard to the occurrence of FoG episodes, the monitoring tests should have increased their number and duration. However, after analysing the results, it was found that monitoring test 2 was the only one that caused these episodes. This may be due to the fact that: (1) disease may be “masked” by medication, causing participants not to suffer from FoG, since data acquisition was performed 1h after medication intake on average, i.e., in “ON” phase; (2) heterogeneity in the origin of FoG, since some participants reported that they suffer from these episodes in stressful situations, other patients suffer right after waking up, as well as, other patients suffer when they are in crowded places. 5.4.2 IMMERSIVE VIRTUAL FRAMEWORK FOR MOTOR TRAINING Regarding virtual environment 1 (dices), one would expect the average ROM and RMS values to increase and the average JERK values to decrease with the use of the biofeedback strategy. However, by analysing the results one notices that the average ROM values for the three axes decreased, in turn the average RMS values for all axes increased and the same happened for the average JERK values. Thus, only the RMS behaved as expected. From the results of the Wilcoxon test, no statistically significant differences were found, making it possible to mention that the visual biofeedback strategy, namely the coloured arrows, had no evident impact. This may have occurred because the smart glasses do not have a sufficient field of view (FOV), forcing patients to look in the direction of the floor. Another reason could be that there was little time in contact with the biofeedback strategy. Virtual environment 2 (doors) showed an increase in the mean values of step and stride duration, a decrease in the mean values of step and stride length as well as a decrease in the mean values of velocity and cadence. Nevertheless, step and stride duration and cadence were expected to decrease, 72 step and stride length were expected to increase, just as velocity, since this biofeedback strategy aims to improve motor performance, obtaining larger and faster steps. In fact, this virtual environment showed some metrics (step and stride duration, velocity, SD velocity and cadence) with statistically significant differences between the monitoring and training tests. Thus, biofeedback may have negatively affected these metrics once the mean values behaved contrary to what was expected. This may be due to (1) the participant was left waiting for the footprints, and (2) the realtime IC and FC detection algorithm was not suitable for these patients' gait causing the footprints not to appear right away, increasing their reaction time. On the other hand, the cadence values behaved as expected. This may be due to the fact that the footprints indicate spatial information to the participant, i.e., where to place the next foot. In turn, virtual environment 3 (narrow spaces) showed an increase in the mean values of step and stride duration as well as step, stride length and velocity. On the other hand, the cadence decreased its mean value with the use of the footprints. Actually, step and stride length, velocity and cadence behaved as it was expected. On the contrary, step and stride duration should have decreased. There are two spatiotemporal metrics that showed statistically significant differences, namely step length and cadence. This may be due to the intention of the footprints to "force" the participant to be aware of them and to help planning where to place his feet, guiding him to the finish line. In addition, the footprints had a pre-defined distance between them (dependent on the height of the participant), influencing the participant to follow and imitate the visual cues. In what concerns the occurrence of freezing of gait episodes, the number and duration of these episodes would be expected to reduce or disappear in the biofeedback training trials. However, this did not occur and may be due to (1) unfamiliarity with visual cues, as participants reported that they had never interacted with these; (2) reduced field of view of the HoloLens 2 causing participants to sometimes fail to see visual cues. 5.4.3 SSQ AND IMI QUESTIONNAIRES After analysing the results obtained for the SSQ it was concluded that they did not reflect any symptoms (nausea, disorientation, and oculomotor) after exposure to the virtual environments. Furthermore, no participants verbally indicated that they had symptoms of simulator sickness. In turn, the results of the IMI questionnaire showed that the system received high ratings on the interest and value subscales. The participants were always happy and willing to participate in the tests, 79 [1] Y. W. Wang, C. H. Chen, and Y. C. Lin, “Balance Rehabilitation System for Parkinson’s Disease Patients based on Augmented Reality,” 2nd IEEE Eurasia Conference on IOT, Communication and Engineering 2020, ECICE 2020 , pp. 191–194, Oct. 2020, doi: 10.1109/ECICE50847.2020.9302018. [2] L. I. Gómez-Jordana, J. Stafford, C. E. Peper, and C. M. Craig, “Crossing Virtual Doors: A New Method to Study Gait Impairments and Freezing of Gait in Parkinson’s Disease,” Parkinsons Dis , vol. 2018, 2018, doi: 10.1155/2018/2957427. [3] A. Mirelman et al. , “Gait impairments in Parkinson’s disease,” Lancet Neurol , vol. 18, no. 7, pp. 697–708, Jul. 2019, doi: 10.1016/S1474-4422(19)30044-4. [4] J. Nonnekes, A. H. Snijders, J. G. Nutt, G. Deuschl, N. Giladi, and B. R. Bloem, “Freezing of gait: a practical approach to management,” Lancet Neurol , vol. 14, no. 7, pp. 768–778, Jul. 2015, doi: 10.1016/S1474-4422(15)00041-1. [5] M. Mancini, K. Smulders, G. Harker, S. Stuart, and J. G. Nutt, “Assessment of the ability of openand closed-loop cueing to improve turning and freezing in people with Parkinson’s disease,” Scientific Reports 2018 8:1 , vol. 8, no. 1, pp. 1–9, Aug. 2018, doi: 10.1038/s41598-018-311564. [6] P. H. Chen, R. L. Wang, D. J. Liou, and J. S. Shaw, “Gait Disorders in Parkinson’s Disease: Assessment and Management,” Int J Gerontol , vol. 7, no. 4, pp. 189–193, Dec. 2013, doi: 10.1016/J.IJGE.2013.03.005. [7] R. Iansek, F. Huxham, and J. McGinley, “The sequence effect and gait festination in Parkinson disease: contributors to freezing of gait?,” Mov Disord , vol. 21, no. 9, pp. 1419–1424, Sep. 2006, doi: 10.1002/MDS.20998. [8] R. Chong et al. , “Closed-Loop VR-Based Interaction to Improve Walking in Parkinson’s Disease,” Journal of Novel Physiotherapies 2011 1:1 , vol. 1, no. 1, pp. 1–7, Dec. 2011, doi: 10.4172/21657025.1000101. [9] A. L. Silva de Lima et al. , “Freezing of gait and fall detection in Parkinson’s disease using wearable sensors: a systematic review,” J Neurol , vol. 264, no. 8, pp. 1642–1654, Aug. 2017, doi: 10.1007/S00415-017-8424-0. [10] S. M. Rissanen, M. Koivu, P. Hartikainen, and E. Pekkonen, “Ambulatory surface electromyography with accelerometry for evaluating daily motor fluctuations in Parkinson’s disease,” Clin Neurophysiol , vol. 132, no. 2, pp. 469–479, Feb. 2021, doi: 10.1016/J.CLINPH.2020.11.039. 80 [11] A. Lheureux et al. , “Immersive Virtual Reality to Restore Natural Long-Range Autocorrelations in Parkinson’s Disease Patients’ Gait During Treadmill Walking,” Front Physiol , vol. 11, p. 1207, Sep. 2020, doi: 10.3389/FPHYS.2020.572063/BIBTEX. [12] M. Yamagami et al. , “Effects of virtual reality environments on overground walking in people with Parkinson disease and freezing of gait,” Disabil Rehabil Assist Technol , 2020, doi: 10.1080/17483107.2020.1842920. [13] Y. Zhao et al. , “Feasibility of external rhythmic cueing with the Google Glass for improving gait in people with Parkinson’s disease,” J Neurol , vol. 263, no. 6, p. 1156, Jun. 2016, doi: 10.1007/S00415-016-8115-2. [14] S. Janssen et al. , “Usability of three-dimensional augmented visual cues delivered by smart glasses on (Freezing of) gait in parkinson’s disease,” Front Neurol , vol. 8, no. JUN, p. 279, Jun. 2017, doi: 10.3389/FNEUR.2017.00279/BIBTEX. [15] S. Janssen et al. , “The Effects of Augmented Reality Visual Cues on Turning in Place in Parkinson’s Disease Patients With Freezing of Gait,” Front Neurol , vol. 11, p. 185, Mar. 2020, doi: 10.3389/FNEUR.2020.00185/FULL. [16] D. J. Moore, A. B. West, V. L. Dawson, and T. M. Dawson, “Molecular pathophysiology of Parkinson’s disease,” Annual Review of Neuroscience , vol. 28. pp. 57–87, 2005. doi: 10.1146/annurev.neuro.28.061604.135718. [17] L. I. Gómez-Jordana, J. Stafford, C. E. Peper, and C. M. Craig, “Virtual Footprints Can Improve Walking Performance in People With Parkinson’s Disease,” Front Neurol , vol. 9, no. AUG, Aug. 2018, doi: 10.3389/FNEUR.2018.00681. [18] D. J. Geerse, B. Coolen, J. J. van Hilten, and M. Roerdink, “Holocue: A Wearable Holographic Cueing Application for Alleviating Freezing of Gait in Parkinson’s Disease,” Front Neurol , vol. 12, Jan. 2022, doi: 10.3389/fneur.2021.628388. [19] J. Jankovic, “Parkinson’s disease: Clinical features and diagnosis,” Journal of Neurology, Neurosurgery and Psychiatry , vol. 79, no. 4. BMJ Publishing Group, pp. 368–376, 2008. doi: 10.1136/jnnp.2007.131045. [20] A. Besharat et al. , “Virtual reality doorway and hallway environments alter gait kinematics in people with Parkinson disease and freezing,” Gait Posture , vol. 92, pp. 442–448, Feb. 2022, doi: 10.1016/j.gaitpost.2021.12.013. 81 [21] S. Badarny, J. Aharon-Peretz, Z. Susel, G. Habib, and Y. Baram, “Virtual reality feedback cues for improvement of gait in patients with Parkinson’s disease,” Tremor Other Hyperkinet Mov (N Y) , vol. 4, no. 0, p. 225, Apr. 2014, doi: 10.7916/D8V69GM4. [22] Y. W. Wang, C. H. Chen, and Y. C. Lin, “Balance Rehabilitation System for Parkinson’s Disease Patients based on Augmented Reality,” 2nd IEEE Eurasia Conference on IOT, Communication and Engineering 2020, ECICE 2020 , pp. 191–194, Oct. 2020, doi: 10.1109/ECICE50847.2020.9302018. [23] T. Tunur, A. DeBlois, E. Yates-Horton, K. Rickford, and L. A. Columna, “Augmented reality-based dance intervention for individuals with Parkinson’s disease: A pilot study,” Disabil Health J , vol. 13, no. 2, Apr. 2020, doi: 10.1016/J.DHJO.2019.100848. [24] R. S. Calabrò et al. , “Improving motor performance in Parkinson’s disease: a preliminary study on the promising use of the computer assisted virtual reality environment (CAREN),” Neurological Sciences , vol. 41, no. 4, pp. 933–941, Apr. 2020, doi: 10.1007/S10072-019-04194-7. [25] J. M. Finley, M. Gotsis, V. Lympouridis, S. Jain, A. Kim, and B. E. Fisher, “Design and Development of a Virtual Reality-Based Mobility Training Game for People With Parkinson’s Disease,” Front Neurol , vol. 11, Jan. 2021, doi: 10.3389/FNEUR.2020.577713. [26] P. Campo-Prieto et al. , “Can Immersive Virtual Reality Videogames Help Parkinson’s Disease Patients? A Case Study,” Sensors 2021, Vol. 21, Page 4825 , vol. 21, no. 14, p. 4825, Jul. 2021, doi: 10.3390/S21144825. [27] Y. Wang et al. , “Efficacy of C-Mill gait training for improving walking adaptability in early and middle stages of Parkinson’s disease,” Gait Posture , vol. 91, pp. 79–85, Jan. 2022, doi: 10.1016/J.GAITPOST.2021.10.010. [28] A. Kim, N. Darakjian, and J. M. Finley, “Walking in fully immersive virtual environments: an evaluation of potential adverse effects in older adults and individuals with Parkinson’s disease,” J Neuroeng Rehabil , vol. 14, no. 1, Feb. 2017, doi: 10.1186/S12984-017-0225-2. [29] H. R. Gonçalves, A. M. Rodrigues, and C. P. Santos, “Vibrotactile biofeedback devices in Parkinson’s disease: a narrative review,” Med Biol Eng Comput , vol. 59, no. 6, pp. 1185–1199, Jun. 2021, doi: 10.1007/S11517-021-02365-3. [30] “HoloLens 2—Overview, Features, and Specs | Microsoft HoloLens.” https://www.microsoft.com/en-us/hololens/hardware# (accessed Oct. 05, 2022). [31] “Hololens System Requirements – Vangos Pterneas.” https://pterneas.com/2016/03/31/hololens-system-requirements/ (accessed Oct. 05, 2022). 82 [32] “MVN User Manual,” 2005. [33] “MTw_Awinda_User_Manual”. [34] “JMU Updated for release MVN Analyze/Animate 4.0 (new HW + support for TCP protocol),” 2013, Accessed: Oct. 07, 2022. [Online]. Available: https://www.assetstore.unity3d.com/en/#!/content/11338 [35] “Nondestructive Evaluation Techniques : Penetrant.” https://www.ndeed.org/NDETechniques/PenetrantTest/Introduction/visualacuity.xhtml (accessed Oct. 14, 2022). [36] R. Moreira, H. Goncalves, A. Rodrigues, and C. Santos, “The Potential of Visual Cues to Overcome Freezing of Gait in Parkinson’s Disease*; The Potential of Visual Cues to Overcome Freezing of Gait in Parkinson’s Disease*,” 2019. [Online]. Available: https://www.ninds.nih.gov/Disorders/All-Disorders/Parkinsons- [37] J. Figueiredo, P. Félix, L. Costa, J. C. Moreno, and C. P. Santos, “Gait Event Detection in Controlled and Real-Life Situations: Repeated Measures from Healthy Subjects,” IEEE Transactions on Neural Systems and Rehabilitation Engineering , vol. 26, no. 10, pp. 1945–1956, Oct. 2018, doi: 10.1109/TNSRE.2018.2868094. [38] H. R. Gonçalves, A. Rodrigues, and C. P. Santos, “Gait monitoring system for patients with Parkinson’s disease,” Expert Syst Appl , vol. 185, Dec. 2021, doi: 10.1016/j.eswa.2021.115653. [39] J. Figueiredo, J. C. Moreno, and C. P. Santos, “A tutorial on human locomotion for developing smart gait rehabilitation devices.” [40] J. Glogowski, “GSU Library Research Guides: Literature Reviews: Types of Clinical Study Designs”, Accessed: Oct. 15, 2022. [Online]. Available: https://research.library.gsu.edu/c.php?g=115595&p=755213 [41] R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness,” Int J Aviat Psychol , vol. 3, no. 3, pp. 203–220, 1993, doi: 10.1207/s15327108ijap0303_3. [42] “Intrinsic Motivation Inventory (IMI) Scale Description.” APPENDICES 84 APPENDIX A – EXPERIMENTAL PROTOCOL 85 86 87 88