Evaluating constrained users ability to interact with virtual reality applications

Universidade do Minho Escola de Engenharia Departamento de Informática Tiago Ramos Ribeiro Evaluating Constrained Users Ability to Interact with Virtual Reality Applications January 2024 Universidade do Minho Escola de Engenharia Departamento de Informática Tiago Ramos Ribeiro Evaluating Constrained Users Ability to Interact with Virtual Reality Applications Master dissertation Master Degree in Informatics Engineering Dissertation supervised by Pedro Manuel Rangel Santos Henriques Nuno Miguel Feixa Rodrigues January 2024 i AUTHOR COPYRIGHTS AND TERMS OF USAGE BY THIRD PARTIES This is an academic work which can be utilized by third parties given that the rules and good practices internationally accepted, regarding author copyrights and related copyrights. Therefore, the present work can be utilized according to the terms provided in the license bellow. If the user needs permission to use the work in conditions not foreseen by the licensing indicated, the user should contact the author, through the RepositóriUM of University of Minho. License provided to the users of this work Attribution-NonCommercial CC BY-NC https://creativecommons.org/licenses/by-nc/4.0/ ii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the University of Minho. Tiago Ramos Ribeiro ABSTRACT This Master’s Project presents a comprehensive exploration of a novel Virtual Reality (VR) application designed to evaluate and enhance user performance within the context of constraints experienced by individuals, including those confined to an Intensive Care Unit (ICU). The work unfolds through a detailed examination of the proposal, development, and assessment phases. The proposal lays the foundation for the project, emphasizing the need for an immersive technology-based solution to assess ICU patients’ abilities. It includes a well-structured system architecture, deployment architecture, and data architecture. This framework guides the subsequent phases, offering insights into the development and assessment processes. In the development phase, the practical realization of the VR application is explored. It highlights adjustments tailored to the specific needs of ICU patients, offering valuable insights into user progress, reducing dependency on external assistance. Eight distinct tasks are detailed, categorized based on complexity and fundamental functionalities. The assessment phase evaluates the real-world impact of the VR application through three interventions. While limited to non-ICU environments, these interventions capture data from users who share critical constraints with ICU patients. The assessment involves correlation analysis of numerous variables, including age, cognitive function (assessed through the Mini-Mental Status Examination), and prior VR experience. The results unearth significant correlations, shedding light on age-related differences, the influence of cognitive ability, and the impact of prior VR exposure on task performance. This comprehensive exploration represents an essential contribution to the burgeoning field of VR applications for healthcare, specifically targeting constrained user groups like ICU patients. The findings underscore the significance of considering user characteristics, prior experience, and cognitive function when designing VR interventions. Further research is warranted to refine assessment methodologies and expand the scope of real ICU patient testing, ultimately paving the way for improved patient care and enhanced rehabilitation practices. Keywords: Virtual Reality, Intensive care, Cognitive resilience, Non-pharmacological therapies iii RESUMO Este projeto de mestrado apresenta uma exploração abrangente de uma nova aplicação de Realidade Virtual (RV) concebida para avaliar e melhorar o desempenho do utilizador no contexto dos constrangimentos vividos pelos indivíduos, incluindo os que estão confinados a uma Unidade de Cuidados Intensivos (UCI). O trabalho desenrola-se através de uma análise pormenorizada das fases de proposta, desenvolvimento e avaliação. A proposta estabelece as bases do projeto, salientando a necessidade de uma solução imersiva baseada na tecnologia para avaliar as capacidades dos doentes da UCI. Inclui uma arquitetura de sistema bem estruturada, uma arquitetura de implementação e uma arquitetura de dados. Esta estrutura orienta as fases subsequentes, oferecendo informações sobre os processos de desenvolvimento e avaliação. Na fase de desenvolvimento, é explorada a realização prática da aplicação de RV. Destaca os ajustes adaptados às necessidades específicas dos pacientes da UCI, oferecendo informações valiosas sobre o progresso do utilizador, reduzindo a dependência da assistência externa. São detalhadas oito tarefas distintas, categorizadas com base na complexidade e nas funcionalidades fundamentais. A fase de avaliação avalia o impacto da aplicação de RV no mundo real através de três intervenções. Embora limitadas a ambientes não UCI, estas intervenções captam dados de utilizadores que partilham restrições críticas com doentes da UCI. A avaliação envolve a análise de correlação de numerosas variáveis, incluindo a idade, a função cognitiva (avaliada através do Mini-Mental Status Examination) e a experiência anterior em RV. Os resultados revelam correlações significativas, lançando luz sobre as diferenças relacionadas com a idade, a influência da capacidade cognitiva e o impacto da exposição prévia à RV no desempenho da tarefa. Esta exploração exaustiva representa um contributo essencial para o crescente campo das aplicações de RV nos cuidados de saúde, visando especificamente grupos de utilizadores limitados, como os doentes de UCI. Os resultados sublinham a importância de considerar as características do utilizador, a experiência anterior e a função cognitiva ao conceber intervenções de RV. Justifica-se a realização de mais investigação para aperfeiçoar as metodologias de avaliação e alargar o âmbito dos testes em doentes reais de UCI, abrindo caminho a melhores cuidados para os doentes e a melhores práticas de reabilitação. Palavras-chave: Realidade Virtual, Cuidados intensivos, Resiliência cognitiva, Terapias não-farmacológicas iv CONTENTS 1Introduction 1 1.1Context and Motivation 1 1.2Objectives 2 1.3Research Hypothesis 3 1.4Research Method 3 1.5Document structure 3 2VR interventions in the ICU 5 2.1Atomic Tasks 7 2.1.1Head Task 7 2.1.2Arms Task 7 2.1.3Fingers Task 8 2.2Atomic Mixed Tasks 9 2.2.1Pick and place Task 9 2.2.2Pointing Task 10 2.2.3Painting Task 10 2.3Extra Secondary Tasks 11 2.3.1Throwing Task 11 2.3.2Finding Task 11 2.3.3Omnidirectional Pointing Task 11 2.3.4Point and go Task 12 2.3.5Point and Grab Task 12 2.4Extra Secondary Tasks - Cognitive Training 12 2.4.1Magnetic Fishing Task 13 2.4.2Catching Butterflies Task 13 2.4.3Simon Says Task 13 2.4.4Piling Task 13 2.4.5Opening Task 14 2.4.6Mixing Task 14 2.5Summary 14 3Proposed Approach 16 3.1System Architecture 16 3.1.1Software Architecture 18 3.1.2Deployment Architecture 19 v contents vi 3.1.3Data Architecture 19 4Development 21 4.1Developed Tasks 23 4.1.1Head Task 23 4.1.2Arms Task 23 4.1.3Fingers Task 24 4.1.4Pointing Task 24 4.1.5Pick and Place Task 26 4.1.6Point and go Task 26 4.1.7Painting Task using free hands 27 4.1.8Pick and Place Task using free hands 27 4.2Tools 28 4.3Data gather 29 5Assessment 31 5.1Metrics (Parameters to measure) 31 5.2Experiment Setup 32 5.2.1First Experiment 32 5.2.2Second Experiment 35 5.2.3Third Experiment 35 5.3Results and Discussion 35 5.3.1First Trial Results 35 5.3.2Second Trial Results 38 5.3.3Variables Correlations 39 5.3.4Qualitative Analysis 40 6Conclusion 41 aAdditional tables 46 LIST OF FIGURES Figure 1 The sequence of events during each experimental trial from the participant’s point of view (Clark et al.,2020). 8 Figure 2V. Nepor , Overview of the Oculus controller buttons. 9 Figure 3Painting task with virtual hand gestures (Penumudi et al.,2020). 10 Figure 4Omnidirectional pointing task (Penumudi et al.,2020). 12 Figure 5Virtual reality block stacking game (Harris et al.,2020). 14 Figure 6Proposed architecture for the system. 17 Figure 7Alternative architecture for the system. 17 Figure 8Proposed software architecture for the system. 18 Figure 9Proposed deployment architecture for the system. 19 Figure 10 Proposed data architecture for the system. 20 Figure 11 Text guides of the tasks. 22 Figure 12 A representation of the timer on the HUD. 22 Figure 13 The developed Head Task. 23 Figure 14 The developed Arms Task. 24 Figure 15 The developed Fingers Task. 25 Figure 16 The developed Pointing Task. 25 Figure 17 The developed Pick and Place Task. 26 Figure 18 The developed Point and go Task. 27 Figure 19 The developed Painting using free hands Task. 28 Figure 20 The developed Pick and Place using free hands Task. 28 Figure 21 An example of a code using the cognitive3D module. 30 Figure 22 Small example of the data gathered on the cognitive3D website. 30 Figure 23 User sitting down. 32 Figure 24 User lying down. 33 Figure 25 A sample of the total data registered on Cognitive3D.33 Figure 26 A sample of the total data of Cognitive3Dabout the error events. 34 Figure 27 Targets and Completion Times in Scenario 1.36 Figure 28 Targets and Completion Times in Scenario 2.37 vii 2 VR INTERVENTIONS IN THE ICU The use of Virtual Reality (VR) technology in healthcare, particularly in the intensive care unit, is a relatively new and developing field. However, studies have suggested that VR can be a useful tool for improving cognitive function and reducing symptoms of delirium and anxiety in ICU patients. Some studies (Bruno et al.,2022b;Ong et al.,2020) have specifically focused on the use of VR in the ICU as a means of reducing delirium. These studies have shown that VR can be effective in reducing the incidence and duration of delirium, as well as improving cognitive function in ICU patients. Another area of research has focused on using VR to reduce anxiety in ICU patients. Studies (Donnelly et al.,2021;Gerber et al.,2017;Merliot-Gailhoustet et al.,2022;Turon et al.,2017) have shown that VR can be an effective tool for reducing anxiety and promoting relaxation in these patients. Additionally, other studies (Bruno et al.,2022b,a) have also explored the use of VR in the ICU as a means of providing distraction and improving overall well-being. These studies have suggested that VR can be a useful tool for improving the overall experience of being in the ICU for patients and their families. Currently, there are many articles (Merliot-Gailhoustet et al.,2022;Hopkins et al.,2015; Chen and Peng,2006;Brummel et al.,2013;Hofhuis et al.,2007;Karnatovskaia et al.,2014; Ferre et al.,2021;Bruno et al.,2022b;Donnelly et al.,2021;Bruno et al.,2022a;Ong et al.,2020; Gerber et al.,2017) reporting how important the use of VR intervention is, however, they still lack information relative to the parameters and variables used around the intervention itself. A more detailed study on this variables would allow a more deep understanding on how important VR intervention is on ICU patients and in which way does it help preventing cognitive decline. Recently it was published an article (Vlahovic et al.,2022) mentioning and studying VR game mechanics in order to better comprehend the impact it does on today’s VR video games, but it also lacks information on their implementation and testing with ICU patients. In the field of human-computer interaction, researchers have sought to understand and predict the speed and accuracy of movement for pointing tasks. One commonly used model for this is Fitts’s law, which was originally developed for two-dimensional pointing tasks on a 5 6 computer screen (Murata and Iwase,2001). However, with the rise of VR applications, there is a need to extend Fitts’s law to better understand pointing in three-dimensional (3D) space. Fitts’s Law is a fundamental principle in human-computer interaction that quantifies the relationship between the time it takes to select a target, the distance to that target, and the size of the target. It states that larger targets are easier to select, shorter distances lead to faster selections, and it offers a mathematical formula to predict movement time based on these factors. This is the focus of the article "Extending Fitts’s law to a three-dimensional pointing task" (Murata and Iwase,2001). The authors conducted a study to determine the accuracy and speed of movement for 3D pointing tasks in VR. They found that the traditional Fitts’ law model could not accurately predict pointing performance in VR and thus developed a modified version of the law specifically for 3D pointing. By understanding the accuracy and speed of movement in 3D pointing tasks in VR, we can better evaluate the user experience for individuals with constraints, such as those with motor impairments. For example, the modified version of Fitts’ law could be used as a benchmark to assess the effectiveness of VR interfaces designed for constrained users. In (Clark et al.,2020), the authors present a study on the extension of Fitts’s law in 3D virtual environments using low-cost VR technology. They carried out experiments to assess the relationship between target distance and selection time for objects located in different depths in a VR environment. The results showed that the traditional Fitts’s law formula does not accurately predict selection time for objects located in different depths in a VR environment, highlighting the need for further research in this area. The authors suggest that the current low-cost VR technology presents new challenges and opportunities for research into the human-computer interaction in VR environments. This study provides important insights into the limitations of current low-cost VR technology in predicting selection times for objects located in different depths, and underscores the need for further research to understand the implications of this technology for humancomputer interaction in VR environments. The findings of this study can help to inform future research on the evaluation of constrained users’ ability to interact with VR applications, particularly with regard to the impact of depth and other factors on selection time and accuracy (Clark et al.,2020). There are a few studies exploring the use of virtual reality in the ICU to provide distraction and improve overall well-being but they still lack information about the parameters and variables used for the intervention. Therefore, the implementation of basic virtual reality tasks in ICU patients and intensive testing is necessary to evaluate which tasks or complexity levels can be achieved. This approach will allow an overview of what elements should be considered for the implementation of virtual reality in ICU patients. In addition, it will allow us to identify the factors that 2.1. Atomic Tasks 7 may affect these patients’ ability to interact with virtual reality applications and improve their experience in the ICU setting. In our project, the approach adopted is to implement and test some basic VR tasks with ICU patients to identify which tasks and what levels were accomplished by patients in order to get an overview on which elements should be considered when going for a VR approach. The next subsections introduce the basic tasks that will be proposed to be completed by ICU patients, categorized by their atomicity. 2.1 atomic tasks The atomic tasks are the simpler and most basic tasks a patient can complete. This also means that subsequent and more complex tasks will use combinations of these atomic ones in order to create some more challenging activities. 2.1.1Head Task In this task, the patient will be asked to control an object using only his head. Starting from an easy and simple level where objects would appear in between X or Y axis range in a low angle range, to a level where the angle would start to raise and both axis would be used. Some related tasks were implemented on users outside of ICU context (Clark et al.,2020), but in this case, the task is simple enough to be used on ICU patients. Figure 1represents a good example of what this task must be. Note that the cube must move using only the patient’s head. This will give us important data on, for example, how many degrees could the patient’s head move, from side to side and up and down. 2.1.2Arms Task In this task, the patient will attempt to hit a object (something like a ball or a balloon) in the air in order to motivate the patient in raising the arms up and around to see how far up can the patient reach. For this to be measured, the idea is to start with low height objects, so it can be easy for the patient to elevate just his forearm, and, as the levels advance, the objects appear increasingly higher and more difficulty to reach. Note that this task only needs for the patient to raise the arms to a certain point, and it does not need any further action, such as, for example, pressing a button. 2.1. Atomic Tasks 8 Figure 1: The sequence of events during each experimental trial from the participant’s point of view (Clark et al.,2020). 2.1.3Fingers Task In this task, patients will try to press each and every button on the VR controller. The objective is to test how well the patient can press each button. For instance, it will be monitored how many users can press each button, how much time it took for them to press them, did they click on wrong buttons instead, how many times it was pressed, etc. This will give an overview of which buttons are easier and which buttons are harder to press. Starting on the easiest button to press, so supposing that the Triggers are the easiest button to press, then this button would be highlighted first, in order for the user to press it. Then we would progressively choose harder buttons to press in order to conclude which ones could not be pressed by most of the patients. Eventually, on the harder difficulty, we would ask for the patient to press for a series of buttons and also a combinations of buttons. The time a button is pressed it is also a variable to be studied and so, on this last levels, it would be asked to press the buttons for a longer time. In terms of VR scenery, each action will trigger an event so that the patient has some feedback each time a button is pressed. For example: once the Grip Button is pressed, a water fall would start flowing; or a dolphin would come out of the sea if the user pressed the B button. 2.2. Atomic Mixed Tasks 9 Figure 2: V. Nepor , Overview of the Oculus controller buttons. GameArter, 2020 [Online]. Available: https://www.gamearter.com/blog/xr-input-manager-controllers. [Accessed: 29-Oct-2022] 2.2 atomic mixed tasks In the new set of VR tasks below, some atomic tasks will be mixed so that it becomes possible to better comprehend which factors are affecting patient’s performance on realizing an activity. 2.2.1Pick and place Task Similar to one of the tasks done in the project developed by Vlahovic (Vlahovic et al.,2022), this task will be based on picking up an object and placing it on a target. Starting with a big target, the task will raise in difficulty by lowering the radius of the target so that the patient’s accuracy needs to be better/higher. As the target size changes, the object’s size itself is also a variable to change in order to increase difficult. So that a starting big object would be more easy to accomplish than a smaller object in harder levels. The 2.2. Atomic Mixed Tasks 10 orientation of the object itself is also a variable. Since it is easier to just place an object with no orientation, that would be the start for the first levels, and the last ones, the orientation would be taken in consideration so that the object needs to face a certain angle for the task to be completed. On top of this mentioned variables, the distance of the target and object is also considered. This means that the first levels the object and the target itself is going to be very close to the patient. On later levels, this is going to vary so that the target and object is positioned far from the patient so that it require extra effort and motion from the patient. 2.2.2Pointing Task In this task, based on Murata and Iwase article (Murata and Iwase,2001), the patient needs to point at a specific target for a couple of seconds. The difficulty on this task would be to reduce the size of the targets and to implement moving targets on harder levels. In addition, the trajectory of the moving target could vary upon harder levels as well as the speed of the target’s movement. 2.2.3Painting Task On the painting task, will be shown to the patient a figure on a rectangle. The patient then needs to clear or paint this figure in order to complete the task. For example, an all black rectangle with a yellow square on it. Then this yellow square needs to be painted black in order to make it disappear from the rectangle itself. Similar on how its done on the Penumudi’s project (Penumudi et al.,2020) shown in Figure 3. Figure 3: Painting task with virtual hand gestures (Penumudi et al.,2020). 2.3. Extra Secondary Tasks 11 2.3 extra secondary tasks These last tasks refers to some extra secondary tasks that will be developed later on. They are a mixture of atomic tasks and mixed atomic tasks, so that it will result on harder tasks to complete. The objective of these is to further test the capability of patients and to make them use their body and cognitive abilities. 2.3.1Throwing Task In this task, the patient will be asked to pick up an object and throw it at a specific target. The difficulty on this task would be to reduce the size of the target, as well make it movable in order to vary the throwing force. There is two ways to approach this task. The first possibility is to have the target on the ground so that the distance from the patient to the target will directly influence on the velocity of the object that needs to be thrown. This means that the further away the target is, the more force the patient needs to execute. The second possibility would be to have the target on a wall or a side plane. Unless the target is placed in a very high place or the position of the wall/plane can be moved, the need of a increased throwing force is nullified. This means that there would need to be an extra variable that would measure the force applied to the object when it hit the target. 2.3.2Finding Task In this task the user will be asked to find a certain object in between many diverse distracting objects. In order to complicate this task, the number of distracting objects will be higher, the size of the target object to find will be reduce, and also the colors and the pattern of the target, or objects around it, will be more camouflaged. 2.3.3Omnidirectional Pointing Task This tasks prompts the patient to point to one of the many targets omnidirectionally positioned. This means that, similar to the pointing task mentioned previously, there would be targets positioned in front of the patient. The idea is identical to one of the tasks used in Penumudi’s project (Penumudi et al.,2020), represented in Figure 4. The difficulty variables in this task would be the number and size of the targets and, as an extra variable, the targets would have circular movement in order to be harder to click precisely on them. 2.4. Extra Secondary Tasks - Cognitive Training 12 Figure 4: Omnidirectional pointing task (Penumudi et al.,2020). 2.3.4Point and go Task The concept on the point and go task is to test the patient movement around a section, so in this task the patient will be asked to point and click on a target in order to teleport it to the respective place. The difficulty would be the size of the targets and the distance to it. 2.3.5Point and Grab Task As seen in Poupyrev’s project (Poupyrev et al.,1997), in this task, the patient will be asked to grab a distant object by pointing at it and click grabbing the object. Similar to the pick and place task but in this one the objective is focus on picking the object with a pointing mechanism. It allow a different approach to the problem and different conclusions can be made if the outcome of the results vary too much. To complicate this task, some variables will be modified. Such as target’s size and the number of distracting objects around the target. 2.4 extra secondary tasks -cognitive training Lastly, this section is also about some extra secondary tasks that was proposed to be more complex than the previously ones. This means that these tasks will not only evaluate a more complex interaction of a task in VR but will also evaluate cognitive capability. 2.4. Extra Secondary Tasks - Cognitive Training 13 2.4.1Magnetic Fishing Task The idea of this task is to implement a fishing pool where fishes are standing still and the patient will be asked to fish target fishes in order to complete the task. The fishing rod would be similar to a magnet so that when close enough on a fish, it would hook it on to the rod. The starting levels would be just one or two big fishes and, at the latest levels, end up with more and smaller fishes. 2.4.2Catching Butterflies Task This task is similar to the previous one (Magnetic Fishing Task), but instead of catching fishes on a fishing pool, the patient will be presented with butterflies up in the air and a bug net tool to use it to catch them. The difficulty on this task would be to add more butterflies to catch and their size, as well adding movement to them. 2.4.3Simon Says Task The idea on this task is to mimic the well known "simon says" memory game. In this task, the patient will be shown some buttons with colors and a sequence on them. This sequence would be show as the form of lights turning on and off on the buttons and also with sounds backing the lights up, so that its easier to memorize the sequence. The first levels would be just 3or 4buttons and short sequences. After raising the levels, more button can be implemented and longer sequences will be shown. Also, the size of the buttons as well infinite sequences can also be implemented. 2.4.4Piling Task In this task, patient will be asked to stack blocks in order to create a tower without them falling. Much similar as the well known game "Jenga" but reversed, that is instead of removing blocks, the patient will stack them together. This idea was based on Harris’s project (Harris et al.,2020), where the game he created is shown in Figure 5. For a more harder version, the number and size of the objects to stack will vary upon levels. 2.5. Summary 14 Figure 5: Virtual reality block stacking game (Harris et al.,2020). 2.4.5Opening Task In this task, it will be shown to the patient a box and various keys. The patient then will need to choose the correct key for the box. This task can be harder by raising the number of keys available to the user, by lowering the size of the key hole or the keys itself. 2.4.6Mixing Task The idea of this task is to have many beakers with different colors liquids. The patient then would need to pour these liquids on a larger beaker in order to create a potion. The difficulty on this task would be to raise the number of liquids to mix, the size of the main beaker to pour liquids into, and input an order to pour the liquids in order to respect a particular sequence. 2.5 summary We presented a series of VR tasks to test the patient’s physical and cognitive abilities divided into two categories: Atomic tasks (Head task, Arms task, Fingers task) and Atomic mixed tasks (Pick and Place task, Pointing task, Painting task). These tasks range from simple movements such as controlling an object with the head or hitting a moving object with the arms, to more complex tasks that require the patient to perform multiple actions and use their cognitive abilities. The difficulty of the proposed tasks shall increase as the patient progresses through the study. 4 DEVELOPMENT Before starting the development of the planned tasks, some basic concepts were settled and aesthetics were thought and implemented in the VR world. After an initial creation of the VR world, some elements were edited to make it acceptable in the ICU context. For example, the letter size was adjusted so that it can be better read. It was concluded that the size of the letter initially created were allegedly too small for the ICU patients to read. So, for better and more clear reading, the size of the letters were increased. The models and colors were adjusted based on feedback from specialists in the field. To maintain a cozy and cheerful atmosphere in the room, the colors were changed to vibrant and lively hues. The decision to create a room, as opposed to another scenario, was arbitrary but driven by the need for a space that promotes comfort, familiarity, and reduces overstimulation. Figures 11a,11band 11cshows some examples of text that will be displayed to guide the challenges proposed to the ICU patients. In Figure 11a, another extra element can be seen called SCORE. This element is a counter incremented, one by one, as the task is being completed. For example, each hit to the targets on a task adds 1point to the score, so if the score is currently at 0, it will be increased to 1. Another element that was later added on was a timer so that if the patient is not being able to complete a certain task, the game can still keep running automatically and the next task shown. Figure 12 represents the Head-up display (HUD) that will be shown to the user so that he can be aware of the time remaining for the conclusion of the task. As it is shown in Figure 12, the timer image is going to be subsequently consumed until there is no HUD image left, representing the end of the task. This timer duration is set to be one minute, but can be adjusted later on, depending on the patient tests. This timer allow for a better control over the tests without the active intervention of a helper. This is especially useful for when using the standalone version of the VR devices, that is, without a laptop supporting/running the application. 21 22 (a) Head Task. (b) Arms Task. (c) Fingers Task. Figure 11: Text guides of the tasks. Figure 12: A representation of the timer on the HUD. 4.1. Developed Tasks 23 4.1 developed tasks In the context of this Master’s Project, a total of 8tasks were implemented and tested on different types of patients. It is these tasks that will now be detailed. 4.1.1Head Task The Head Task is the first and one of the most important basic tasks. Since this task only needs the head movements to be completed, it is completion is very important because the subsequently task will have this type of head movement in order to complete them. So this task is the base of all the others atomic tasks (first atomic level). Figure 13 shows an example of the task developed. The blue ball represents the target and the user has to touch it with the green laser. This green laser is attached to the head, so whenever the user moves the head it also moves the laser accordingly. After this, it will spawn harder targets on higher and wider positions so that it is possible to measure the max reach of the head movement for the user playing it. Figure 13: The developed Head Task. 4.1.2Arms Task The second task is called Arms Task and it allows to understand if the user has mobility of the arms. This task requires some head movement (like it was mention previously) and also the movement of the arms. Subsequently tasks will also have this atomic task in the base of them but since this one is already using an atomic task (Head Task) behind it, it is consider as a second layer of the atomic level, assuming the previous one was on the first layer level. 4.1. Developed Tasks 24 Figure 14 shows an example of the task developed. The red ball is the target that needs to be touched. The user needs to simply move his arms to touch the ball. After this, another target appears on the left side in order to test the left arm as well. When this is done, subsequently harder targets will spawn on harder positions, such as, on a more higher and wider positions. With this, it is possible to conclude on how far the user can reach the targets using his arms. Figure 14: The developed Arms Task. 4.1.3Fingers Task The third task is called Fingers Task and its objective is to test whether or not the user can press the different buttons on the controllers. Since on the later on tasks it will be asked to press specific buttons to interact with the task, it is one of the important atomic tasks since without completing this one, the subsequently tasks will be harder to complete. This task is consider as a second layer of the atomic level. It is consider the same layer as the previous task because this one does not need any type of arms movement, but it always need the head movement to look around. Figure 15 shows an example of the task developed. Notice that there is a highlighted button on the right controller which is the task’s target. The user is asked to press each highlighted button until the task is complete. 4.1.4Pointing Task The fourth task is called Pointing Task and its purpose is to understand whether or not the user has the precision to point at a certain target. This task is consider as a third layer of the atomic level because it also requires some arm movement. So, and because it 4.1. Developed Tasks 25 Figure 15: The developed Fingers Task. requires previous done atomic tasks, it should be in a higher level layer. Furthermore, since subsequently task will require precision motion, this task is also considered atomic. Figure 16 represents an example of the task developed. The purple ball is the target that needs to be touched with the pink laser. Note that the laser is coming out of the user’s hands (first the right hand and eventually the left hand) and the user has to point it, or touch it, to the ball. After touching it, the ball/target will get subsequently smaller and in different positions. After this, it is replicated, with the same ball size and positions, to the left side of the room and the laser is moved to the left hand. Figure 16: The developed Pointing Task. 4.1. Developed Tasks 26 4.1.5Pick and Place Task The fifth task is called Pick and Place Task, and its objective is to understand if the user is able to transport an object from one place to the other. Since this task includes doing more than one movement tested on the previous tasks, it is not considered an atomic task and therefore it is considered a normal primary task. Figure 17 shows an example of the task developed. The green ball on top of the right pedestal is the target and it is required that the user picks it up and place it on the left pedestal. After this is done, the ball and the pedestals will decrease in size. Figure 17: The developed Pick and Place Task. 4.1.6Point and go Task The sixth task is called Point and go Task, and it allows to understand if the user is able to have the precision to point at one place and click on a button to teleport into it. This task is also not considered an atomic task due to include concepts/movements from the atomic tasks mentioned previously. Figure 18 shows an example of the task developed. Note that the big blue circle zone on the right side of the figure is the target and it is required that the user uses the left controller analog to teleport to this zone. It can also be seen, on the left side of the figure, the trajectory line or preview of the teleport being done when the user holds the left analog stick. After teleporting to the first zone, another zone is shown in a different place so that the user is forced to look and move around. 4.1. Developed Tasks 27 Figure 18: The developed Point and go Task. 4.1.7Painting Task using free hands The seventh task is called Painting Task using free hands, and its objective is to test if the user is able to clear a painting using his own hands. It will be asked for the user to lay down the controllers and use the hands to complete the task. Figure 19 represents an example of the task developed. The target is the painting on the right side of the figure and it is required for the user to clean the yellow paint that it is on it. 4.1.8Pick and Place Task using free hands The eight and final task is called Pick and Place Task using free hands, and its purpose is to test the same task done before but now using the free hands method. Similar to the Pick and Place task, it will be asked for the user to pick the ball from one pedestal to the other but using his own hands. Figure 20 shows an example of the task developed. The light green ball is the target that the user has to pick it up and pass it to the left pedestal on the figure. 4.2. Tools 28 Figure 19: The developed Painting using free hands Task. Figure 20: The developed Pick and Place using free hands Task. 4.2 tools The application was developed using a software called Unity which is a cross-platform game engine developed by Unity Technologies. The editor version used on the development of this 4.3. Data gather 29 project was the ’2022.2.19f1’. It is also to be noted that all the 3D models used in this project were taken from the official Unity Asset Store. The device used to test the application was the Oculus Quest 2, a VR headset developed by Reality Labs, a division of Meta Platforms. The resolution employed for testing was 4128 x 2096 pixels, with a refresh rate set at 72 Hz. The hand tracking method was implemented using the official SDK downloaded from the Meta Quest website. 4.3 data gather In order to better gather and organize several user’s data and performance, an application called Cognitive3Dwas used. Cognitive3D is a software company that provides spatial analytics solutions for virtual, augmented, and mixed reality applications. Their platform collects and analyzes data on user behavior in 3D environments, providing insights into how users interact with products, environments, and training simulations. This software was used for tracking the performance of each patient in order to conclude which targets were completed, how many time it took and overall progression of the test. Figure 21 shows a small example of one of the parts of the code written using the cognitive3D module. As shown in Figure 21, the following line: 1new Cognitive3D . CustomEvent ( " ( Task 1) Looked at t arg et " + i ) . Send ( ) ; allows to initiate an event to cognitive3D application. This sends data to its cloud or it is stored on the computer if there is no active internet connection. It will automatically upload the buffer stored on the local computer to the cloud once there is a internet connection available. Once a test is done to an user, its data is easily seen on the cognitive3D software. Figure 22 represents an example of some of the data gathered. Note that all the data gathered here was then transported to an Excel sheet for further analysis and conclusions. 4.3. Data gather 30 Figure 21: An example of a code using the cognitive3D module. Figure 22: Small example of the data gathered on the cognitive3D website. 5.3. Results and Discussion 37 User is lying down User ID Task Targets completed Total time Errors Score Total time (all tasks) Total time (all tasks with timeouts) Head 0/80s (-timeout) - Arms 4/812s (-timeout) - Fingers 8/8 17s0 Head 0/80 (-timeout) - Arms 4/825 (-timeout) - Fingers 8/8 50 39 User 112/24 32s137s User 212/24 75s180s Table 3: Data gathered on lying down users. Figure 28 plots the total time and times of the respective targets that each User took to complete the second and the third tasks. Figure 28: Targets and Completion Times in Scenario 2. As it can be seen in Figure 28, the Head Task was not possible to execute while lying down. This was due to the fact that the targets spawn first close to eye level (assuming the standing/sitting position). So while lying down, the target really gets out of eye and head position, resulting in the impossibility of completing the target and eventually the task. 5.3. Results and Discussion 38 There are targets in this task that would, most likely, be able to be completed, but since the approach of showing the targets to the user is spawning the second target after the first one is completed, the user could not try to complete the latter targets. This may be changed in the future to better analyze the limitations of each patient in the ICU. 5.3.2Second Trial Results The second results are divided into two categories: The people who tested the first version of the application which consisted in 3tasks; And the people who tested the second version of the application which consisted in 8tasks. Note that the second version of the application include the first 3tasks of the first version, so it is possible to place together the data of these two interventions to better and more firmly conclude some results. It is also noted that there was some positive correlations when the data was analyzed with only 8people but, it was decided to conclude with all 15 people. The reason why 15 people and not 16 as said before (8people on first experiment and 8on the second) is because one of the user that tested in the first experiment, also tested the second one, which was more advanced. This was interesting to see if the performance would be much better than the other users which it actually end up being true (user A256 end up having much better score on second run of experiments). It is also to be noted that, in this second results, all the test were made sitting down, leaving the laying down part for future work. Table 5represents the first results of tests done to the first 8users. After this results, another set of users were tested but this time with an improved version. Note that this version has increased number of targets on the first tasks, as well as more tasks implemented. But, in order to have more number of people to analyze, it is only shown the performance of the first 3tasks. Table 6represents the data processed from the second experiment. As it can be seen, the user A256 was the only user that was on both experiments and as such, it had an noticeable increase of score on the second round of experiments. This could mean that being able to have some experiment or have at least a first try at VR can help understand and be more natural at doing task and perform them better. Since for most of the users, this was their first time trying out VR, most of them if not all of them have a harder time to adapt and interact with this new technology. This was noticeable while doing the tests as the users were very scared to take actions or even moving the head or looking around. It is to be noted that this 16 tests were aggregated in two test sessions, with the first session being tested on a 3-task application, and the second session on a improved 8-task application. Since only 8users were tested on each application version, it was decided to 5.3. Results and Discussion 39 merge the first 3tasks (since they were the same) and analyze the correlations with a higher number of users for a stronger conclusion. 5.3.3Variables Correlations After analyzing and organizing the data gathered from all users, it was time to correlate them with the data given through MMSE. Table 4shows the Pearson correlations coefficients of the variables measured. Just to clarify, the columns variables Score and Total Time represents the score and total time of all tasks without any timeouts mentioned previously. And on the rows, all of the variables were gathered by a specialized person and are self explanatory. Only needs to be noted the BMI acronym stands for Body Mass Index and was calculated by taking the person’s weight in kilograms and divide it by the square of height in meters. Score Total Time Sex -0,032 0,065 Age -0,423 -0,420 Scholarship 0,127 0,087 MMSE_score 0,312 0,040 Height_cm -0,055 -0,143 BMI 0,155 0,186 Body_fat_percentage 0,171 -0,006 Table 4: Pearson correlations coefficients. As it can be seen, most of the correlations are close to 0, which indicates no correlation at all. But there are some of them that are almost closer to 0.5, which means that they have some correlations but not that strong correlated. The negative values means that they are negative correlated, which means that when one variable raises, the other one decreases. The "Age" variable exhibited a negative correlation of approximately -0.42. While this correlation isn’t particularly strong, it does suggest that age inversely affected both the "Score" and "Total Time". Specifically, as age increased, both "Score" and "Total Time" decreased. The interpretation of the "Total Time" variable is nuanced: when there are no timeouts, a user who did not complete any tasks would register 0s for "Total Time". Evaluating using the variable that accounts for timeouts is problematic, as a large number of users reached a timeout due to the increasing difficulty of tasks at later levels, leading to a convergence in time values. This means that a direct relationship between increasing age and decreasing time isn’t as straightforward as one might expect. In this context, the decreased time is understandable, given that unfinished tasks result in shorter times recorded for the variable. 5.3. Results and Discussion 40 The MMSE_score exhibited a modest correlation with the Score variable, suggesting that cognitive ability is related to user performance. While the results aren’t as robust as desired, these preliminary indicators point out some research avenues worth further exploration. 5.3.4Qualitative Analysis While a User Experience (UX) survey was not integrated into the testing procedures, an overall analysis of the UX can still be derived from personal opinions. The majority of users expressed satisfaction with their experience experimenting with VR technology, particularly as it was their first encounter with such a novel technology. One user even described it as an amazing experience and expressed excitement about trying it again in the future. However, a few users reported feeling a bit queasy after spending 1to 2minutes in the test. Despite this, they persevered and completed the full test. Additionally, nearly half of the users admitted to harboring reservations about trying and experimenting with VR. Many were hesitant to move their heads around, possibly due to unfamiliarity with the technology, resulting in them mostly remaining stationary during the test. This highlights the importance of conducting a pre-training session to allow participants to acquaint themselves with VR glasses and controllers before progressing to the actual test. 6 CONCLUSION In this Master’s Project, we embarked on a comprehensive journey to evaluate the ability of constrained users, particularly those with physical disabilities or cognitive impairments, to interact with virtual reality applications. The investigation delved into the realm of virtual reality systems, scrutinizing their adaptability and applicability for these unique user groups. A first objective of this Master’s project was to establish a set of guidelines to design VR applications that cater to the distinct abilities and needs of individuals facing cognitive and physical challenges. Some experiments were conducted in settings designed to mimic the real-life constraints experienced by patients in an Intensive Care Unit (ICU). User actions and behaviors were meticulously recorded, scrutinizing the impact of age, experience with VR technology, and cognitive abilities on their performance. In the first experiment, it was observed a notable variation in user performance based on age and familiarity with VR technology. It was apparent that prior experience with VR technology could lead to improved user performance, while age exhibited a negative correlation with performance. The data collected indicated that some tasks were especially challenging for users, reinforcing the importance of designing user-friendly VR applications for individuals with cognitive and physical limitations. In the second experiment conducted, it was further investigated the impact of age, experience, and cognitive abilities. The inclusion of an improved VR application with more tasks provided a broader perspective. A significant improvement in user performance in the second trial, especially for the user with previous exposure to VR technology, highlighting the potential benefits of familiarity and practice. Additionally, it was explored correlations between user characteristics and their performance in VR tasks. While the correlations were not overwhelmingly strong, the project findings suggest that cognitive abilities, as measured by the MMSE, are related to user performance in virtual reality applications. This shows the importance of considering the cognitive capabilities of users when designing VR applications for individuals facing cognitive challenges. 41 42 The research carried on along this Master’s work underscores the need for more extensive studies, particularly in real ICU settings, to validate our findings and draw more conclusive results. The complex interplay of factors affecting user performance in VR environments calls for continued exploration. In conclusion, this project provides a foundation for future research and application development in the realm of virtual reality for constrained users. Designing VR applications that account for cognitive and physical limitations holds promise in improving the quality of life for these individuals, and this thesis paves the way for more inclusive and accessible VR experiences. BIBLIOGRAPHY N E Brummel, T D Girard, E W Ely, P P Pandharipande, A Morandi, C G Hughes, A J Graves, A Shintani, E Murphy, B Work, B T Pun, L Boehm, T M Gill, R S Dittus, and J C Jackson. Feasibility and safety of early combined cognitive and physical therapy for critically ill medical and surgical patients: the activity and cognitive therapy in ICU (ACT-ICU) trial. Intensive Care Med,40(3):370–379, November 2013. Raphael R Bruno, Nico Bruining, Christian Jung, Malte Kelm, Georg Wolff, Bernhard Wernly, and the VR-ICU Study group. Virtual reality in intensive care. Intensive Care Medicine, 48(9):1227–1229, September 2022a. Raphael Romano Bruno, Georg Wolff, Bernhard Wernly, Maryna Masyuk, Kerstin Piayda, Susannah Leaver, Ralf Erkens, Daniel Oehler, Shazia Afzal, Houtan Heidari, Malte Kelm, and Christian Jung. Virtual and augmented reality in critical care medicine: the patient’s, clinician’s, and researcher’s perspective. Crit Care,26(1):326, October 2022b. Yang Chen and Nana Peng. A design of virtual reality based coma stimulation program system. In 2006 7th International Conference on Computer-Aided Industrial Design and Conceptual Design, pages 1–6,2006. doi: 10.1109/CAIDCD.2006.329453. Logan D. Clark, Aakash B. Bhagat, and Sara L. Riggs. Extending fitts’ law in threedimensional virtual environments with current low-cost virtual reality technology. International Journal of Human-Computer Studies,139:102413,2020. ISSN 1071-5819. doi: https://doi.org/10.1016/j.ijhcs.2020.102413. URL https://www.sciencedirect.com/ science/article/pii/S1071581920300173. Miranda R Donnelly, Renee Reinberg, Kaori L Ito, David Saldana, Meghan Neureither, Allie Schmiesing, Esther Jahng, and Sook-Lei Liew. Virtual reality for the treatment of anxiety disorders: A scoping review. Am J Occup Ther,75(6), November 2021. Maria Ferre, Edgar Batista, Agusti Solanas, and Antoni MartÃnez-Ballesté. Smart healthenhanced early mobilisation in intensive care units. Sensors,21(16), 2021. ISSN 1424-8220. doi: 10.3390/s21165408. URL https://www.mdpi.com/1424-8220/21/16/5408. Stephan M. Gerber, Marie-Madlen Jeitziner, Patric Wyss, Alvin Chesham, Prabitha Urwyler, René M. Müri, Stephan M. Jakob, and Tobias Nef. Visuo-acoustic stimulation that helps you to relax: A virtual reality setup for patients in the intensive care unit. Scientific 43 BIBLIOGRAPHY 44 Reports,7(1):13228, Oct 2017. ISSN 2045-2322. doi: 10.1038/s41598-017-13153-1. URL https://doi.org/10.1038/s41598-017-13153-1. David Harris, Mark Wilson, and Samuel Vine. Development and validation of a simulation workload measure: the simulation task load index (sim-tlx). Virtual Reality,24(4): 557–566, Dec 2020. ISSN 1434-9957. doi: 10.1007/s10055-019-00422-9. URL https: //doi.org/10.1007/s10055-019-00422-9. Jose G M Hofhuis, Peter E Spronk, Henk F van Stel, Guus J P Schrijvers, Johannes H Rommes, and Jan Bakker. The impact of critical illness on perceived health-related quality of life during ICU treatment, hospital stay, and after hospital discharge: a long-term follow-up study. Chest,133(2):377–385, October 2007. Ramona O Hopkins, Karen Choong, Carleen A Zebuhr, and Sapna R Kudchadkar. Transforming PICU culture to facilitate early rehabilitation. J Pediatr Intensive Care,4(4): 204–211, December 2015. Lioudmila V Karnatovskaia, Margaret M Johnson, Roberto P Benzo, and Ognjen Gajic. The spectrum of psychocognitive morbidity in the critically ill: a review of the literature and call for improvement. J Crit Care,30(1):130–137, October 2014. Lili Merliot-Gailhoustet, Chloé Raimbert, Océane Garnier, Julie Carr, Audrey De Jong, Nicolas Molinari, Samir Jaber, and Gerald Chanques. Discomfort improvement for critically ill patients using electronic relaxation devices: results of the cross-over randomized controlled trial e-choisir (electronic-choice of a system for intensive care relaxation). Critical Care,26(1):263, Sep 2022. ISSN 1364-8535. doi: 10.1186/s13054-022-04136-4. URL https://doi.org/10.1186/s13054-022-04136-4. Atsuo Murata and Hirokazu Iwase. Extending fitts’ law to a three-dimensional pointing task. Human Movement Science,20(6):791–805,2001. ISSN 0167-9457. doi: https://doi. org/10.1016/S0167-9457(01)00058-6. URL https://www.sciencedirect.com/science/ article/pii/S0167945701000586. Triton L Ong, Matthew M Ruppert, Maisha Akbar, Parisa Rashidi, Tezcan Ozrazgat-Baslanti, Azra Bihorac, and Marko Suvajdzic. Improving the intensive care patient experience with virtual Reality-A feasibility study. Crit Care Explor,2(6):e0122, June 2020. M H Park, D Y Kwon, J M Jung, C Han, I Jo, and S A Jo. Mini-Mental status examination as predictors of mortality in the elderly. Acta Psychiatr Scand,127(4):298–304, August 2012. Sai Akhil Penumudi, Veera Aneesh Kuppam, Jeong Ho Kim, and Jaejin Hwang. The effects of target location on musculoskeletal load, task performance, and subjective discomfort during virtual reality interactions. Applied Ergonomics,84:103010,2020. ISSN 0003-6870. BIBLIOGRAPHY 45 doi: https://doi.org/10.1016/j.apergo.2019.103010. URL https://www.sciencedirect. com/science/article/pii/S0003687019302194. Ivan Poupyrev, Suzanne Weghorst, Mark Billinghurst, and Tadao Ichikawa. A framework and testbed for studying manipulation techniques for immersive vr. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST ’97, page 21–28, New York, NY, USA, 1997. Association for Computing Machinery. ISBN 089791953X. doi: 10.1145/261135.261141. URL https://doi.org/10.1145/261135.261141. Marc Turon, Sol Fernandez-Gonzalo, Mercè Jodar, Gemma Gomà, Jaume Montanya, David Hernando, Raquel Bailón, Candelaria de Haro, Victor Gomez-Simon, Josefina LopezAguilar, Rudys Magrans, Melcior Martinez-Perez, Joan Carles Oliva, and Lluís Blanch. Feasibility and safety of virtual-reality-based early neurocognitive stimulation in critically ill patients. Annals of Intensive Care,7(1):81, Aug 2017. ISSN 2110-5820. doi: 10.1186/s13613-017-0303-4. URL https://doi.org/10.1186/s13613-017-0303-4. Sara Vlahovic, Mirko Suznjevic, and Lea Skorin-Kapov. A framework for the classification and evaluation of game mechanics for virtual reality games. Electronics,11(18), 2022. ISSN 2079-9292. doi: 10.3390/electronics11182946. URL https: //www.mdpi.com/2079-9292/11/18/2946. A ADDITIONAL TABLES This section presents supplementary tables that provide further details and data related to the research. These tables are not included in the main text to maintain readability but are important for a comprehensive view of the results. 46