Eye communication system for nonspeaking patients

Faculdade de Engenharia da Universidade do Porto Eye Communication System for Nonspeaking Patients Maria Soares da Eira VERSÃO PROVISÓRIA Dissertação realizada no âmbito do Mestrado Integrado em Bioengenharia Major Engenharia Biomédica Orientador: Prof. Dr. Jean Lorenceau Co-orientador: Prof. Dr. Aurélio Campilho Setembro 2014 ii iii The present document is based on the work carried out under the direct supervision of the investigator Jean Lorenceau, researcher at Laboratoire des Systèmes Perceptifs de Paris, and the co-supervision of Aurélio Campilho, professor in the department of Electrical and Computer Engineering of Faculdade de Engenharia da Universidade do Porto. © Maria Soares da Eira, 2014 iv v Jusqu’alors, je n’avais jamais entendu parler du tronc cérébral. Ce jour-là, j’ai découvert de plein fouet cette pièce maîtresse de notre ordinateur de bord, passage obligé entre le cerveau et les terminaisons nerveuses, quand un accident cardiovasculaire a mis ledit tronc hors circuit. Autrefois, on appelait cela « transport au cerveau » et on mourait en toute simplicité. Le progrès des techniques de réanimation a sophistiqué la punition. On en réchappe mais flanqué de ce que la médecine anglo-saxonne a justement baptisé le locked-in syndrome : paralysé de la tête aux pieds, le patient est enfermé à l’intérieur de lui-même avec l’esprit intact et les battements de sa paupière gauche pour tout moyen de communication. ("Le Scaphandre et le Papillon", Bauby, 1997) vi vii Resumo Após um internamento nas unidades de cuidados intensivos (UCI), os pacientes relatam frequentemente experiências e vivências negativas. A frustração e desconforto sentidos são, em parte, devidos a dificuldades de comunicação causadas pela presença de ventilação artificial e intubação. Uma vez que a capacidade de comunicação por meios convencionais, como a fala e gestos, é limitada, devem ser encontrados métodos alternativos. Até ao momento, não se encontra disponível nenhum método padrão e eficaz para melhorar a comunicação não-verbal nas UCI. O seguimento ocular (eye tracking) representa uma solução promissora, uma vez que, mesmo em condições de invalidez severa, o movimento dos olhos é geralmente controlado. Um seguidor ocular (eye tracker) é um dispositivo que mede a posição e movimento dos olhos, e pode ser utilizado para interagir ou comunicar com o meio ambiente. Nesta dissertação é apresentada uma revisão da investigação realizada na área de eye tracking, desde o momento da sua invenção até aos atuais dispositivos disponíveis comercialmente. Apesar de ter sido reconhecido como um recurso poderoso, há ainda desafios a superar, incluindo o conflito entre a ação de ver e selecionar - the Midas touch problem -, calibração, ruído no sinal e limitações ao nível da precisão. Combinando a informação relativa às necessidades dos pacientes com as vantagens e limitações dos eye trackers, pretendeu-se criar um sistema de comunicação com o olhar para pacientes conscientes e intubados ou ventilados. A solução proposta consiste num eye tracker e numa interface gráfica apresentando ícones representativos das necessidades básicas e preocupações do paciente. Após uma calibração rápida, o paciente informa a sua intenção/necessidade olhando para o ícone desejado. Este é selecionado após um determinado tempo de observação e um sub-menu é aberto, a fim de especificar a opção anterior. Testes de usabilidade foram efetuados em pessoas saudáveis, de forma a validar a funcionalidade do sistema de comunicação. Os movimentos oculares foram registados durante a visualização da interface gráfica, e foi realizado um inquérito de forma a avaliar a experiência dos utilizadores. Relativamente aos desafios inerentes à utilização do eye tracker como periférico de entrada, foram propostas e testadas diversas soluções que se revelaram adequadas e funcionais. viii Os resultados obtidos através dos testes de usabilidade confirmam a funcionalidade do dispositivo médico e viabilizam a passagem para a fase de teste em pacientes. Assim, após um processo de regularização a nível de segurança e saúde, uma equipa de especialistas de pneumologia do Hospital de Pitié-Salpêtrière em Paris, irá avaliar o contributo do dispositivo médico na comunicação não-verbal e na redução das fontes de desconforto do paciente. ix Abstract A stay in an intensive care unit (ICU), although potentially life-saving, may be a traumatic experience to patients. The frustration reported is partly due to communication difficulties caused by the presence of artificial ventilation and intubation. Once the ability to communicate by conventional ways, as speech and gesture, may be limited, alternative methods must be found. So far, there is no standard, reliable and effective tool to improve non-verbal communication in the ICU. Eye trackers represent a promising solution, since even in high disability conditions, eye movement is usually controlled. An eye tracker is a device for measuring eye’s positions and movement and can be used to translate the intention of the person into functional interactions or as a communication way with the surrounding environment. This work, presents a review of the research carried out in the eye tracking field, since the moment of its invention to the current and commercially available devices. Although it has been proved to be a powerful solution, there are still challenges to overcome, including the conflict to distinguish casual viewing from the desire to produce intentional commands - “the Midas Touch” problem -, calibration, noise and accuracy issues. Combining the information concerning the patient's needs with the advantages and limitations of eye trackers, we intended to create a system to communicate with the eyes addressed to conscious and intubated or ventilated patients. The proposed solution consists on an eye tracker and a graphical user interface displaying representative icons of the basic needs and concerns of the patient. After a quick calibration, the person just needs to look at the desired icon on the screen to select it and subsequently open a sub-menu in order to specify the previous choice. Usability tests were performed to access the functionality of the communication system. Eye position was recorded during observation of the graphical user interface and a survey was conducted to evaluate user’s experience. Solutions were proposed and tested addressing the challenges inherent the use of eye tracking as an input device and the outcomes confirm its validity and effectiveness. xvi Figure 4.13 – GUI for eye communication: Anxiety page. .......................................... 44 Figure 4.14 – GUI for eye communication: TV page. ................................................ 45 Figure 4.15 – GUI for eye communication: Free communication page. .......................... 45 Figure 4.16 – GUI for eye communication: Keyboard page. ........................................ 45 Figure 4.17 – GUI for eye communication: Preferences page. ..................................... 46 Figure 4.18 – GUI for eye communication: example of a Piano page. ............................ 46 Figure 4.19 – Eye communication system installation for the usability tests. .................. 48 Figure 4.20 – Variation of the observational angle depending on the distance to the screen when observing an icon with 150x150 pixel area. ...................................... 48 Figure 5.1 – Free exploration of the Welcome page from the GUI for eye communication. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 2500 frames. Data of the eye’s coordinates are represented by yellow circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ............................... 54 Figure 5.2 – Free exploration of the Welcome page from the GUI for eye communication. Records were performed with The Eye Tribe™ in 60 Hz mode, during 2500 frames. Raw data without axis offset. Data of the eye’s coordinates are represented by blue points. The first twenty coordinates (x,y) are depicted in red. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ............................................ 55 Figure 5.3 – Successive observation of the icons presented on the Guided communication page from the GUI for eye communication. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 2500 frames. Data of the eye’s coordinates are represented by green circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. .................................................................... 56 Figure 5.4 – Observation of one icon from the Hunger page from the GUI for eye communication. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 1000 frames. Data of the eye’s coordinates are represented by yellow circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ............ 57 Figure 5.5 – Observation of the point corresponding to specific questions from the Pain page of the GUI for eye communication. Participants were asked to locate: a right leg cramp, a migraine headache and a stomach ache / colic. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 1000 frames. Data of the eye’s coordinates are represented by blue circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ........................................................ 58 Figure 5.6 – Observation of the region/word corresponding to specific questions addressing the Pain level page of the GUI for eye communication. Participants were asked to locate: mild pain, great pain and unbearable pain. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 1000 frames. Data of the eye’s coordinates are represented by white circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ........................................................ 59 Figure 5.7 – Subjects observation of the keys from the keyboard page of the GUI for eye communication. Participants were asked to write “HOUSE”, “AVION”, “HELLO” and “MAISON” respectively. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 2500 frames. Data of the eye’s coordinates are represented by blue circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. ...... 60 xvii Figure 6.1 – GUI for eye communication adapted to ALS patients: Guided communication page. .................................................................................................... 79 Figure 6.2 – GUI for eye communication adapted to ALS patients: Entertainment page. ..... 79 Figure 6.3 – GUI for eye communication adapted to ALS patients: Health page. ............... 80 Figure 6.4 – GUI for eye communication adapted to ALS patients: WC page. ................... 80 xviii Tables List Table 2.1 — Suggested AAC equipment and supplies to be used at ICU (Garrett et al., 2007). ................................................................................................... 10 Table 3.1 — Technologic specifications of The Eye Tribe™ tracker (The Eye Tribe™, 2014). . 34 xix List of Abbreviations AAC Assistive and Alternative Communication ADOREPS Organisation de la Recherche en Pneumologie et sur le Sommeil ALS Amyotrophic Lateral Sclerosis APHP Assistance Publique Hôpitaux de Paris API Application Programming Interface ATICE Adaptation To Intensive Care Environment CAM-ICU Confusion Assessment Method for the Intensive Care Unit COGAIN Communication by Gaze Interaction CPP Commité de Protection des Personnes EOG Electro-Oculography EOL EyeOnLine GUI Graphical User Interface HCI Human-Computer Interaction ICU Intensive Care Unit IP Internet Protocol IR Infra-Red JSON JavaScript Object Notation SGD Speech-Generating Device SLP Speech Language Pathologists TCP Transmission Control Protocol VOG Video-Oculography WPS Words Per Second xx 1 Chapter 1 Introduction “Communication is recognized as an essential human need and, therefore, as a basic human right. Without it, no individual or community can exist, or prosper.” (Centre for Communication Rights – CCR, 2011). Communication is the meaningful exchange of information between two or more living creatures through expression forms that include speech, signals, writing, or behaviour. Part of the civil and scientific responsibility is to provide communication access to all individuals, even if they are unable to achieve it on their own. 1.1 Motivation Each year, seven million people in the European Union are admitted to medical and surgical Intensive Care Unit (ICU) (European Parliament Statistics, 2014). More than one-third of those patients experience inability to speak because of intubation or mechanical ventilation (Fowler, 1997). This condition may cause feelings of panic and insecurity, sleep disturbances and stress levels (Bergbom-Engberg & Haljamae, 1989). Nurses also report impotence when patients are unable to express their symptoms, pain levels and needs (Alasad & Ahmad 2005). Occasionally, hospitalizations with expected duration of 2-3 days are extended, increasing the patient’s stress. Better understanding and interpretation of intentional messages of patients may improve assessment of symptoms, selection of treatments and even facilitate the expression of important thoughts and sentiments at the end of life. Besides, it is widely accepted that psychosocial factors are related to illness behaviour, and there is some evidence that they may influence the rate of recovery from post-traumatic disorders (Radanov et al., 1991). Therefore, it is of general interest to avoid acute stressful states, as it will improve patients’ health conditions and the overall logistics of the hospital system. 2 Literature demonstrates that providing assistive and alternative communication (AAC) strategies might enhance patients’ daily interaction and by extension, clinical outcomes (Patak et al., 2006). Although the problems associated with the inability to speak during critical illness have been clearly established, few solutions have been offered or systematically tested with ICU patients (Happ, 2001). The movement of the eyes can be used as an alternative communicating method for patients in the ICU who are unable to speak. The eyes are one of the most expressive features of the human body for non-verbal implicit communication. The human gaze is postulated to be the best indicator for attention or intention (Dalton & Ellis, 2003). Besides being a source of information, the eyes have very fast and thorough movements (Krauzlis, 2005). In the last century, eye tracking systems have had a significant evolution and offer, nowadays, a wide range of possibilities and different technical features, operating modes and application fields (Dongheng Li, 2006). Eye trackers have been employed in several areas, ranging from the fundamental scientific research to practical applications for HumanComputer Interaction (HCI). Therefore, an eye tracking system can be a powerful solution to solve the aforementioned communication problems of patients at ICU. The present work was conducted within the Laboratoire des Systèmes Perceptifs, at École Normale Supérieure in Paris, integrated in the EyeOnLine (EOL) project and based on the past experience of the team of Jean Lorenceau in eye tracking systems. The research was developed in collaboration with a medical team from the Pulmonary Service of the PitiéSalpêtrière Hospital. The evaluation of the medical device will be conducted by Doctor Camille Rolland-Debord in collaboration with Professor Alexandre Demoule. 1.2 Objective At the moment, there is no standard, reliable and effective tool to facilitate the nonverbal communication between patients and the medical personnel in the ICU. The main objective of this study is to develop a medical device to improve the communication ability of patients who are intubated or receiving mechanical ventilation. It will be applied an eye tracking system to control a graphical user interface (GUI) for communication with the eyes. The main concern is to inform patients’ symptoms, pain levels and needs in order to access their health condition and improve their day life at the hospital. An overhaul of the current assistive technologies, in particular eye trackers, as well as a general knowledge of the anatomy and physiology of the eye, is needed to better understand the functioning of these systems. Since this technology is directed to people at debility states, reliability, robustness, safety, and mounting issues must be carefully taken into account. Its interface must be intuitive, once the direct users will be patients of different ages and education levels. It is also desired an attractive and playful design in order to encourage its usage. 3 The device will be handled primarily by paramedics, so it is intended to be practical and easy to implement, in order to not be an extra charge on paramedics’ working routines. Preliminary tests conducted in healthy subjects will confirm the functionality and usability of the system. Finally, to evaluate the contribution of the medical device in facilitating non-verbal communication with the patient, a team of pulmonology specialists will assess patient discomfort, evaluate the intensity of the target discomfort and analyse the effectiveness of the medical device on reducing discomfort sources. 1.3 Contributions A medical device for communication via eye movements targeting nonspeaking patients at ICU was created. It was developed an application to integrate an eye tracker on the current system and extract the user eye’s position. A graphical user interface was designed to easily express patient’s symptoms, feelings and needs through the navigation on menus. An experimental protocol was elaborated to test healthy subjects while using the graphical interface developed in this work. A survey was conducted afterwards to evaluate the user’s experience. The data collected during these different tests was analysed and discussed. A user’s manual of the respective medical device was elaborated to be included into the health and safety regulation document. 1.4 Dissertation Overview This dissertation is divided in six chapters including: Introduction, Augmentative and Alternative Communication in the ICU, Eye Tracking, The Eye Communication System, Results and Discussion, Conclusions and Future Work. The Introduction describes the motivation, the main objectives, my contribution to this work, and finally, the present overview of the dissertation organization. Chapter 2 describes the problem which was the motivating engine of this research. It provides an overview of the actual discomforts and communication problems felt by patients at ICU, to better describe the underneath concern. The major topic is related to augmentative and alternative communication techniques directed to patients at intensive care units. It also gives a picture of the context to where it was created for - the PitiéSalpêtrière hospital - in order to better characterize the target population. Chapter 3 starts with a short description of the eye anatomy and physiology and it provides a definition of eye tracking followed by its history. After, it describes the existing 4 systems and technologies for eye tracking, focusing in the most common video-based eye trackers. Furthermore, it addresses the application of eye trackers as a scientific tool and for Human-Computer Interaction, discussing the current input eye tracking challenges and giving some examples of eye writing systems. In this chapter is also introduced the eye tracker used in this project, The Eye Tribe™. Chapter 4 describes the medical device created to accomplish the defined objectives. In short, it provides the methodology used in the development of the system and its general operating mode. Herein are also described the usability tests performed in order to evaluate the functionality of the device. It also delineates the regulatory issues to implement the medical device in the hospital. Chapter 5 reports the results and discussion of the usability tests carried out in healthy subjects. It analyses the user’s eye position recordings during the graphical user interface observation and the survey conducted after system usage. Chapter 6 discusses the potential features to be improved and outlines directions for future research. 5 Chapter 2 Augmentative and Alternative Communication in the ICU Augmentative and Alternative Communication (AAC) encompasses the methods used to supplement or replace the production and/or comprehension of spoken or written language. It is addressed to individuals with temporary or permanent impairments, activity limitations, and participation restrictions that constrain the normal speech-language (American SpeechLanguage-Hearing Association, 2005). After staying in the Intensive Care Unit, the survived patients have often painful memories of this experience. Such is caused naturally by the pain and discomfort arising from their health condition, but also due to a communication failure between the patient and the health personnel. In this chapter the main causes of patient’s discomfort in the ICU will be set out to characterize the problem and ascertain the major concerns of this study. The existence of a communication failure between patients and medical personnel creates a need for communication resources. This chapter addresses AAC technologies targeting patients in the ICU, reviewing the current methods and analyzing viable solutions. In order to be aware of the problem’s scale, it is necessary to locate it in the physical context of study. In this case, the medical device will be applied in the pulmonology service of the Pitié-Salpêtrière hospital, hence an overview of the key numbers of this service is necessary to understand the complete picture of the clinical need. 12 2.5 The Context of the Pitié-Salpêtrière Hospital The development of the present project aims the intensive care unit of the PitiéSalpêtrière Hospital. The APHP (Assistance Publique Hôpitaux de Paris), including 44 hospitals, is considered to be the largest hospital complex in Europe, and occupies the eighth position worldwide. The Pitié-Salpêtrière, with more than 400 years of existence, is the largest hospital of APHP. Some key numbers about this hospital and its patients in 2012 are listed below (APHP, 2014): • 2 132 beds • 11 poles of distributed activities • 87 713 number of short period stays (<24h) • 64 693 number of long period stays (>24h) • Average duration of hospital stays: 5 days. • 53% of the patients are intubated and ventilated via a breathing tube. • 32% of the patients are under non-invasive ventilation via a breathing mask. Hence, it can be concluded that, for about 85% of the patients, it is impossible to communicate by verbal language due to the physics constraints (intubation, artificial ventilation, mask...). The pulmonology service of this hospital is included in the pole “PRAGUES” shared with Anesthesia, Geriatrics, Emergencies and Explorations of the Pulmonary function. Within this pole some of the key numbers are listed below: • 152 beds • 2 591 number of short period stays (<24h) • 13 505 number of long period stays (>24h) Considering that this system will be applied, in a first instance, to patients at pulmonology department with long period stays at ICU, we can compute that this will be available for about 10 000 patients per year, just in this hospital. Once approved by the national health system, this communication method can be applied in other hospitals in France or even in the world. Thus, the developed device shows enormous potential in improving the quality of life of millions of patients around the world. 13 2.6 Concluding Remarks According to the previously published literature, pain, anxiety and dyspnea are the most referred discomfort sources during a patient’s stay in the ICU. The problem was studied with the medical team of the Pitié-Salpêtrière Hospital, which selected nine needs considered to be priority at ICU: - Pain - Anxiety - Lack of air - Excessive respiratory effort - Need to be aspired - Desire to be informed about their health state - Bad installation - Thirst Numerous studies demonstrated that the use of AAC techniques in the ICU increases patient’s sense of control and emotional health and may have a positive impact on healthcare outcomes. A literature canvas proved that, to date, the current techniques to improve communication in the ICU do not meet the needs of patients and caregivers. Several reasons can explain this: - the lack of a consensus validated technique on a sample of patients, - for some of them, the need for learning is in practice not feasible at ICU because it would require hospitalization programming, - difficulties related to the presence of contention or myopathy can limit gesture, - additional workload for medical and paramedical personnel, - significant expenditures in high technologies, - infant-like communication language. Concluding, for critically ill patients that are either intubated or mechanical ventilated, there is, so far, no standardized reference system neither an effective available tool to improve non-verbal communication in the ICU. In this sense, this work aims to develop a new tool which overcomes the constraints of the ICU stay, based on the detection of eye movements. The aforementioned needs will be included in medical device setting and ultimately, they will be the object of assessment when testing the device at the hospital. Besides those, other items will be included namely hunger, anxiety and TV control as suggested by medical professionals. The analysis of the current number of patients at Pitié-Salpêtrière hospital, has shown that there is a large population of people who can potentially benefit from this system. 14 15 Chapter 3 Eye Tracking After discriminating the problem and the need, it is time to propose a viable solution. This chapter will demonstrate how eye tracking can be an answer to the previously described problem, providing a review of the current methods and analysing the advantages and disadvantages of this technique. Eye tracking is a technique whereby the individual’s eye movement is measured, so it is possible to know both where a person is looking at any given time and the motion of an eye relative to the head. An eye tracker is a device for measuring eye’s positions and movement, as well as eye features such as pupil diameter and blink rate (Poole and Ball, 2005). Video-oculography (VOG) and electro-oculography (EOG) are the main technologies incorporated into commercially available eye trackers. The eye trackers market is dominated by video-based trackers due to their non-invasiveness and high accuracy (Bates, 2006). Eye movement information can be applied into two strands. Firstly, it can provide an objective source of how individuals observe the scenery and what kind of eye movements are performed in specific contexts. Eye trackers are powerful tools in scientific research of the visual system, psychology, cognition, medical diagnosis and rehabilitation. Additionally, they can be used in market research and advertise testing, providing user evaluation data and clues as to design more attractive brand images (Dongheng Li, 2006). Secondly, eye movement information can be applied in Human-Computer Interaction (HCI) research, either in usability-evaluation studies to enhance the user-friendliness of interfaces, or for capturing people’s eye movements as an input mechanism to control system functions. This represents a major advantage for certain populations of users who are temporarily unable to communicate verbally or even permanently unable to move their limbs (Pinheiro, 2011). 16 3.1 Anatomy and Physiology of the Ocular Motor System To study eye tracking systems, it is necessary to comprehend both sides of the interaction – the eye tracker and the human eye. Before describing in detail the first one, this section will present general knowledge about the eye anatomy and physiology, in order to understand the basic principles that will be developed in the next chapters. The eye is the organ responsible for the vision, by detecting light and converting it into electrochemical impulses in neurons. 3.1.1 Eye Anatomy The eye is not shaped like a perfect sphere, being rather a fused two-piece unit (Horn & Leigh, 2011). An image of eye anatomy is shown in figure 3.1, depicting a right-eye from the front and inside point of view. Figure 3.1 – Right eye viewed from the front and cross section of the eyeball viewed from above (Custers, 2014). The smaller frontal unit, more curved, is called the cornea and is linked to the larger unit called the sclera. The cornea is a transparent coat that covers both the pupil and the iris. This is the first and most powerful lens of the optical system and allows, together with the crystalline lens, the production of a sharp image at the retinal photoreceptor level. The sclera forms part of the supporting wall of the eyeball and appears white in the eye image. 17 The cornea and sclera are connected by a ring called the limbus that surrounds the iris (Judd & Wyszecki, 1975). The iris, the coloured part of the eye, is a circular muscle that controls the size of the pupil so that more or less light, depending on the environmental conditions, is allowed to enter the organ. This aperture appears dark because of the absorbing pigments in the retina. The sensory part of the eye, the retina, has two different types of light sensors: rods and cones. While rods have a high sensitivity to brightness and enable night vision, the cones are less sensitive and detect chromatic light. Most parts of the retina have a very low density of light receptors, and only a small circular area with a diameter of 1º to 2º, called fovea, has a high density of cones. Everything inside this region is seen in detail, while everything outside this narrow field is seen indistinctly. Thus, people see only a small portion of any full scene accurately in time. This narrow vision generates the need to move the eyes rapidly around to form a full view of the world. 3.1.2 Eye Movements Each eyeball is rotated by six extraocular muscles that are organized as antagonistic pairs and give three degrees of freedom. One pair is responsible for horizontal movement, another pair controls the vertical movement, and the third pair allows rotational movement around the direction of view. Figure 3.2 depicts the three types of eye’s muscles responsible for the different movements. Figure 3.2 – Anterior view of the right eye showing the three types of muscles that provide the different movements (OpenStax CNX, 2014). Eye movements are divided into six different types, each of which performs a specific, quantifiable function. The six types are: fixations, saccades, smooth pursuit, vergence, vestibular and optokinetic (Sharpe & Wong, 1986). Fixations are the moments when the eyes are relatively immobile, holding the image of a stationary target on the fovea to encode the displayed information (McConkie et al. 1988). 18 Saccades are discrete ballistic movements occurring between fixations. The purpose of most saccades is to move the eyes to the next viewing position. Visual processing is automatically suppressed during saccades, to avoid blurring of the visual image. Figure 3.3 presents a famous picture of the pioneering study made by Yarbus showing how eyes scan a human face through fixations and saccades (Yarbus, 1967). Figure 3.3 – Eye movements while watching a girl’s face (Yarbus, 1967). Smooth pursuit is performed while eyes are tracking an object's movement, so that its moving image can be retained on the fovea. It compensates the motion of the visual target and, thus, minimizes the drift of the target’s image across the retina that could otherwise blur the image and compromise visual acuity. Optokinetic helps to stabilize the eyes during head and body movements by minimizing the motion of the entire visual surrounding field. Vestibular movements stabilize the image on the retina during brief head movements, helping to maintain the vision clear. Vergence occurs when eyes move in opposite directions so that images of a single close target are placed simultaneously on both foveae. Considering eye movements in smaller time and length scales, drift, tremor and microsaccades can be also mentioned. These small movements occur even during fixations to keep the nerve cells in the retina active and to correct slight drifting in focus (Engbert & Mergenthaler, 2006). Several measures can be extracted from these basic movements, such as the gaze, which corresponds to the sum of all fixation durations within a prescribed area. The main measurements used in eye-tracking research are fixations, saccades and smooth pursuit movements. Gaze, pupil size and blink rate are also studied (Poole et al., 2005). The typical duration of each movement is naturally different. While fixations last for 218 milliseconds on average, with a range of 66 to 416 milliseconds, saccades typically last for 20 to 35 milliseconds (Poole et al., 2005). Some studies have demonstrated that the eye’s dynamics may be influenced by the kind of activity performed by the person, namely by the regular practice oh high level sports (Williams, 2002). More recently, investigations have shown that the control of some kind of eye movements may even be learnt specifically the smooth pursuit eye movement as it will be demonstrated hereafter (Lorenceau, 2012). 19 3.2 The History of Eye Tracking The first qualitative descriptions of eye movements date back to the 18th century. The first data regarding eye movements were obtained by Porterfield at 1737, through introspection or observation using a mirror, telescope or peephole. In the 19th century, Javal and Lamare observed the eye movements during reading and introduced the French word “saccade” for the abrupt movements of the eye. They used a mechanical coupling of the eyes and the ears using a rubber band to make the eye movements audible. Initially, the eye tracking devices that produced objective and accurate data were highly invasive and uncomfortable. At the end of that century, Delabarre developed a system that used an eye cup with a lever extending do draw the eye movements on a surface covered with soot. The eye cup was attached directly to the surface of a sufficiently “cocainized eye” (usually Delabarre’s eye) and had a hole in it through which he could see (Wade & Tatler, 2005). In the beginning of the 20th century, appeared the first unobtrusive measurements. In 1901, Dodge and Cline used a photographic method and light reflections from the eye movements in horizontal direction only. Some years later, in 1905, Judd applied motion picture photography in eye movement recording. The invention of motion picture photography gave the possibility of frame-by-frame analysis of the eyes’ motion and enabled quantitative research on a solid basis (Horsley et at., 2013). In 1939, Jung measured vertical and horizontal eye movements simultaneously, with electrodes applied on the skin close to the eyes. This method, known as Electrooculography (EOG), measures the electric field of the eyeball, which is a dipole and will be covered in the next chapter. In the 1950s, Alfred L. Yarbus did important eye tracking research, having his book often been quoted (Yarbus, 1967). He showed that the task given to a subject has a very large influence on the eye movement and wrote about the relation between fixations and interest (Majaranta, 2009). Figure 3.4 is often referred to as evidence on how the task given to a person influences his or her eye movement. He also gave important contributions to the understanding of eye movements (see figures 3.3 and 3.10). 20 Figure 3.4 – Study showing that eye movements are dependent on the person’s task (Yarbus, 1967) In the 1970s, there was great progress in eye-tracking technology. The eye trackers became less intrusive, provided better accuracy, and were able to dissociate eye from head’s movements by multiple reflections from the eye (Cornsweet & Crane, 1973). In the 1980s, the development of computing power made possible real-time eye tracking and this enabled the use of video-based eye trackers for human-computer interaction (Bolt, 1981). It was also the time of the emergence of assistive technology systems aimed directly at people with disabilities (Majaranta and Räihä, 2002). These first systems were typically based on eye typing, where the user could produce text by using the focus of gaze as a means of input. One of the earliest eye typing systems, the Eye-Letter-Selector, detected eye movements with two phototransistors attached to eyeglass frames (Ten Kate et al., 1979). From the 1990s up to now, there has been a steady increase in the use of eye trackers. Tracking systems falling prices led to their wider use, typically in marketing research or usability studies (Majaranta, 2009). 21 3.3 Current Tracker Types Eye trackers measure rotations of the eye as well as additional information, such as blink frequency and changes in pupil diameter. There are several ways to extract these features, but generally they fall into three main categories. One type requires the use of large contact lenses or in-eye magnetic coils that cover the cornea and sclera, with a metal coil embedded around the edge of the lens. Figure 3.5 shows a scleral coil contact lens, which was inserted in the eye of the subject. Eye movements are then measured by fluctuations in an electromagnetic field, when the metal coil moves along with the eyes. These measurements have provided extremely sensitive recordings of eye movement and are the method of choice for medical and psychological research to gain insight into human behaviour and perception (Duchowski, 2003). Figure 3.5 – A scleral coil contact lens being inserted into a subject’s eye (Kumar, 2007). The second type, the already mentioned Electrooculography, relies on electrodes placed on the skin around the eye that measure its steady electric potential field. The method may seem impractical to the everyday life, but commercially available devices such as EagleEyes™ (figure 3.6) have revealed great success in augmentative and alternative communication area due to their high accuracy (Gips et al., 1993). Another advantage is the ability to detect eye movements in total darkness and even when the eyes are closed, allowing the performance of sleep cognitive studies (Elbert, Lutzenberger, Rockstroh & Birbaumer, 1985). Figure 3.6 – EOG approach – EagleEyes™ - for eye tracking measures the potential difference between eye muscles (Gips et al., 1993). 28 characteristics. User’s posture changes leads to a different position/angle of the gaze. These issues have been studied, and some authors have proposed solutions. Stampe and Reingold proposed a correction of the drift, by dynamically realigning the gaze position to the centre of any object selected while using the software (Stampe & Reingold, 1995). It is assumed that the user is looking at the centre of the object he wishes to select. If the point of gaze does not match the coordinates of the object’s centre, an automatic drift correction is then applied. The Tobii tracker uses data from both eyes to minimize drifting effects (Tobii, 2006). The system may continue with data from one eye, if the other is deteriorated. This enables longlasting calibration with very little drifting and saves the user from continuous recalibration. Nevertheless, some problems inherent to certain medical conditions deserve further indepth studies in order to optimize the current systems, and to make them available to any physical condition. For example, thick eyeglass lenses or frames may cause extra reflections. When contact lenses are used, the reflection is obtained from the surface of the contact lens instead of the cornea. Both situations can cause problems if the lenses are displaced over time, causing degradation in tracking accuracy. This may be prevented or minimized through careful condition’s setup, for example, by controlling illumination and camera position. Finally, most eye trackers are powerless over severe involuntary head or eye movements. Certain medical conditions prevent successful calibration (Donegan et al., 2005). If the calibration fails, some systems provide default calibration. Additionally, there are already some systems where no calibration is needed. A commercial Eye Contact Sensor called Eyebox2TM has this property (Dickie, Vertegaal, Sohn & Cheng, 2005). When looking directly at the combination of an eye tracker camera and an infrared LED, the glint of the corneal reflection appears in the centre of the pupil. The attention sensor only detects whether the glint is inside the pupil, to know whether an eye is looking at it or not. 3.6.4 The Midas Touch Problem The primary function of the eyes is to enable vision. Therefore, using the eyes for computing input might result in conflicts. This effect is commonly referred to as the “Midas Touch” problem (Jacob, 1991). Since the eyes are used for both perception and control, the system should be able to distinguish casual viewing from the desire to produce intentional commands. Misinterpretation by the interface can trigger unwanted actions. Distraction by moving or blinking might also cause conflicts. The challenge is to design an intelligent interface to minimize false activations and to disambiguate the user’s intention from the user’s attention. To overcome this limitation, it is common to combine some other modality for selection. If the person is able to produce some voluntary movement, a separate switch can be activated to select the item in focus, such as a blink, a sip, a wrinkling of the forehead, or 29 even smiling or other muscle activity available to the user (Barreto et al., 2000; Fono & Vertegaal, 2005; Huckauf & Urbina, 2008). Since people blink naturally several times per minute, an intentional blink needs to be longer than an automatic, i.e. longer than 300-400 ms. If the user is capable of moving only the eyes or has very limited motor control, the system must be able to separate casual viewing from intentional eye control. The most common solution is to use dwell time, i.e. prolonged gaze with duration longer than the normal fixation (about 200 ms), typically, 500-1000 ms. Fixations longer than 1000 ms are often broken by blinks or saccades or may be tiring to the eyes. Additionally, systems based on dwell time cause slower interaction that can be speed up for more able or experimented users. Therefore, the dwell time might be adjusted to the user (Majaranta et al., 2006). Another solution is to create a special selection area or an on-screen confirming button (Yamada & Fukuda, 1987; Ohno, 1998). For example, in the “quick glance” method developed by Ohno, selection is done by first fixating briefly on the desired command button and then confirming the selection by quick glance a selection area. These movements can be detected via electromyography or using the same video images, removing the need for additional switch equipment. 3.7 Eye Writing Systems The basic methods for producing text by gaze have been researched since the early 1980s. Text entry methods can be categorized according to complexity of the input movement - eye switches, direct gaze pointing, gaze gestures, continuous pointing gestures and, more recently, smooth pursuit eye movement (Majaranta & Raiha, 2007; Lorenceau, 2012). Eye switches, direct gaze pointing and smooth pursuit eye movements, whose performance will be evaluated based on the typing speed, are further characterized in this section. Typing speed is measured in words per minute (wpm) where a word is any sequence of five characters, including letters, spaces, punctuation, and others (MacKenzie, 2003). For a complete list of the commercially available eye writing systems, consult COGAIN network (COGAIN, 2014). 3.7.1 Eye Switches In some physical conditions or health states, people may have difficulties in fixating or moving the eyes in one direction, for example, in the lock-in syndrome (Donegan et al., 2005; Chapman, 1991). In such cases, voluntary eye blinks or winks can be used as a binary switch (Grauman et al., 2003). This is the most rudimentary method but, unfortunately, is the only applicable in some conditions. 30 A famous example is the memoir “Le Scaphandre et le Papillon”, written by JeanDominique Bauby (figure 3.15). In 1995, the French editor of the Elle magazine, suffered a stroke causing a locked-in syndrome. He wrote an entire book describing the feelings of this condition by just blinking his left eyelid. Locked-in his own body, the writer defines his body as useless but still painful - ''my hands, lying curled on the yellow sheets, are hurting, although I can't tell if they are burning hot or ice cold''. (Bauby, 1997). A transcriber repeatedly recited the alphabet frequency-ordered in the French language (E, S, A, R, I, N, T, U, L …), until Bauby blinked to choose the desired letter. The average writing rate was one word every 2 minutes (0,5 wpm). The book took about 10 months and 200 000 blinks to write but become a number one bestseller across Europe (The New York Times, 1997). Figure 3.11 - Photograph of Jean-Dominique Bauby paralyzed, dictating his last book “Le Scaphandre et le Papillon” one letter at a time by eye movements to Claude Mendibil before dying of heart failure (Mallon, 1997). 3.7.2 Direct Gaze Pointing The most common way of eye typing is direct pointing by looking at the desired letter. A typical setup has an on-screen keyboard with a static layout. An external eye tracker monitors the eye movement and a specific software is responsible for the post processing. The user starts to focus the desired letter by looking at one of the keys of the virtual keyboard. The system gives feedback on the selected letter by jittering, highlighting the item or by moving a cursor over the key in focus, among several other ways. Once focus is achieved, the item can be selected via different cues, such as a blink, a wink, or even a wrinkle or any other muscle activity. For severely disable people, dwell time is often the best and the only selection method. If the user keeps focusing a key long enough, then the key is selected. The period time is typically between 500-1000 ms, but some software programs offer the possibility of adjusting this time to the user. The dwell time determines the typing speed, typically below 10 wpm. Experienced users may require considerably shorter dwell times as low as 200-300 ms, which, naturally, increases the text entry rate, achieving 20 wpm. The selected letter appears in the text field, often located above the keyboard. The system may also give feedback on the successful selection by speaking out the letter or playing a “click” sound. Different feedback methods can be applied to support the writing task. This feedback should be considered carefully. If the user needs to switch the point of 31 gaze from the virtual keyboard to the typed text field, to review the previously written text, the writing process will become very time consuming and, therefore, inefficient. Typically, the keyboard has a QWERTY layout design, where all characters are visible and letters can be directly pointed at and selected. Alternately, the keys and controls can be organized hierarchically or by most common letters of certain language. Figure 3.17 shows a typical QWERTY on-screen keyboard, while figure 3.16 displays an alternative ABCD keyboard. Figure 3.12 - EC Key, an example of a gaze-driven keyboard using an ABCD structure (Istance et al., 1996). Special techniques such as automatic word prediction can be used to speed up the text entry, with constantly changing and adapting keyboard layouts. It is the fastest eye typing method, with an average rate of 15 wpm for an expert user. Figure 3.17 shows an example of on-screen keyboard, highlighting the most probable next letter. Figure 3.13 - Letter prediction is used to highlight the next probable letters on the keyboard. The empty boxes between the text input field and letter keys are filled with predicted words if word prediction is enabled (MacKenzie & Zhang, 2008). 3.7.3 Smooth Pursuit Eye Movements Learning to control smooth pursuit movement is the latest method of cursive writing with eyes. In contrast to the current and aforementioned systems, in which it is only possible to type letters and words with the aid of virtual keyboards, Jean Lorenceau has developed a novel method, which enables people to draw and write by using free eye movements (Lorenceau, 2012). Although the ocular system has a high precision when it comes to follow a moving subject, it is not capable of performing smooth and regular movements, when drawing a voluntary trajectory. Indeed, in front of a static background, eyes only perform saccades showing a very rudimentary muscle control (Lisberger et al., 1987). Using a visual illusion called reverse-phi, it is possible to control smooth pursuit eye movements and perform 32 regular trajectories. The device takes advantage of this effect by displaying several hundred disks, whose contrast varies over time on a screen. While moving the eyes over this flickering background, the users feel that the disks move with the displacement of their eyes (figure 3.18). Figure 3.14 - Retinal spatiotemporal luminance profile with static and moving eyes. An eye movement produces a reverse-phi stimulus on the retina, generating a positive feedback to the pursuit system. The resulting directionally broadband motion flow (red arrows) provides a perceptual substrate to orient pursuit and control eye trajectory. (Lorenceau, 2012). Since the human eye is capable of following, with precision, moving objects, the illusory movement of the disks, induced by the movement of the eyes, acts as a feedback effect that allows users to perform smooth trajectories. An eye tracker records the movement of user's eye and a software tracks the position of the pupil over time and compiles them, processing the designed letters by segmentation of the ocular trace (figure 3.19). Figure 3.15 - Examples of eye-generated digits after segmentation. Vertical (blue) and horizontal (red) eye position and eye velocity are also shown (Lorenceau, 2012). Thereafter, a computational model based on Bayesian inference can translate the information through a probabilistic method of character recognition (Diard, Rynik & Lorenceau, 2013). 33 The training process to control smooth eye muscle movements typically consists of 4 to 6 training sessions that last for about 30 minutes, in different days. The subjects first learn to perceive the reverse-phi movement and then to progressively use it as support for drawing figures. In the beginning, it can be a tiring task, but well-trained individuals create automatisms that facilitate writing, reaching an eye writing speed almost equal to the one obtained with hand writing. Even if the throughput is lower with this method than with virtual keyboard, it can allow motor impaired people to continue expressing themselves in an emotionally rich manner, despite their motor disability. 3.8 The Eye Tribe™ The Eye Tribe™ tracker is a low-cost and small device created by “The Eye Tribe”™ Company and it was used for the studies presented in this dissertation. In 2013, the Danish start-up launched their eye tracking technology. This device, shown in figure 3.10, has broken the record for smallest eye tracker device in the world, measuring 20×1.9×1.9 cm (The Eye Tribe™, 2014). Figure 3.16 – The small dimensions of The Eye Tribe™ (20 x 1.9 x 1.9 cm) (The Eye Tribe™, 2014). Currently, the company is selling the eye tracker for $99 along with a software development kit using C++, C# and Java programming platforms for developers to start incorporating the technology into their apps. The Eye Tribe™ tracker starts by detecting the face region, applying a face detection method, and then it searches for the eyes based on the bright pupil detection method. A high resolution sensor combined with the IR illumination system provides the conditions for track the tiny movements of the pupils with high precision while maintaining a wide field of view. A pair of (x, y) eye gaze coordinates are calculated with respect to the screen with an average accuracy of around 0.5 to 1º of visual angle. Assuming the user sits approximately 60 cm away from the screen/tracker, this accuracy corresponds to an on-screen average error of 0.5 to 1 cm. The eye tracker can be mounted in a tripod to be used in a desktop or in a specific support when used on a tablet. The summary of the tracker features is displayed in the table 3.1. 34 Table 3.1 — Technologic specifications of The Eye Tribe™ tracker (The Eye Tribe™, 2014). Sampling rate 30 Hz and 60 Hz mode Accuracy 0.5° – 1° Spatial Resolution 0.1° (RMS) Latency < 20 ms at 60 Hz Calibration 9, 12 or 16 points Operating range 45 cm – 75 cm Tracking area 40 cm x 30 cm at 65 cm distance (30 Hz) Screen sizes Up to 24” API/SDK C++, C# and Java Data output Binocular gaze data Dimensions (W/H/D) 20 x 1.9 x 1.9 cm Weight 70 g Connection USB 3.0 Superspeed In order to track the user’s eye movements and calculate the on-screen gaze coordinates, the tracker must be placed below the screen and pointing at the user. The user needs to be located within the Tracker’s trackbox. The trackbox is defined as the volume in space where the user can theoretically be tracked by the system. The size of the trackbox depends on the frame rate, with a higher frame rate offering a smaller trackbox. Figure 3.17 – On the left, the right position of The Eye Tribe™ related to the screen and the user. The tracker must be placed below the screen and pointing at the user. On the right, the tracking area known as trackbox, which is defined as the volume in space where the user can theoretically be tracked by the system (The Eye Tribe™, 2014). Prior to using an eye tracker, the user needs to undergo a calibration process. As it was already mentioned, the eye tracking software needs to model the eye characteristics of each person in order to estimate gaze accurately. A typical user calibration process of The Eye Tribe™ takes approximately 20 seconds to complete. It consists in a circular target that 35 is displayed at different locations of the screen on a blank background during around 2 seconds each. The user needs to look at the target as this is displayed on the screen (figure 3.12). Once all the calibration targets have been displayed on the screen the calibration process is completed. A star rating evaluation indicates if the process was successfully finished. A minimum of 9 calibration locations covering most of the screen is recommended. Using more locations (e.g. 12 or 16) will improve the accuracy of the gaze coordinates computed by the system. Figure 3.18 – On the left, The Eye Tribe™ user interface. There is a feedback that indicates the user if he is in the right position and therefore the eyes are correctly detected. On the right, the calibration process and the evaluation window above indicating if the process was successfully finished (The Eye Tribe™, 2014). Gaze coordinates correspond to the point on the screen that the user is currently looking at and are defined as pixels in a top-left oriented 2D coordinate system (figure 3.13 on the left). They are available in both raw and smoothed forms for the left and right eye. Additionally, pupil coordinates, pupil size and information about the state of the eye tracking process are also given. Pupil coordinates are the position of a tracked person’s pupil relative to the tracker sensor and are defined in normalized relative values (figure 3.13 on the right). Figure 3.19 On the left, gaze coordinates in pixels in top-left oriented 2D coordinates system. On the right, pupil coordinates in normalized values relative to tracker device (The Eye Tribe™, 2014). 36 At this stage, The Eye Tribe™ tracker is not bundled with any applications for common use, and it is intended only for software developers. Therefore, it is necessary to develop the script according to the specific purpose. 3.9 Concluding Remarks The general survey of eye tracking systems allowed to get acquainted with important aspects of these systems, namely the different formats and methodologies that are offered but also the current limitations to take into account. The “Midas Touch” problem, as well as calibration, noise and accuracy issues are still challenges to overcome in this field. Depending on the specific problem and need, efforts should be made in order to find the best solution to each problem. Regarding the available eye-writing systems, three strategies have been analysed. Eye switches are the most basic but do not provide effectiveness in writing speed. Cursive writing with smooth pursuit eye movements, although allowing great freedom of expression, requires several training sessions. Finally, keyboards controlled by direct gaze pointing seem to be the best option in this case. It does not require prior learning and it is available for immediate use. That was thereby the solution of choice for this project. By reviewing the wide-ranging technologies and solutions, it can be concluded that the approach that best fits the problem and need might be a remote video-based eye tracker. It should also be small, light and practical but ensuring a good accuracy and high data transfer rate. The Eye Tribe™, chosen for the performance of the present study, presents all these features. 37 Chapter 4 The Eye Communication System The previous survey of the actual problems and needs as well as the review of the current eye tracking systems were the basis for developing an adequate eye communication system for nonspeaking patients. The introduction of the medical device is expected to promote non-verbal communication with patients, to improve the understanding of their wishes, and act in anticipation, in order to define the user’s requirements and to reduce the anxiety at ICU. Meetings with the hospital team were carried out in order to optimize the entire system and fulfil the patients’ needs. The research team will remain in contact with the medical personnel for continuous optimization and validation of the overall system and future projects. This chapter describes the methodology that has been adopted for the creation of the medical device, the usability tests on healthy subjects to verify its effectiveness and the following steps of device testing in patients. 4.1 The Equipment Once the device will be applied in a hospital environment, ergonomics is extremely important. The hardware was thought to be lightweight and of practical usage. The eye tracker used in this project was the already mentioned The Eye Tribe™. This device has several advantages regarding its size, weight and low-cost. The use of a laptop allows conceiving a mobile, practical and economic solution, applicable in the hospital’s everyday life. To connect the eye tracker, it is necessary that the computer features an USB 3.0 port, usually found on the most recent computers. The main screen is used for controlling the eye tracker and additional features, while a second screen displays the communication GUI. The technical research team continues working to find more ergonomic solutions using a single screen or even a tablet. 44 Discomfort This menu intends to minimize the patient’s discomfort. Thus, the corresponding button is selected in case of feeling tired (Je suis fatigué), hot (J'ai chaud) or cold (J'ai froid) or needing mobility assistance (Mauvaise installation). Figure 4.11 – GUI for eye communication: Discomfort page. Breathing This menu allows reporting eventual pulmonary problems that might occur after an operation or treatment. Therefore, following the specialists recommendations, there are available the options: lack of air (Manque d'air), congestion (Encombrement), need to be aspired (Nécéssité d'etre aspiré) and respiratory effort (Effort respiratoire). Figure 4.12 – GUI for eye communication: Breathing page. Anxiety This menu tries to minimize the frustration felt by patients who cannot communicate their anxiety. The wide range of emotions was reduced to four elementary options: loneliness (Solitude), stress (Stress), sadness (Tristesse) and fear (Peur). Figure 4.13 – GUI for eye communication: Anxiety page. 45 TV Once the patient is alone for many hours, the television is a major concern. This menu allows the patient to say if he wants to change the channel (Changer de chaine), the volume (Moins fort / Plus fort) or even turn off the TV (Éteindre). Figure 4.14 – GUI for eye communication: TV page. Free communication Once in the Welcome Page, the patient can chose a free communication by selecting the respective button and opening the page of figure 4.15. The patient has then the possibility to answer questions posed by the nursing staff selecting "OUI" (yes), "NO" (no) and “JE NE SAIS PAS” (I do not know). If patients wishe to clarify their answer they can choose the “CLAVIER” and to write freely. Figure 4.15 – GUI for eye communication: Free communication page. Between the aforementioned methodologies for eye writing (see section 3.7), the direct gaze point strategy using a QWERTY keyboard was chosen, since it does not require previous learning and its usage is fast and universal. Keyboard In this case, the typical QWERTY keyboard was adapted to French speakers, and so keys are disposed in an AZERTY structure. This allows patient to write any word by simply look at each key. After a dwell time, the letter is selected and appears at the top of the screen. There are also the options “Espace” to make a space and “Reset” to delete. Figure 4.16 – GUI for eye communication: Keyboard page. 46 SOS By observing this icon, an alarm is triggered for 17 seconds. Preferences In this menu, the patient can set the dwell time for icons’ selection. The "+" provides a longer duration while the "-" a shorter one. Figure 4.17 – GUI for eye communication: Preferences page. In the future, ludic menus can be added to play games or even make music with eyes. Piano An example of a ludic piano is shown in figure 4.18. By staring at each key, the corresponding note is played loud. Figure 4.18 – GUI for eye communication: example of a Piano page. 4.4 Addressing the Input Challenges Certain strategists attempted to address the reported input eye tracking challenges. The noise coming from the restless movement of the eyes was atenuated by an average of the (x, y) coordinates. The Eye Tribe™ gives the information regarding the eye position in both raw and smoothed forms. In the present system, it was used the average data that filters microsaccades, eliminating some of the movement’s noise. As seen before, eye’s accuracy is limited to the size of foveal region which corresponds to 1º to 2º. In order to increase the accuracy of the system’s response, we created icons with large dimensions. Each icon has a circular dimension of 150 x 150 pixel which corresponds to 5° visual angle from the user, on a 17-inch display with a resolution of 1024 x 768 pixel viewed from a distance of 50 cm. Under each image there is an explanatory text, resulting in 47 a sensitive area of 200 x 200 pixel (approximately 7° visual angle). The spacing between icons was also considered. Although variable, depending on the number of icons per page, efforts were made to ensure that the spacing between two consecutive icons would be greater than 100 pixels. Large and spaced targets minimize the choice uncertainty caused by fixation difficulty and provide a comfortable navigation through the menus. In order to solve the “Midas Touch” problem, certain strategies were useful. First, in each page it was attempted to leave a central empty space with no action or information. Some studies demonstrated that in free viewing of static images, gaze fixations are biased toward the screen centre (Tseng et al., 2009; Smith & Mital, 2013). For that reason, the empty space in the centre allows people to have a general overview without selection. Secondly, it was applied the already mentioned dwell time solution. Once a normal fixation has an average period of 200 ms (Poole et al., 2005), we need to use a gaze with longer duration. By default, it is set a dwell time of 1000 ms but this parameter can be adjusted by the user. Each icon starts to jitter when under fixation, as a feedback signal. Other solutions could have been applied as a feedback signal such as icon’s highlighting or color change. Furthermore, outside the selection areas a small red point follows the eye gaze so that people know exactly where the fixation is all the time. The usefulness of this feature is controversial, since it can be considered as a distraction source. However, in most cases it was helpful to control the gaze movement. Moreover, this point has an additional function on the need for recalibration that will be further explained. Since The Eye Tribe™ is not head-mounted, calibration is an important process. Once set, very sudden movement of the head/body will lead to bias in the data. However, in this case, unobtrusiveness was a crucial factor in the context in which it was inserted. This factor was clogged with a tracking of both eyes minimizing drifting effects. A remote eye tracking system constantly updates the user’s position, however, it is difficult to avoid deviations on the calibration derived from marked changes of the initial position. Our assumption is that, in this case, the gaze-locked point also helps the user himself to detect a shift in calibration. Ultimately, aware that the point does not coincide exactly to where he is looking at, the user can try to make an empirical fit to match the point with the desired icon. This can be a temporary solution until a new recalibration. However, recalibration might not be a critical issue, since patients are normally bedridden and with reduced mobility. These conjectures may be confirmed in future, during the usability tests in healthy subjects and in patients at ICU. 4.5 Usability Tests In order to evaluate the developed graphical user interface, usability tests were performed in four healthy subjects. Their eye movements were recorded while observing six pages of the GUI, in order to know exactly their eye’s behaviour. After a free navigation, a questionnaire was conducted to obtain user’s feedback about functional and aesthetic 48 aspects of the GUI. The GUI was displayed on a 1024 x 768 resolution screen located at about 50 cm from the subject (figure 4.19). Figure 4.19 – Eye communication system installation for the usability tests. The data acquisition was performed using Windows 7 Pro ©2009 at a 64 bits / Dell Precision T5600 with a processor Intel Xeon CPU E5-2609 at 2.40GHz (2 processors) and a memory RAM of 32.0 GB. The movements of both eyes were recorded with a remote infrared video-based eye tracker (The Eye Tribe Tracker Server, version 0.9.36) via JEDA (Jean Lorenceau © 2014) with a recording frame rate of 60 Hz. Calibration was done using 9 points with a period of fixation of 2 seconds each. The eye traces were analysed offline using Matlab (Matlab R2013b, Version 8.2.0701) and consisted on: - No smoothing - Average data normalization - Manually offset correction in both axis - Data superimposition with corresponding image Data was also analysed using Excel (Microsoft Excel 2010, Version: 14.0.7128.5000 32 bits), using the same procedures. The four participants were young adults (24–32 years old, 2 females and 2 males, 3 French-native and 1 Chinese-native), familiar with laboratory experiments. The distance between the observer and the display influences the observational angle. Shorter distances to the screen allow higher eye movement angles to scan the display’s area. However, the eye tracker used in this study has an operating range of 45 – 75 cm for its best performance. In order to measure the influence of the distance to the screen we considered that the eye is positioned in the middle line as shown in figure 4.20. Analysing the extreme distances of the operating range when observing an icon of 150x150 pixel it can be concluded that the angle’s variation is about 2º. Figure 4.20 – Variation of the observational angle depending on the distance to the screen when observing an icon with 150x150 pixel area. 49 Participants were initially asked to explore an image from the GUI in a freely way (Welcome Page – see figure 4.3). The recording was made during 2500 frames (about 42 seconds). Afterwards, participants were asked to observe another five pages from the GUI following certain indications. In the Guided Communication page (see figure 4.4) participants were asked to successively observe each icon for 3 seconds while in the Hunger page (see figure 4.5), participants were asked to focus only one of the icons. In order to simulate real situations, in the Pain page (see figure 4.7) participants were suggested to locate: a right leg cramp, a migraine headache and a stomach ache / colic. Then they were asked to quantify a pain using the Pain level page (see figure 4.8) and following the indications: mild pain, great pain and unbearable pain. In the last part, participants were asked to write a word using the keyboard (see figure 4.16). The suggested words were namely “HELLO”, “MAISON”, “AVION” and “HOUSE”. The results will be analysed and discussed afterwards. The final questionnaire consisted on 12 questions with YES-NO answer and an open question for comments and opinions (questions are shown in Annex 1). The current survey was written in French and translated into English, in order to be available to all participants. 4.6 The Device Assessment at the Hospital According to the European guidelines for medical devices classification, this system is included in Class I since is non-invasive, and does not have physical contact with the patient (European Commission - DG Health and Consumer, 2014). The device will be tested at the department of Pulmonology and Medical Intensive Care Unit, Pr T. Similowski ER10 - University Pierre and Marie Curie, Groupe Hospitalier PitiéSalpêtrière (Paris, France). The study will be promoted by ADOREPS (Organisation de la Recherche en Pneumologie et sur le Sommeil) and it will be initiated after a favourable opinion of the CPP (Commité de Protection des Personnes). The main objective of the medical team study is to evaluate the impact of the new tool on the quality of non-verbal communication in ICU intubated patients. The specific objectives of this study are the following: - to evaluate the benefit of this tool in the detection of discomfort sources at ICU. The clinician will evaluate the origin and the cause of the discomfort experienced by the patients. Additionally, the physician will perform an assessment of the patient's nervousness and anxiety level, due to communication difficulties. - to evaluate the benefit of detecting the intensity of the discomfort. Once detected the discomfort source, this communication tool will allow quantifying discomfort using a visual scale. - to evaluate the effectiveness of this tool on reducing the sources of discomfort. This study intends to determine whether the presence of the tested communication tool allows acting on earlier discomfort detection and thus meeting the patient wishes more efficiently. 50 As secondary objectives, it will be evaluated the nurse ease in understanding the patient's discomfort and the workload represented by the tool for the medical personnel. The assessment of the medical device will consist on an observational study in patients hospitalized for acute respiratory failure requiring mechanical ventilation by endotracheal tube. This study compares two strategies: non-verbal communication of intubated patient at ICU, using the eye tracking, and non-verbal communication, by the use of current techniques of the service, namely communication boards. The medical team will evaluate the nine discomfort items that were considered as priority concerns at ICU (see 2.6 section) namely: pain, anxiety, lack of air, excessive respiratory effort, need to be aspired, desire to be informed about their health state, bad installation and thirst. 4.6.1 Patients Inclusion Criteria Patients will be randomly included in the study if they meet the following criteria: - Adult patients hospitalized in the ICU; - Mechanical ventilation through an endotracheal tube or tracheostomy orotracheal; - Remaining estimated mechanical ventilation > 48 hours of mechanical ventilation provided for ventilator weaning; - Conscious and communicating patients, defined by criteria score ATICE (Adaptation To Intensive Care Environment) (Karampela et al., 2002). This assessment is defined by the ability to answer the following five commands: 1) Open / close eyes, 2) Open mouth and sticking out the tongue, 3) Watch the examiner, 4) Nod, 5) Lift eyebrows after the examiner count until 5. - No confusion, defined by criteria score CAM-ICU (Confusion Assessment Method for the Intensive Care Unit) (Ely et al., 2001). This is a current score for the diagnosis of delirium that consists in a sequence of questions and tasks. The patient needs to be able to follow a sequence of letters and squeeze the examiner’s hand when detecting the letter “A”. If not able to do it, the examiner will evaluate the level of disorganized thinking by asking the following questions: 1) Will a stone float on water? 2) Are there fish in the sea? 3) Does one pound weigh more than two? 4) Can you use a hammer to pound a nail? 51 Patients will not be included in the study if one of the following criteria is present: - Minor patients; - Protected adults, whose mental faculties are impaired thereby making it impossible for them to attend to their own welfare; - Pregnant women; - Patients with difficulties in communicating, namely with impaired visual or auditory visual or auditory acuity; - Patients with insufficient knowledge of the French language; - Patients diagnosed with psychiatric disorders; - Patients diagnosed with cognitive disorders. 4.6.2 Assessment’s method Patients that were selected from the group that met the inclusion criteria will be exposed successively to two strategies in a random order namely: - 24 hours with the eye tracker communication tool; - 24 hours without the tool, using the non-verbal communication methods available in the service. The patient will have access to the communication tool 3 times per day. This process will be monitored by a nurse that will evaluate the communication process with the patient according to the criteria described below. - Time spent to communicate. - The overall content of the communication range. - Method of communication used: eye tracking medical device, head shaking, lip reading, communication board. - Graduated scale of the additional workload. - Graduated scale of the difficulties in understanding the patient. Similarly, the doctor will evaluate twice a day (during the morning and night visit) the communication with the patient according to the criteria detailed below. - Graduated scale of the patient's nervousness related to communication difficulties. - Graduated scale for the patient's fear related to communication difficulties. - Rating Scale of difficulties' type: o Communication difficulties on communication in general o Communication difficulties on the physical needs o Communication difficulties for care. 52 4.6.3 Regulatory compliance The test consists on a non-interventional study that does not amend the diagnostic practices or the therapeutic management of patients. Indeed, the communication with the patient addressing his discomfort is an essential step in the management of patients during ICU stay. This study is therefore an integral part of clinical practice and is routinely performed in the ICU concerned in this study. Tests will be initiated and conducted by the clinician and the nurse in charge of the patient. Their implementation will be consistent with current practices in the participating services. Moreover, the employed methods of measurement are completely painless and atraumatic. The ventilatory parameters collected will not require any invasive procedure. Measurement of blood gases to allow the specialist in charge to weaning from respiratory support and to better adjust the parameters of mechanical ventilation will be performed routinely. The study involves no risk to the subjects. 53 Chapter 5 Results and Discussion In order to evaluate the developed eye communication system, usability tests were performed in four healthy subjects followed by a survey to assess the user’s experience. This chapter contains the results together with the discussion. This assessment may confirm its functionality and whether the strategies followed to solve the challenges inherent to eye tracking were successful. This is an important preliminary phase before starting the tests on patients and to verify if the developed interface suits the needs both on a functional and aesthetic level. 5.1 Usability Evaluation using Eye Tracking Participants were positioned inside the eye tracker’s trackbox (see section 3.8), between 50 and 57 cm from the display. This variation caused a difference on the observation angle of about 0.5º which was not considered since it did not influence the participant’s performance. Data concerning eye movements was obtained with The Eye Tribe™ and analysed with MatLab or Excel software. Calibration evaluation provided by the eye tracker was higher than 3/5 for all the subjects. The results are disposed in a xy graphic which corresponds to the 1024 x 768 resolution screen. The origin (0,0) is located at the top-left corner of the display. The eye position in each frame is represented by a point with different colours in contrast to the background. Data was superimposed on the original images of the GUI pages. 60 5.1.6 Eye Writing with the Keyboard Using the keyboard, participants were asked to write “HOUSE”, “AVION”, “MAISON” and “HELLO”. The purpose of this test was to inspect the efficiency of the keyboard. The results are provided in figure 5.7. Figure 5.7 – Subjects observation of the keys from the keyboard page of the GUI for eye communication. Participants were asked to write “HOUSE”, “AVION”, “MAISON” and “HELLO” respectively. User’s eye movements were recorded with The Eye Tribe™ in 60 Hz mode, during 2500 frames. Data of the eye’s coordinates are represented by blue circles. A), B), C) and D) represent subjects AL, DA, JM and LC respectively. As it can be seen, all subjects achieved to write the demanded word. Despite the small size of each key and the reduced space between them, participants were successful in their task. Between fixations, participants gazed upon other keys but did not remain there time enough to select them involuntarily. All the words but one were composed by 5 letters. Writing speed was not analysed, since that was not relevant for the usability evaluation. However, previous studies shown that, in eye writing using QWERTY keyboards, the writing rate is about 15 wps. A) B) C) D) 61 5.2 Navigation Survey Analysing the questionnaires conducted after free navigation on the menus, it can be concluded that the overall user’s experience was very positive. The current survey was written in French and translated into English, which permitted a response rate of 100%. As for the functional aspect, all participants affirm that it is easy to navigate through menus using the eyes and that the GUI’s structure is not confusing, i.e., there are not too many sub-menus. None of the users needed to adjust the dwell time and, therefore, all of them navigated using the default time of 1000 ms. All participants claim no difficulties in maintaining gaze to select options. None felt the system to be too sensitive, namely by accidently selecting unintended options. These last two facts confirm the proper setting of the default time. As for the red gaze-locked point, 3 in 4 participants confirm that it is useful while 1 affirms to be a distraction source. The same user has detected that, during navigation, the point no longer coincided exactly with the position on the screen to where he was looking at. In this case, the point helped him to detect a need for recalibration. With the remaining 3 users this situation did not occur, discarding the need for a new calibration. This situation confirms our previous assumption that the gaze-locked point is helpful on the detection of recalibration’s need. All users confirmed that the feedback – jitter – is helpful to know the probable selection and to navigate in the GUI. Regarding the graphical aspect, users claim that the developed GUI has an attractive and appealing design. All respondents confirm that the texts are legible and that the combination of the text with the image has a clear understanding. The Chinese participant had just classified the size of the text and the contribution of the image once he is not French speaking and therefore did not understand the menus’ information content. Among the 4 participants, 3 consider that the colour associated to each icon helps in communication. On the other hand, half of the users think that the images are infant-like particularly on the Anxiety page. The comments and suggestions left in the open part showed that the ruler to express pain level is poorly scaled. Generally, respondents consider the system to be playful and easy to operate. 62 5.3 Concluding Remarks All participants were familiar with experiences of this nature which may cause a bias in the results. Their activities can have an influence on the ocular dynamics, namely on the ease to fixate an icon to select it or in the ocular behaviour. We can exclude the influence on the fixation time for selection once the dwell time may be adjustable. Besides, the default value has been set based on the values presented in literature and therefore adequate to normal subjects. Other bias that may be due to the participant’s activities were difficult to discriminate. The range of distances between the participants (50 – 57 cm) was irrelevant on the variation of the observational angle. Recalling the issues in which we focused more efforts (see section 4.4) we can conclude that, for the 4 participants: - the noise coming from the eye’s movement was prior filtered by The Eye Tribe™ ; - the buttons are large allowing an easy and effective selection; - the buttons are sufficiently spaced avoiding confusion when selecting an option; - default dwell time is adequate to common use; - empty space in the middle of each page avoids premature selections; - the jitter is a helpful feedback; - the gaze-locked point is not considered in most of the cases as a distraction source; - the gaze-locked point is helpful on detecting a calibration bias; - recalibration is not required in most of normal uses; - the design is appealing and user friendly. On the other hand some weak points were detected that could be improved in a newer version, namely: - pain ruler poorly scaled; - few number of body regions to indicate pain symptoms; - lack of right / left indication on the human body; - icons of Anxiety page considered as infant-like. Other features that could enhance the present communication system will be discussed in detail in the next chapter. 63 Chapter 6 Conclusions and Future Work In this work, we developed a medical device for communication, based on an eye tracker and an interactive graphical user interface, targeting nonspeaking patients at ICU. The objectives that we proposed to accomplish were successfully achieved. This technology aims people at debility states and thus reliability, robustness, safety, and mounting issues were carefully taken into account. It does not require any kind of prior learning and thus can be applied in short-period stays or unpredictable hospitalizations at ICU. It is also practical and easy to implement, not representing an extra charge on paramedics’ working routines. The final device is cost-effective, which makes it affordable for the Health National System and tempting for private clinics. The communication is established via figurative icons represented by symbolic images and texts so that the patients can inform the medical staff about their symptoms, location and intensity of pain, feelings and needs. This will ultimately access their health condition and improve their daily life at the hospital. Usability tests conducted on healthy subjects proved that the system presents an intuitive and easy operating mode, enabling people of different ages to communicate clearly and understandably. If it had been available a more extensive working period, other parameters could have been tested, namely, the screen distance, the screen size, the default dwell time, the presence / absence of the gaze-locked point, the subjects’ conditions and groups etc. The survey conducted on users after GUI interaction showed that its design is appealing and allows an easy communication. Since this system is in process of validation to be tested at the hospital, there is no information on practical cases to the date, nor concrete data on the patient’s response. However, the review of the impact of providing AAC technologies to patients at ICU proved that these approaches are valuable and essential tools on improving life conditions at the hospital. Furthermore, the doctors’ feedback was favourable since this device accomplishes 64 the essential health and safety requirements, meets the needs of the patients and fills the gap of assistive technology for communication at ICU. In the proposed medical device, some features could be enhanced leaving the door open for future interventions. At first sight, the device could incorporate a speech synthesizer in order to reproduce loudly the patient’s message. On a further stage, the device could be integrated on a networked infrastructure allowing, for example, the TV control using the respective menu. These enhancements would provide greater autonomy to the patient and leave the caregiver available to do more important tasks. This device was created targeting patients from the pulmonology department. The GUI is directed to people who suffer interventions to the respiratory system and so, some of the options concern, specifically and exclusively, this sector. However, other health interventions involving intubation and mechanical ventilation, namely due to anaesthesia, would also benefit from this system. Therefore, new menus directed towards other hospital departments could be created. For that, it would be necessary to conduct a detailed study of each one of them, in order to obtain data related to the specific patients’ treatments, problems and needs. Alternatively, it could be developed an easy GUI allowing doctors and paramedical staff to modify the menu’s structure according to the current needs and requirements of their department and patients. This upgrade would give the opportunity to adapt the system to each hospital department or even to each situation / patient. To date, if it is necessary to make any alterations or create new layouts, it is required someone with specific programming skills. Once it is not desirable to give further formation to the medical staff, an intuitive way to change and update the program would be an add-value to the present solution. On the other hand, the system was thought to be standard and directed to patients at ICU of one certain department. Standardization is important in order to discard the need for continuous adaptation. In a broader application, the system could also be adapted to fit other cases of severe permanent motor impairment. The GUI was also adapted to patients with ALS. This test is under process with awaited results. The present system does not constitute an innovation in the field of AAC technologies targeting ALS patients, since many studies have been conducted and different devices have been developed for this population. However, this is, nonetheless, an available option that presents many advantages, namely its low-cost, ease to acquire and versatility of adaption to each person. In a wider and higher perspective, this system could be used to have an advanced functionality of diagnosis. For the time being, the eye movements’ data are exclusively used for functional purposes i.e., to select the buttons to where the patient is looking at. Additionally, the data of each patient could be saved and used for further purposes. 65 In an immediate analysis, one could evaluate how frequently the patient selects, for example, the icon “Anxiety”. This would describe a profile across different patients and even, for the same patient, across different days and health states. The same analysis could be done for the selection of “Pain”, to obtain an overview of the more frequently complains after a certain kind of intervention. Tracing such profiles could provide valuable information to enhance the postoperative care. In the same lines, the statistical study of the announcement of the everyday needs, such as room temperature or hungry/thirsty, could give important cues about the general hospital life conditions, but also help improving the way in which the institution’s physical and human resources are managed. In a more ambitious outlook, data of the eye movements could be taken effortlessly to a higher level of in-depth analysis. The extensive data of patients could be applied on physiologic and psychophysics finalities. Pupil size is known to be affected by physical activation, strong emotional experiences and cognitive effort. Several studies have supported that it is possible to infer the physical and emotional state of one person from his eye features (see section 3.4). Henceforth, the eye’s dynamics and pupil size extracted from the eye tracker may be fundamental in the assessment of the patient’s health and emotion conditions. This analysis and consequent secondary function, either at a superficial or at a deeper level, would gather large and consistent data that could be useful for visual and oculomotor studies. This would be an innovative and ambitious project that would contribute to a better understanding of both individual user behaviour and statistical results from several users. However, this functionality and future application need further studies to confirm and validate a direct assessment of health/emotional status by analysing the gaze characteristics. To conclude, by providing a communication way to nonspeaking patients at ICU, I expect to improve the interpretation of intentional messages and to reduce the sources of discomfort and stress that will ultimately improve patient’s life during the ICU stay and its aftermath. I hope that my contribution will improve the quality of life and promote the recognition of the patient as a person. 66 67 References Alasad, J., & Ahmad, M. Muayyad. (2005). Communication with the critically ill patients: The nurses’ experience. Journal of Advanced Nursing, 50 (4), 356-362. Al-Rahayfeh, A. & Faezipour, M. (2013). Eye Tracking and Head Movement Detection: A State-of-Art Survey. Translational Engineering in Health and Medicine, IEEE Journal of 2013 (1):11-22. American Speech-Language-Hearing Association. (2005). Roles and Responsibilities of Speech-Language Pathologists With Respect to Augmentative and Alternative Communication: Position Statement. [Online] Available from: http://goo.gl/KjR15h [Accessed: 17 July 2014] APHP. (2014). Les grands chiffres clés. [Online] Available from: http://goo.gl/8axD3n [Accessed: 24 July 2014] Bahill, A. T., Stark L. (1975). Overlapping saccades and glissades are produced by fatigue in the saccadic eye movement system. Exp Neurol., 48:95–106. Barreto, A. B., Scargle, S. D. & Adjouadi, M. (2000). A practical EMG-based humancomputer interface for users with motor disabilities. Journal of Rehabilitation Research and Development, 37(1), 53 - 63 Bartlett, G., Blais, R., Tamblyn, R., Clermont, R. J., MacGibbon B. (2008) Impact of patient communication problems on the risk of preventable adverse events in acute care settings. CMAJ Can Med Assoc J, 178(12):1555-1562. Bates, R. E. A. (2006). Enhancing the Performance of Eye and Head Mice: A Validated Assessment Method and an Investigation into the Performance of Eye and Head Based Assistive Technology Pointing Devices. PhD thesis, De Montfort University. Bauby, J.-D. (1997). Le Scaphandre et le Papillon. Paris, Robert Laffont, 1997, 139 p. Bergbom-Engberg, I. & Haljamäe, H. (1989). Assessment of patients´ experience of discomfort during respirator treatment. Critical Care Medicine, 17:1068-1072. Blackstone, S. (2007). AAC in the ICU. Augmentative Communication News, 19, 1-3. 68 Bolt, R. A. (1981). Gaze-orchestrated Dynamic Windows. In Proceedings of the 8th Annual Conference on Computer Graphics and interactive Techniques, SIGGRAPH '81. ACM Press 109 – 119. Boost. (2014). C++ Libraries. [Online] Available from: http://www.boost.org/ [Accessed: 1 March 2014] Centre for Communication Rights – CCR. (2011). [Online] Available from: http://goo.gl/EHxABj [Accessed: 10 January 2014] Coetzer, R. C. & Hancke, G. P. (2011). Eye detection for a real-time vehicle driver fatigue monitoring system, in Proc. IEEE Intell. Veh. Symp: 66-71. COGAIN. (2014). Communication by Gaze Interaction. [Online] Available from: http://www.cogain.org/ [Accessed: 1 February 2014] Cometa, M. (2011). Student learns to control computer with a blink of an eye, Rochester Institute of Technology. Cornsweet, T. N. & Crane H. D. (1973). Accurate two-dimensional eye tracker using first and fourth Purkinje images. J Opt Soc Am, 63(8):921-928. Costello, J. (2000). Augmentative Communication in the Intensive Care Unit: The Children’s Hospital Boston Model, Augmentative and Alternative Communication. 16(3) 137– 153. Custers, E. (2014). Anatomy Notes. Anatomy of the eye. [Online] Available from: http://goo.gl/CV8fTj [Accessed: 1 February 2014] Cuthbertson B. H., Hull A., Strachan M. & Scott J. (2004). Post-traumatic stress disorder after critical illness requiring general intensive care. Intensive Care Med, 30(3):450–455. Chanques, G., Jaber, S., Barbotte, E., Violet S., Sebbane, M. et al. (2006). Impact of systematic evaluation of pain and agitation in an intensive care unit. Critical Care Medicine, 34, 1691-1699. Chapman, J.E. (1991). The use of eye-operated computer system in locked-in syndrome. Proceedings of the Sixth Annual International Conference on Technology and Persons with Disabilities (CSUN’91), Los Angeles, CA. Charlier, J., Buquet, C., Dubus, F., Hugeux, J.P. & Degroc, B. (1997). VISIOBOARD: A new gaze command system for handicapped subjects. Medical and Biological Engineering and Computing 35(416). Dalton, A. B. & Ellis, C. S. (2003). Sensing User Intention and Context for Energy Management. In Proceedings of 9th Workshop on Hot Topics in Operatings Systems (HotOS IX): USENIX Press. Daugman, J. (2009). Chapter 25 – How Iris Recognition Works. The Essential Guide to Image Processing (Second Edition): 715-739 Demasco, P. W. & McCoy, K. F. (1992). Generating text from compressed input: An intelligent interface for people with severe motor impairments. Communications of the ACM 35(5), 68-78. 69 Deubel, H., Hyona, J. & Radach, R. (2003). The mind’s eye: Cognitive and applied aspects of eye movement research. Elsevier: 493-516. Diard, J., Rynik, V., & Lorenceau, J. (2013). A Bayesian computational model for online character recognition and disability assessment during cursive eye writing. Frontiers in psychology, 4:843. Dickie, C., Vertegaal, R., Sohn, C., & Cheng, D. (2005). EyeLook: Using Attention to Facilitate Mobile Media Consumption. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology. UIST '05. ACM Press, 103 – 106. Doble, J. E., Haig, A. J., Ander son, C. & Katz, R. (2003). Impairment, activity, participation, life satisfaction and survival in persons with locked-in syndrome for over a decade. Journal of Head Trauma Rehabilitation, 18: 435-444. Donegan, M., Oosthuizen, L., Bates, R., Daunys, G., Hansen, J.P., Joos, M., et al. (2005) D3.1 User requirements report with observations of difficulties users are experiencing. Communication by Gaze Interaction (COGAIN). Deliverable 3.1. Dongheng, Li, Winfield, D. & Parkhurst, D. J. (2005). Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3:79. Dongheng, Li. (2006). Low-cost Eye-tracking for Human Computer Interaction. Ames, Iowa State University. Dowden, P. A. & Marriner N. A. (1995). Augmentative and alternative communication: treatment principles and strategies. Semin Speech Lang, 6(2):140-57; quiz 157-8. Drewes, H. (2010). Eye Gaze Tracking for Human Computer Interaction. Dissertation, LMU München: Faculty of Mathematics, Computer Science and Statistics. Duchowski, A. & Vertegaal, R. (2000). Eye-based interaction in graphical systems: theory and practice. In Siggraph Course Notes. Duchowski, A. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods Instruments and Computers, 34(4):455-470. Duchowski, A. (2003). Eye tracking methodology: Theory and practice. London: SpringerVerlag Ltd. DynaVox. (2014). DynaVox T-Series. [Online] Available from: http://goo.gl/lN2zCH [Accessed: 21 July 2014] Ekman, I., Poikola, A., Mäkäräinen, M., Takala, T. & Hämäläinen, P. (2008). Voluntary pupil size change as control in eyes only interaction. 115-118. Elbert, T., Lutzenberger, W., Rockstroh, B. & Birbaumer, N. (1985). Removal of ocular artifacts from the EEG. A biophysical approach to the EOG. Electroencephalogram Clinical Neurophysiology, 60: 455-463. Ely, E. W., Gautam, S., Margolin, R., Francis, J., May, L., Speroff, T. et al. (2001).The impact of delirium in the Intensive care unit on hospital length of stay. Intensive Care Med; 27:1892-1900. 76 77 Annex 1 Survey after navigation on graphical user interface 1. Is it easy to navigate through the menus? _______ 2. Is the organization structure of menu confused? Too many sub-menus? _______ 3. Is it difficult to maintain the gaze long enough to select? _______ 4. Are the options being selected without wanting? System too "sensitive"? _______ 5. Is the red dot following your eyes a distracting source? _______ 6. Does the red dot stop matching with the screen region under gaze? _______ 7. Is the feedback (jitter) useful? _______ 8. Are the texts readable / big enough? _______ 9. The combination of the text with the image allows clear understanding? _______ 10. Is the colour associated with each icon helpful for the communication? _______ 11. Is the overall design appellative? _______ 12. Are the images infant-like? _______ 13. Opinions / Comments: 78 79 Annex 2 Version of the graphical user interface addressed to ALS patients In the Guided Communication page, the Breathing (Respiration) option was replaced by Entertainment (Divertissements) as shown in figure 6.1. Once ALS patients do not have necessarily respiratory problems, the specialists suggested to delete that button and add a playful option. Figure 6.1 – GUI for eye communication adapted to ALS patients: Guided communication page. In this new menu, the patient can chose between take a walk (Promenade), watch a movie (Film), play a game (Jouer) or listen to music (Musique). Figure 6.2 – GUI for eye communication adapted to ALS patients: Entertainment page. 80 Inside the pre-existent Bathroom’s and Health’s menus some adaptations were also taken into account. Once ALS is a chronic disease, its medication is usually a single fixed dosage. Therefore, the icon to obtain information about medication was considered as irrelevant by the ALS specialists. However, these patients frequently depend on artificial ventilation. Thus, inside the menu Health, it was added the option “Need of Non-Invasive Ventilation” (Besoin de VNI (Ventilation Non-Invasive)) as shown in figure 6.3. Figure 6.3 – GUI for eye communication adapted to ALS patients: Health page. In the same way, the specialists that accompanied these patients suggested the alteration of the menu Bathroom. Therefore, the previous options were both replaced by “going to the bathroom” (Aller au toilettes) or “take a shower” (Être lavé). Figure 6.4 – GUI for eye communication adapted to ALS patients: WC page.