Eye-tracking introduction considerations in vestibular telerehabilitation in Latvia

Periodicals of Engineering and Natural Sciences ISSN 2303-4521 Vol. 7, No. 1, June 2019, pp.177-186 Available online at: http://pen.ius.edu.ba 177 Eye-tracking introduction considerations in vestibular telerehabilitation in Latvia Aleksandrs Gorbunovs1, Zanis Timsans2, Ieva Grada3 1, 2, 3 Distance Education Study Centre, Riga Technical University Article Info ABSTRACT Received Dec 3, 2018 The terms of telemedicine, telerehabilitation, and e-care have come into our everyday lives as the necessity to make medical assistance and care more effective. These terms are not buzzwords. Indeed, new technology, innovative solutions, offered by programming tools, modern sensor equipment, wearable devices and bio-signal data sets, provide an ability to ensure necessary remote support to a person with special needs anytime and anywhere. This assistance may include person’s online health monitoring, telerehabilitation and health improvement measures, mobile training, rapid alert button initiation, advisory support, and a set of many other measures supporting patient’s care. This paper displays achievements in creation of vestibular telerehabilitation tools and systems, particularly in Latvia, and discuss further necessary developments, including eye-tracking instruments, which would improve existing telerehabilitation systems. Keyword: Eye-tracking Facial recognition Gaze data Postural balance Telerehabilitation Corresponding Author: Aleksandrs Gorbunovs, Distance Education Study Centre, Riga Technical University, 1 Kronvalda Boulevard, Riga, LV-1010, Latvia. Email: aleksandrs.gorbuno[email protected] 1. Introduction Ever new technologies are entering in today's rapidly changing world. Nowadays, people are not only talking about e-health, e-medicine and e-care, but also are actively involved in the opportunities and activities offered by these new systems and platforms. Patients may feel satisfied when receiving medical advice almost immediately, and practitioners are given an opportunity to provide assistance as soon as an appropriate online request is received. New tele-care approaches have brought new requirements for information systems, display graphs, figures, notifications and so on. For example, the Council of the European Union in its Directive defines certain requirements on the accessibility of the websites and mobile applications of public sector bodies [1]. Moreover, telemedicine or telerehabilitation system developers ought to take into account prospective system users interests and needs, as well ensure system reliability and sustainability [2]. Such e-, m-, and/or trehabilitation platforms ought to be aimed to improve life quality for people with special needs, facilitate rehabilitation process anytime anywhere and provide necessary online support to the system users. Considering that “the balance is one of the most important skills of the postural control and ability to move” [3], it might be said that persons’ active involvement in the vestibular telerehabilitation activities can improve life quality of these people. Herdman points out three main human body systems which are responsible for postural balance control. They are: visual, vestibular and somatosensory ones [4]. Previous studies underline four vestibular rehabilitation tasks: PEN Vol. 7, No. 1, June 2019, pp.177186 178 - “To enhance gaze stability, - To enhance postural stability, - To improve vertigo, and - To improve activities of daily living” [5]. Vestibular telerehabilitation challenges might be facilitated by proposing an efficient telerehabilitation system which includes postural balance function measurement and improvement system model, balance measurement board with an interactive interface, vestibular exercises, available both from telerehabilitation platform and mobile application, and continuous health monitoring and advisory support. Eye-tracking tools might be added to this system, providing necessary analysis of the gaze data to be used to improve patients’ telerehabilitation efficiency. 2. Postural balance early diagnostics systems and tools development in Latvia In order to promote early diagnosis of postural balance and improvement of these functional capabilities, from November 2014 to June 2018 the postural balance function measurement and improvement system model has been developed by researchers of Riga Technical University (RTU) within the framework of the Project No.2 „Development, approbation and implementation of new prototypes and innovative methodology (approaches, methods, techniques, ways) in social rehabilitation for the rendering of new services” (in Latvian: „Jaunu prototipu, inovatīvas metodikas (pieeju, metožu, tehniku, paņēmienu) sociālajā rehabilitācijā izstrāde, aprobācija un ieviešana jaunu pakalpojumu sniegšanai”) of the Latvian National Research Program “Innovative solutions in social rehabilitation in Latvian schools in the context of inclusive education” (in Latvian: „Inovatīvi risinājumi sociālajā telerehabilitācijā Latvijas skolās iekļaujošās izglītības kontekstā – VPP INOSOCTEREHI”; Contract No. 10-4/VPP-8-7) [6]. The model includes an assessment of the postural balance of a person (the user of this system) and a set of activities for improving these capabilities (Fig.1). Postural balance assessment is performed (the 1st step, Fig.1) by the person completing a 4-mode test mCTSIB (modified Clinical Test of Sensory Organization and Balance) on the balance testing board platform (consisting of mounted in four load sensors (tenzoresistors), Arduino Uno microcontroller and two signal amplifiers). Test performance data are read and processed, further allowing to obtain information about the identified postural balance ability (or disability) of the person, and represent results in a visual form, as well as in the form of csv files. In accordance with the obtained measurements results, the person chooses appropriate exercises offered by the telerehabilitation e-platform to improve his/her balance ability (the 2nd step, Fig.1). An access to the exercise database with the offered audio-visual, interactive materials, is enabled via e-environment (e.g., computer, laptop, etc.) and/or m-environment (e.g., smartphone, tablet, etc.). Load sensors HX711 amplifiers Half-bridge circuit Arduino Uno microcontroller Balance board System interface Dignostic test report Mobile apps Computer Database / Ecercise file repository Telerehabilitation e-platform m-CTSIB test database USB Macro CSV files 1st step: Diagnostics of balance capability 5th step: Assessment (repeated diagnostics) 3rd step: Exercise practice 2nd step: Choosing appropriate exercise 4th step: Systematic exercise practice Figure 1. Postural balance function measurement and improvement system model. PEN Vol. 7, No. 1, June 2019, pp.177186 179 Audio-visual materials are arranged in a certain order allowing the person to choose from content sets of video files the most suitable appropriate exercises, practising (the 3rd step, Fig.1) and systematically re-practising them (the 4th step, Fig.1). These steps result in person’s postural balance improvement. The model also recommends to perform repeated assessment of postural balance ability (repeated diagnostics; the 5th step, Fig.1), which allows to measure vestibular rehabilitation progress. To test postural balance, the person is asked to stand on the force plate and perform specific balance keeping tasks according to m-CTSIB test modes which are already approbated in many clinical studies demonstrating its simplicity, accessibility, usability and efficiency in balance disorder measurements [3], [7], [8]. In other words, the person tries to keep postural balance which is related to keeping sphere symbol on the screen during the test. An apprehensible, child-friendly interface, as well as the data-processing program has been developed to visualize and interpret the balance measurement resulting data, providing an opportunity to make an initial diagnostics of possible balance disorders. Encouraging, kids-friendly interface helps the person to follow requirements for implementation of trials (Fig. 2): interactive monkey image animation gives an advice how the person should prepare himself/herself for the next test condition (m-CTSIB trial fulfilment) – does the person need to close the eyes (the monkey image hides its eyes behind lifted up palms) or the trial must be fulfilled with open eyes (the monkey opens its eyes and looks on you). Figure 2. Postural balance test fulfilment interface. Postural balance test fulfilment interface (Fig. 2) gives both for the patient and medical personnel a possibility to follow all trials paces. Moving sphere’s trajectories are tracked by remaining colour traces in the circle. Exercise data are processed and visualized, demonstrating possible balance distortion (deviations from the average sway index). That allows to use m-CTSIB test results directly at the test venue and make initial balance capability assessments. Test results (Fig. 3) are represented in the four graphs (stabilograms) corresponding to the each of four balance test modes [9]. Each of the executed test modes can show potential postural balance problems according to the test mode implementation requirements which are set for one organ or organ groups responsible for human balance capability. Consequently, the test provides an opportunity to assess not only balance problems in general, but also indicate the potential dysfunction of the particular human organ or group of organs. Besides, created system provide to the medical staff detailed test results for each completed test phase (mode). Figure 4 gives an example of the m-CTSIB 4th test phase (eyes open, foam surface) data visualization. It assesses postural balance: person’s ability to maintain the balance in the anteroposterior (AP) direction (in the upper part of Fig. 4, the AP sway index is marked in blue) and in mediolateral (ML) direction (in the upper part of Fig. 4, the ML sway index is marked in red). The bottom part of Fig. 4 provides the information on the test fulfilment progress (lower left part of the Fig. 4) and the average sway indexes, including the AP and ML ones. PEN Vol. 7, No. 1, June 2019, pp.177186 180 Figure 3. m-CTSIB trials’ data visualization [9]. Figure 4. Anteroposterior (AP) and mediolateral (ML) sway index visualization. The prototype is already approbated during several pilots in schools in Latvia. The prototype is aimed to measure persons’ postural balance functional capability and may serve as an early diagnostics instrument. However, questions regarding future improvements of the interface and the developed model remain. It can include also embedded eye-tracking tools with gaze analytical system, evaluation of patient’s activities (eye movements) in a form of gaze data which might be caused due to external interferences. Defining specific tasks during postural balance training exercises can contribute to vestibular rehabilitation. In turn, the data provided by the eye-tracking device would make it possible to clarify and update rehabilitation program. 3. Developed in Latvia mobile application supporting social telerehabilitation The mobile application (utilizing Android operation system) is developed by RTU Distance Education Study Centre researchers to provide anytime-anywhere postural balance training support accordingly promoting postural balance function improvements and enabling vestibular and social telerehabilitation [6]. It is an integral part of the balance function measurement and improvement system model. The application is available at: https://drive.google.com/file/d/1u_-5ptb6LPWCNe1X6qhuZaHxJjdGg6Vn/view. PEN Vol. 7, No. 1, June 2019, pp.177186 181 Mobile application, installed in the mobile device, further provides an access to the balance improvement exercises video content without Internet connection. Exercises (video content) in the mobile application are sequentially positioned: - In a stationary and transparent environment: with and without variations; - In a changing and non-transparent environment, with variations. Users can choose and implement the necessary balance exercises from the offered training sets taking into account the developed methodology in according to their abilities and needs. Exercises are placed in three levels (Fig. 5): in the cases of stationary and transparent environment with and without variations when the body is not moving (with and without manipulation) and is moving (with and without manipulation). Figure 5. Three levels of training exercises. The application provides user-friendly, easy navigation to search for requested exercise video materials. For example, navigation buttons (Fig. 6, arrow "A") provides the ability to move between the proposed exercises to the right, left, up and down, according to the chosen training methodology in three levels. In turn, interactive notifications (Fig. 6, arrow "B"), as well as scoring points/ badges in the form of gesture of appreciation (Fig. 6, arrow "C") as the award/recognition of user’s achievements, encourage and motivate system users to continue an improvement of their postural balance capabilities. These feedback statements are particularly important for the children involvement in the rehabilitation process. Figure 6. Navigation buttons (arrows “A”), feedback notifications/praises (arrow “B”) and achievement recognition points/badges (arrow “C”). PEN Vol. 7, No. 1, June 2019, pp.177186 182 Orientation sensor can be used for self-assessment of postural balance ability (Fig. 7). The application user must place and hold the phone for at least 5 seconds, so that the indicator "N" shows to the North. Then the user can place the phone in a transparent place and continue to watch videos and fulfil training exercises. If the application user fails to keep the phone in the balance position to the North, the compass figures will move. The user must use the compass at a time to improve own ability to hold the object in balance position. Even if it is failed initially from the first time, it is important to go back to exercises, executing them and improving postural balance. Then again and again, the user ought to return to the compass mode in order to check own postural balance and/or its improvements. Figure 7. Orientation sensor. 4. Usage of eye-tracking tools in telemedicine, e-care and telerehabilitation Researchers and medical engineers have already recognized the needs of society for robotic assistants to provide support to people with special needs, thus ensuring some kind of telerehabilitation - an automated system responding to patient activities, while at the same time a physician or healthcare professional can stay at a distance. From time to time, scientific papers show the rationale and practical implementation of such robotic assistants and systems. Nowadays these systems are able not only to assist but also communicate with patients, as well monitor their activities, recognize gestures to provide emergency call or advice, remind the time of taking medication and performing rehabilitation exercises [10]. Face as well as gesture recognition systems can track patient’s emotions which might be represented as a set of this person’s activity feedback signals [11; 12] which are based on some kind of reward processing [13]. They could be detected as the emotional data sets and analysed to provide necessary response. The use of eye-tracking tools and an assessment of gaze data opens new possibilities in telemedicine, e-care and telerehabilitation. That includes the care of elderly people, persons after stroke, with movement disabilities, as well as eye training and many other directions. Some areas, such as diagnosis of mental diseases [14], could be further developed to be used also in tele-mode platforms, at least in tele-monitoring and emergency/assistance areas. Detection of persons with depressive disorder by use of eye tracking methods in lab environment [15] could be transferred to tele-diagnostics performed remotely by monitoring behaviour / eye movements of appropriate e-platform users, with prior authorization to use their personal data. Eye training gives positive results in vision improvement [16; 17]. Kasprowski et al. describe game implementation examples in vision training [18] which add to exercises some sort of fun, interaction between a child and computer interface – a player (child) is asked to assemble the puzzle by focussing his/her eyes on one piece, then triggering it and transferring to correct place by focussing eyes on this place and triggering it again (Fig. 8). Similarly, eye tracking technology with embedded gaming solution is used in children behavioural therapy [19]. PEN Vol. 7, No. 1, June 2019, pp.177186 183 Figure 8. Vision training by eye tracking (example) [18]. Useful systems for patients with movement disorders and paralyzed persons are developed by exploiting similar gaze controlled approach [20]. Developed software gives eye movement / gaze access to web applications, which enable patient web browsing, communication and even article typing by eye gaze means only [21]. Persons who are limited in motor-movement can browse Internet by triggering available links on the Web and typing words with own eyes and voice [22]. Eye-tracking technology and gaze control is also used in European Union Framework Program for Research and Innovation “Horizon 2020” project MAMEM (www.mamem.eu). Eye movements are controlled by especially developed Web browser (GazeTheWeb), facilitating Web browsing by eyes only [22; 23]. Moreover, wireless bio-signal acquisition system is aimed to help persons with motor disabilities interact with gaming application developed by MAMEM project scientists and play the game Tetris by eyes and mind commands [24]. 5. Discussions and further developments Pursuant to considerations described in the second section, in further research activities it would be useful to find relationships and correlations between person’s ability to keep the balance when different external disturbing factors influence test fulfilment. Another research direction ought to be dedicated to the studies which would find out possible correlations between persons’ postural balance capability and mental abilities, achievements in different educational subjects. Besides, it is important to find the ways in how the balance prototype production costs could be decreased. Affordable price can promote postural balance system employment in organisations and at home, and as result, facilitate vestibular and social rehabilitation services and improve quality of life of persons with postural balance disorders. Elements of gaming might be embedded into postural balance test procedures. It could bring attractiveness and engage people in vestibular rehabilitation processes. It could be useful to investigate possible correlation between person’s eye movements, ability to focus attention during tests and exercises, and different external distractions, such as suddenly opened or closed door, alarm beep, and so on. Two of used trial modes in m-CTSIB define postural balance test accomplishment with closed eyes (accordingly, standing on the stable and soft surface). It means that in other two m-CTSIB trial modes, when patient’s eyes are open, it could be useful to introduce eye tracking tools which would help a lot to a physician with analytical gaze data to provide initial diagnostic considerations. Possibilities to develop appropriate easily perceived e-, t-, and m-environment also could be considered. People react differently on various changing content layouts on display. Pop-ups, advertisements, and many other graphics carousels impact person’s attention and eye movements. Another issue is the Big Data. It is estimated that medical data at least doubles every three years [25]. The amount of the data grows exponentially. Artificial intelligence tools allow gathering and processing a huge amount of structured and unstructured real-time data, coming from various clinical records, medical devices (among them, implanted in human body ones), sensors, etc. Big Data and patients’ data protection, as well the steps to be taken against Internet threats to prevent data breaches, ought to be considered as the primary steps in telemedicine, e-care and telerehabilitation services. IBM Watson Health artificial intelligence (AI) system was introduced to help medical professionals providing data analytics to improve health care services and make necessary predictions considering that the AI machine has ability to learn [25]. Taking into account that there were two different types/approaches of data storage PEN Vol. 7, No. 1, June 2019, pp.177186 184 and processing: on the one hand – data servers on medical premises, on the other hand – public data available for public use, IBM decided to shift e-health analytics services to a hybrid cloud which, due to links between these data storages make it possible to move the data between them in and out. This way offers greater and more effective access to medical data both for medical practitioners and insurers [26]. At the end of November 2018 Amazon launched its Web Services Platform (WSP) which processes and analyses patient’s medical e-records and tons of other medical unstructured data [27]. Such systems, like IBM Watson Health or Amazon WSP, analyse massive amount of unstructured patients’ and their treatment data which transform patients care into personalised treatment approach, available and accessible almost immediately [28]. This opens up the possibility for further employment of eye tracking systems in patients’ care, health monitoring and enabling immediate feedback and/or emergency response remotely. 6. Conclusions Proposed postural balance function measurement and improvement system model is ready for implementation in Latvian schools via National Centre for Education of the Republic of Latvia: every year, starting from the first beginning school day, in addition to the measurement of pupils' physical parameters, it would be useful to assess the initial state of pupils’ postural balance capacity using postural balance testing system. When detecting possible abnormalities, the timely advice would be given to pupils’ parents. The model also enables the support to its users through telerehabilitation e-platform which provides a methodology for daily improvement of balance capabilities, offers balance-improvement training video materials. In addition, the model also includes the mobile application developed for supporting postural balance improvement anytime anywhere by offering appropriate exercise sets. Implementation of the eye-tracking approach in vestibular telerehabilitation could be seen as the part of other complex rehabilitation measures. Gaming component can make postural balance testing and training systems more attractive and effective, increasing patients’ interest in postural balance improvement activities, thereby ensuring goals of vestibular telerehabilitation. Eye tracking tools ought to be introduced in this process to assess person’s eye motor and attention abilities, as well to be aimed to detect possible health disorders (related to neuro-, eye-, vestibular system, etc.). Artificial intelligence technologies bring the new wide opportunities in telerehabilitation and e-care field. However, the Big Data issue and patients’ data protection challenge still remains actual both from data processing and cybersecurity point of the view. Acknowledgements This work has been supported by the European Regional Development Fund within the Activity 1.1.1.2 “Postdoctoral Research Aid” of the Specific Aid Objective 1.1.1 “To increase the research and innovative capacity of scientific institutions of Latvia and the ability to attract external financing, investing in human resources and infrastructure” of the Operational Programme “Growth and Employment” (No.1.1.1.2/VIAA/1/16/042). Authors would like to thank also Dr. paed. Aivars Kaupuzs and physician Lorita Rizakova for their great support in testing of developed postural balance board in secondary schools in Latvia, and Dr. paed. Helena Vecenane whose invaluable ideas motivated us to improve postural balance function measurement and improvement system model. References [1] Council of the European Union, “Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies,” Official Journal of the European Union, Brussels [Internet], December 2, 2016 [cited 20.12.2018], available from: http://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=uriserv:OJ.L_.2016.327.01.0001.01.ENG&toc=OJ:L:2016:327:TOC. [2] S. Vukicevic, Z. Stamenkovic, S. Murugesan, et al., “A New Telerehabilitation System Based on Internet of Things,” Facta universitatis - series: Electronics and Energetics, vol. 29, no. 3, pp. 395-405, Sep 2016. PEN Vol. 7, No. 1, June 2019, pp.177186 185 [3] A. Kaupuzs, V. Larins, and L. Rizakova, “Effects of Vestibular Exercises on Postural Balance for Children,” in Proceedings of International Scientific Conference “Society. Integration. Education (SIE2016)”, 2016, vol. 3, pp. 453–463. [4] S. J. Herdman, “Advances in the treatment of vestibular disorders,” Physical Therapy, vol. 77, no. 6, pp. 602–618, 1997. [5] B. I. Han, H. S. Song, and J. S. Kim, “Vestibular Rehabilitation Therapy: Review of Indications, Mechanisms, and Key Exercises,” Journal of Clinical Neurology, vol. 7, no. 4, pp. 184–196, 2011. [6] A. Gorbunovs, “Scientific final report on the Project No.2 „Development, approbation and implementation of new prototypes and innovative methodology (approaches, methods, techniques, ways) in social rehabilitation for the rendering of new services” (in Latvian: „Jaunu prototipu, inovatīvas metodikas (pieeju, metožu, tehniku, paņēmienu) sociālajā rehabilitācijā izstrāde, aprobācija un ieviešana jaunu pakalpojumu sniegšanai”) results of the Latvian National Research Program “Innovative solutions in social rehabilitation in Latvian schools in the context of inclusive education” (in Latvian: „Inovatīvi risinājumi sociālajā telerehabilitācijā Latvijas skolās iekļaujošās izglītības kontekstā – VPP INOSOCTEREHI”), in Scientific final report on the Project No.2 results of the Latvian National Research Program VPP INOSOCTEREHI, pp. 1-50, Sep 2018 (unpublished). [7] M. K. Park, K.-M. Kim, J. Jung, et al., “Evaluation of Uncompensated Unilateral Vestibulopathy Using the Modified Clinical Test for Sensory Interaction and Balance,” Otology & Neurotology, vol. 34, no. 2, pp. 292–296, 2013. [8] N. Murray, A. Salvatore, D. Powell, and R. Reed-Jones, “Reliability and Validity Evidence of Multiple Balance Assessments in Athletes with a Concussion,” Journal of Athletic Training, vol. 49, no. 4, pp. 540–549, 2014. [9] A. Gorbunovs, Z. Timsans, A. Kapenieks, R. Gulbis, “Development of Human Balance Capability Testing Prototype,” in Proceedings of the 11th International Scientific and Practical Conference «Environment. Technology. Resources (ETR-11)», 2017, vol. 3, pp. 62–68, doi: http://dx.doi.org/10.17770/etr2017vol3.2518. [10] I. Kostavelis, D. Giakoumis, S. Malasiotis, and D. Tzovaras, “RAMCIP: Towards a Robotic Assistant to Support Elderly with Mild Cognitive Impairments at Home,” Communications in Computer and Information Science, Springer Nature, vol. 604, pp. 186–195, 2016. [11] J. Broekens, T. Bosse, and S. C. Marsella, “Challenges in Computational Modeling of Affective Processes,” IEEE Transactions on Affective Computing, vol. 4, no. 3, pp. 242–245, 2013. [12] T. M. Moerland, J. Broekens, and C. M. Jonker, “Emotion in Reinforcement Learning Agents and Robots: A Survey,” Mach Learn, vol. 107, pp. 443–480, 2018. [13] E. T. Rolls and F. Grabenhorst, “The Orbitofrontal Cortex and Beyond: From Affect to DecisionMaking,” Progress in Neurobiology, vol. 86, no. 3, pp. 216–244, 2008. [14] S. Wang, M. Jiang, X. M. Duchesne, et al., “Atypical Visual Saliency in Autism Spectrum Disorder Quantified through Model-Based Eye Tracking,” Neuron, vol. 88: pp. 604–616, 2015. [15] A. Duque and C. Vazquez, “Double Attention Bias for Positive and Negative Emotional Faces in Clinical Depression: Evidence from an Eye-Tracking Study,” Journal of behavior therapy and experimental psychiatry, vol. 46, pp. 107-14, 2015, doi: 10.1016/j.jbtep.2014.09.005. [16] B. Takeshita, “Developing Your Children’s Vision. A Guide for Parents of Infants and Young Children with Vision Impairment,” in The Center for the Partially Sighted, Los Angeles, 2012. [17] K. Harezlak and P. Kasprowski, “Application of Eye Tracking in Medicine: A Survey, Research Issues and Challenges,” Computerized Medical Imaging and Graphics, Elsevier, vol. 65, pp. 176–190, 2018, doi: 10.1016/j.compmedimag.2017.04.006. [18] P. Kasprowski, M. Dzierzega, K. Kruk, et al., “Application of Eye Tracking to Support Children’s Vision Enhancing Exercises,” in Piętka E., Badura P., Kawa J., Wieclawek W. (eds) Information Technologies in Medicine. ITiB 2016. Advances in Intelligent Systems and Computing, Springer, Cham., vol. 471, pp. 75–84, 2016. [19] A. Al-Shathri, A. Al-Wabil, and Y. Al-Ohali, “Eye-Controlled Games for Behavioral Therapy of Attention Deficit Disorders,” in Stephanidis C. (eds) HCI International 2013 - Posters’ Extended