Decoding human movement intentions and postural reactions through brain signals

Universidade do Minho Escola de Engenharia Departamento de Informática Raimundo Nonato Barros Neto Decoding human movement intentions and postural reactions through brain signals January 2023 Universidade do Minho Escola de Engenharia Departamento de Informática Raimundo Nonato Barros Neto Decoding human movement intentions and postural reactions through brain signals Master dissertation Integrated Master’s in Informatics Engineering Dissertation supervised by Cristina Manuela Peixoto dos Santos Luís Ricardo Monteiro Jacinto Nuno Miguel Ferrete Ribeiro January 2023 COPYRIGHT AND TERMS OF USE FOR THIRD PARTY WORK This dissertation reports on academic work that can be used by third parties as long as the internationally accepted standards and good practices are respected concerning copyright and related rights. This work can thereafter be used under the terms established in the license below. Readers needing authorization conditions not provided for in the indicated licensing should contact the author through the RepositóriUM of the University of Minho. LICENSE GRANTED TO USERS OF THIS WORK: CC BY https://creativecommons.org/licenses/by/4.0/ ACKNOWLEDGEMENTS All the work done over the last year would not have been possible without the assistance, understanding, and support of many people who deserve to be here recognized. First and foremost, I want to express my gratitude to my supervisor, Prof. Cristina Santos for her availability, knowledge and support that were important for the elaboration of this work. To my co-supervisor, Prof. Luís Jacinto, for his availability and guidance, which were important for the preparation of this work, mainly in the area of Electroencphalogram. I’d also like to express my gratitude to my co-supervisor, Nuno Ribeiro, for his encouragement and understanding at difficult times, especially to be always willing to help and support me to achieve my best. I’m also grateful to everyone in the BiRDLab for the help during the run of the experimental protocol and data acquisition, and mainly to Luís Martins, to share his knowledge regarding the BiRDLab python framework. I want to express my gratitude to my family, even living far away, they always are present sharing all their love and assistance, not only during my dissertation but throughout my entire life. For all my friends from Brazil, who lives in Portugal and Portuguese whom I had the pleasure to meet throughout this journey, my sincerely grateful for all your support and motivational words. Finally, but certainly not least, my wife, Thaís, especially deserves a heartfelt thank you for her support and patience throughout the past year. a STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the University of Minho. b ABSTRACT Falls are one of the most common causes of injuries in the elderly population. As a result, treatment costs have also increased. Recent efforts to restore lower limb function in these populations have seen an increase in the use of wearable robotic systems, however, fall prevention measures in these systems require early detection of loss of balance to be effective. In short, the development of technologies, such as a brain-computer interface, that is capable of recognizing situations at risk of falling based on the loss of balance caused by several factors, is essential. Previous studies have investigated whether kinematic variables contain information about an impending fall, but few have examined the potential of using electroencephalography (EEG) as a predictor of falling and how the brain responds to prevent a fall. Perceived disturbances of balance are always accompanied by a specific cortical activation, called disturbance-evoked potential (PEP). In this study, the recognition of daily activities (walking, lifting, crouching, going up and down stairs) was also part of the initial objective, however, due to the challenges encountered, the object of study of the present work was focused on the recognition and binary classification of the presence of loss of balance (PEPs) in brain signals. Thus, this dissertation intends to take the first steps toward the decoding of brain activity in response to imbalanced events. Initially, to acquire the data, an experimental protocol was designed, so that the participants, using EEG, were submitted to gliding-like perturbations while walking on the treadmill. Two healthy subjects were exposed to a glide-like perturbation, and these perturbations occurred interspersed over a period lasting 30 to 60 seconds. Each subject performed 2 experiments, that is, perturbations provoked while the individual walked on the treadmill: i) at a speed of 1.6 km/h and ii) at a speed of 2.5 km/h. Based on the approached methods, the perturbation evoked potential (PEP) components were found between 70-155 ms after the onset of the external perturbation. To decode pre-processed EGG data, four (4) artificial neural networks were tested and different network architecture parameters and electrode layouts were compared. Overall, the convolutional neural network trained to predict EEG balance disturbances had a far superior classification performance than the other architectures, whose mean accuracy was 91.51 ˘ 2.91%, using a short window length of 200 ms. The electrode layout composed of 5 channels (Fz, C3, Cz, C4, and Pz) presented the shortest execution time to train the model, whose average value was 196 ˘ 44.24ms. In addition, it was possible to verify that the use of a single electrode (Cz) obtained satisfactory precision results (86.47 +/- 0.03%). These discoveries may contribute to the development of a system capable of detecting equilibrium disturbances in real-time. K E Y W O R D S Brain-computer interface, Electroencephalogram, Falls recognition, Pertubation-Evoked Potential Deep learning. c RESUMO As quedas são uma das causas mais comuns de lesões na população idosa. Como resultado, custos com o tratamento têm também aumentado. Esforços recentes para restaurar a função dos membros inferiores nessas populações viram um aumento no uso de sistemas robóticos vestíveis, no entanto, as medidas de prevenção de quedas nesses sistemas exigem a detecção precoce da perda de equilíbrio para serem eficazes. Em suma, é fundamental o desenvolvimento de tecnologias, tal como interface cérebro-computador, que seja capaz de reconhecer situações risco de queda com base na perda de equilíbrio ocasionada por diversos fatores. Estudos anteriores investigaram se as variáveis cinemáticas continham informações sobre uma queda iminente, mas poucos examinaram o potencial do uso da eletroencefalografia (EEG) como um sinal de previsão de queda e como o cérebro responde para evitar uma queda. As perturbações do equilíbrio percebidas são sempre acompanhadas por uma ativação cortical específica, chamada de potencial evocado por perturbação (PEP). Neste estudo, também fazia parte do objetivo inicial o reconhecimento das atividades diárias (andar, levantar, agachar, subir e descer escada), porém, devido aos desafios encontrados, o objeto de estudo do presente trabalho foi focado no reconhecimento e classificação binária da presença de perda de equilíbrio (PEPs) em sinais cerebrais. Assim, esta dissertação pretende dar os primeiros passos em direção a decodificação da atividade cerebral em resposta a eventos de desequilíbrio. Inicialmente, para adquirir os dados, um protocolo experimental foi delineado, de forma que os participantes, usando EEG, fossem submetidos a perturbações do tipo deslizamento enquanto andavam sobre a esteira. Dois sujeitos saudáveis foram expostos a perturbação do tipo deslizamento, e estas perturbações ocorreram intercaladamente em um período de duração de 30 a 60 segundos. Cada sujeitou realizou 2 experimentos, ou seja, perturbações provocadas enquanto o indivíduo andava na passadeira: i) a uma velocidade de 1.6 km/h e ii) a uma velocidade de 2.5 km/h. Com base nos métodos abordados os componentes de potencial evocado de perturbação (PEP) foram encontrados entre 70-155 ms após o início da perturbação externa. Para decodificar dados pré-processados do EGG, quatro (4) redes neurais artificiais foram testados e diferentes parâmetros da arquitetura de rede e layouts de eletrodos foram comparados. No geral, a rede neural convolucional treinada para prever as perturbações do equilíbrio do EEG teve um alcance desempenho de classificação superior as demais arquiteturas, cuja precisão média foi de 91,51 ˘ 2,91%, usando um comprimento de janela curto de 200 ms. O layout de eletrodo composto por 5 canais (Fz, C3, Cz, C4 e Pz) apresentou o menor tempo de execução para treinar o modelo, cujo valor médio foi 196 ˘ 44.24ms. Além disso, foi possível verificar que a utilização de um único eletrodo (Cz) obteve resultados de precisão satisfatórios (86.47 +/- 0.03%). Essas descobertas podem contribuir para o desenvolvimento de um sistema capaz de detectar perturbações do equilíbrio em tempo real. PA L AV R A S -C H AV E Interface cérebro-computador, Eletroencefalograma, Reconhecimento de queda, Potencial Evocado de Perturbação, Aprendizagem profunda. d CONTENTS Contents iii List of Acronyms 5 1INTRODUCTION 6 1.1 Motivation 6 1.2 Problem Statement 7 1.3 Dissertation Goals and Research Questions 9 1.4 Scientific Contributions 10 1.5 Dissertation Outline 10 2BRAIN COMPUTER INTERFACES AND DECODING BRAIN SIGNALS 12 2.1 EEG-Based Brain Computer Interfaces 12 2.1.1 BCI Features and Resources 13 2.1.2 Electroencephalography 14 2.1.3 EEG Control Signals 16 2.1.4 P300 16 2.1.5 Perturbation Evoked Potential (PEP) 17 2.2 Decoding 23 2.2.1 Data Preprocessing and Feature Extraction 23 2.2.2 Classification - AI Algorithms 25 2.2.3 BCI Applications for Activity Daily Life 28 2.2.4 Prevent Fall Risk 29 2.3 Discussion 29 3 ACTIVITIES OF DAILY LIVING AND RESPONSE TO BALANCE LOSS:L I TE R AT U R E R E - VIEW 31 3.1 Methods 32 3.1.1 Research Strategy 32 3.1.2 Inclusion and Exclusion Criteria 32 3.2 Experimental Protocol and Setup 32 3.2.1 Experimental Procedure 32 3.2.2 EEG Setup 36 iii CONTENTS iv 3.3 EEG Data Acquisition and Processing 37 3.4 Outcomes 41 3.4.1 Gait Deconding from EEG with LSTM 42 3.4.2 Gait Intention Using Spatio-Spectral CNN 42 3.4.3 Real-Time EGG-based BCI 43 3.4.4 Decoding Sensorimotor Rhythms 43 3.4.5 Decoding Movement-Related Cortical Potentials 44 3.4.6 Neural Decoding of Gait in Developing Children 45 3.5 Discussion 45 4 DECODING LOWER LIMB LOSS OF BALANCE FROM BRAIN SIGNALS:DESIGN CONCEPTION 47 4.1 Introductory insight 47 4.2 Research Hypothesis 48 4.3 Project Conceptual Design 48 4.3.1 Slip-like perturbations experimental data analysis 49 4.3.2 Decoding perturbations events 50 4.4 Outcomes 50 4.5 Discussion 51 5INDUCED LOSS BALANCE DATA ANALYSIS 52 5.1 Introductory insight 52 5.2 Methods 52 5.2.1 Experimental Procedures 53 5.2.2 Data Pre-processing 55 5.3 Outcomes 57 5.4 Discussion 59 6DECODING LOWER LIMB SLIP-L I K E P E R T UR BAT I O N 61 6.1 Introductory insight 61 6.2 Methods 61 6.2.1 Data Labeling 62 6.2.2 Classification Models and Architectures Parameters 63 6.2.3 Model Building and Evaluation 66 6.2.4 Model Evaluation Metrics 68 6.3 Results 71 6.3.1 Deep Learning Architecture 72 6.3.2 Models and Parameters Evaluation 74 LIST OF TABLES Table 1 Experimental Protocol - Procedure 34 Table 2 Experimental Protocol - Sensor Detail 37 Table 3 Dataset Split (Train, Validation, and Test) & Accuracy 41 Table 4 New datasets extracted from data segmentation 63 Table 5 Specifications for the use of the Deep Learning models 66 Table 6 Subset of Features per Dataset and Deep Learning-Based Model 67 Table 7 Classification Metrics Results for Channels’ Dataset 72 Table 8 Classification Metrics Results for ICs’ Dataset 73 Table 9 Number of results from Channels data (Accuracy ěTarget) 76 Table 10 Number of results from ICs data (Accuracy ěTarget) 77 Table 11 Comparative table between the data set of Channels and ICs that obtained better results in their respective network architectures of the model 78 Table 12 Dataset extracted from EEGLAB 102 xi ACRONYMS ACC Anterior Cingulate Cortex. 43 ADL Activities of Daily Living. 7,9,10,29,30,41,49,51,56,81 AI Artificial Intelligence. 30,31,47,48,50,51,61,79 AM Amplitude Modulated. 45 ANN Artificial Neural Networks. 9,26,65 ASR Artifact Subspace Reconstruction. 23,24,39 BCI Brain Computer Interface. 6,8,9,10,12,13,14,15,16,18,23,25,26,28,29,30,31,41,43,44,45,48,50 BF Biceps Femoris. 37 BiRDLab Biomedical Robotic Devices Lab. 6,50,51,52,53 CAR Common Average Reference. 23,24,38,56,73 CCA Canonical Correlation Analysis. 44 CLDA Closed-Loop Decoder Adaptation. 41 CMEMS Center for Micro-Electro-Mechanical Systems. 6 CNN Convolutional Neural Network. 9,26,42,43,46,63,73,76,77,80,81 DL Deep Learning. 6,10,11,26,30,50,62,67,73,74 ECG Electrocardiographic. 13,16,18 EEG Electroencephalogram. 6,7,8,9,10,12,13,14,15,16,17,18,19,23,24,25,26,28,29,30,31,36,37, 38,39,41,42,43,44,45,46,47,48,49,50,51,52,53,55,56,57,60,61,62,77,78,80 EKG Electrocardiographic. 13 EMG Electromyography. 13,16,18,23,37,80,81 EOG Electrooculogram. 13,16,36,37,41 ERD Event-Related Desynchronization. 39,42,43 ERP Event-Related Potential. 8,17,77,78,79 ERS Event-Related Synchronization. 39 FDA Food and Drug Administration. 7 FIR Finite Impulse Response. 37 fMRI functional Magnetic Resonance Imaging. 8,14 FN False Negative. 69 fNIRS functional Near-Infrared Spectroscopy. 8 FP False Positive. 69 3 Acronyms 4 GL Gastrocnemius Lateralis. 37 GM Gastroc Medialis. 37 GND Ground. 36 GRU Gated Recurrent Units. 9,27,63,73,74,78,80,81 hBCI hybrid Brain Computer Interface. 13 HMI Human-Machine Interaction. 8 IC Independent Components. 24,38,39,72,74,77,78,80 ICA Independent Component Analysis. 23,24,38,39,56,74,81 IPL Inferior Parietal Lobe. 43 LOB Loss of Balance. 53,61 LSTM Long Short Term Memory. 9,27,42,63,78,80,81 MCC Matthews correlation coefficient. 71,79 ME Motor Exection. 41 MEG Magnetoencephalography. 14 MI Motor Imagery. 16,28 ML Machine Learning. 30 MLM Mode Labeling Method. 64 MMP Movement Monitoring Potential. 41 MRCP Movement Related Cortical Potentials. 41,42,46 NTrue Negative. 69 PPositive. 69 pBCI Passive Brain Computer Interface. 29 PCA Principal Component Analysis. 24 PEP Perturbation-Evoked Potential. 7,11,17,18,19,21,22,23,49,56,57,58,60 PET Positron Emission Tomography. 8,14 PPC Posterior Parietal Cortex. 43 REF Reference. 36 RELICA Reliable Independent Component Analysis. 39 RF Rectus Femoris. 37 RNN Recurrent Neural Network. 26,27 RP Readiness Potential. 41 SNR Signal-to-Noise Ratio. 13,15 SSAEP Steady-State Auditory Evoked Potential. 16 SSEP Steady-State Evoked Potential. 16 Acronyms 5 SSSEP Steady-State Somatosensory Evoked Potential. 16 SSVEP Steady-State Visually Evoked Potential. 16,28 STD Standard Deviation. 24 STG Superior Temporal Gyrus. 43 TA Tibialis Anterior. 37 TN True Negative. 69 TP True Positive. 69 UKF Unscented Kalman Filter. 41,43 VL Vastus Lateralis. 37 VM Vastus Medialis. 37 WHO World Health Organization. 6 1 INTRODUCTION This dissertation presents the work developed during my Masters’s Degree in Informatic Engineering, at the Biomedical Robotic Devices Lab (BiRDLab) included in the Center for Micro-Electro-Mechanical Systems (CMEMS), a research center of the Department of Industrial Electronics in the University of Minho. The purpose of this dissertation was to investigate the decoding of human brain signals in response to slip-like perturbations. The data that supports this work was acquired with electroencephalography (EEG) recordings of subjects who were submitted to slip-like perturbations while walking on a treadmill at speed of 1.6km/h and 2.5km/h. The work developed represents a step towards a Brain-Computer Interface (BCI) device useful to promptly detect gait disturbances, possibly giving more time margin for fall prevention mechanisms to act and be more effective. This strategy aims to help seniors improve their quality of life, through a BCI that reads brain signals and translates them into commands that will be relayed to the device when it detects a fall event. The project was divided into 3 main phases, namely: i) research and design of EEG data acquisition protocols during balance loss experiments in a laboratory setting; ii) research and development of data pre-processing framework using EEGLAB to remove artifacts from the data and extract relevant features; and iii) research, development, and validation of AI-based Deep Learning (DL) models for the detection of balance loss using EEG data collected and preprocessed from experiments. 1.1 M OT I VAT I O N The aging process has a wide-reaching impact on the human body and brain. It can affect various aspects of daily life, such as the development of new tissues, the maintenance of mental health, and the activities of daily living. Older adults may face difficulties in communicating, focusing, remembering, speaking, moving, or maintaining balance. Thus, the elder may find it challenging to interact with family members, use stairs securely, remember new information, or drive safely due to these impairments [30]. The World Health Organization (WHO) [ 50 ] estimates that over the course of a year, at least one-third of the elderly population suffers a fall, provoked by cognitive, physical, and sensory deficits which arise with aging. Accidental falls are one of the leading causes of injuries and the second leading cause of death related to unintentional injuries. The negative psychological and social repercussions of falls include not fulfilling daily tasks due to constant threat by the unpredictability of fall risk events and practicing fewer physical activities, which leads to a lower 6 1.2. Problem Statement 7 quality of life [ 2 ]. As a result, there is an urgent need to create fall prevention and prediction technologies in order to reduce the individual and societal burden of falls. Conventional therapies such as exercising, managing medications, having vision checked, and making home safer homes have been successful in fall reduction and prevention, however many individuals with severe illness or injury remain unable to participate in activities of daily living (ADL) or complete standard care protocols [ 35 ]. Due to the need to improve the recovery of the locomotion impaired, robotic-assisted devices such as prostheses, electrical stimulators, and exoskeletons have been developed. Additionally, similar types of devices can be also applied for fall prevention purposes [7]. Powered exoskeletons are categorized by the U.S. Food and Drug Administration (FDA) as Class 2 medical devices subject to additional regulations. They are commonly utilized in rehabilitation applications since they may offer active support for sitting, standing, and walking. Exoskeletons can be used for rehabilitation as well as to lower the risk of falling and/or help avoid falls [ 35 ]. However, falls while wearing exoskeletons pose a substantial risk while using these technologies. Current exoskeletons approved by the FDA employ a variety of tactics to prevent falls and are intended for use in conjunction with a trained partner. The efficiency of these approaches has not been researched and is currently unknown [35]. Near-fall detection could provide new opportunities to identify older people at high risk of falling before a fall occurs. Near falls are defined as trips, slips, and missteps and involve a loss of balance that does not result in a fall because corrective action is taken to recover balance. Near falls occur more frequently than actual falls, and older people who frequently experience near falls are at increased risk of future falls [ 20 ]. Therefore, for patients to have a natural and effective recovery, they must fully control the devices they use, such as exoskeletons, prostheses, orthoses, and robotic canes. 1.2 P R O B L E M S TAT E M E N T It is suggested that human bipedal balance control is the combined activity of dispersed brain regions. However, while there is evidence linking the cortical activity of specific brain regions, such as the motor cortex, to reactive balance regulation, less is known about the functional interplay of additional cortical regions [49]. Electroencephalography signals recorded from the scalp reflect synchronous activation of cortical neurons [ 48 ]. During active walking, robot-assisted passive walking, and walking with interactive feedback in a virtual environment, EEG recordings revealed activation of electrocortical sources distributed across different cortical regions including the anterior cingulate, prefrontal cortex, pre-motor cortex, supplementary motor area, posterior parietal, and sensorimotor cortex [49]. Recent research shows that brain involvement occurs during balance reactions induced by whole-body postural disturbances. An external disturbance evokes an evoked potential in humans, known as the perturbation-evoked potential (PEP). PEP are broadly dispersed throughout the fronto-centro-parietal regions, with the greatest amplitude occurring at the FCz/Cz electrodes [ 48 ]. Specifically, PEP can be used to study sensory and cortical control of postural responses. 1.2. Problem Statement 8 BCI based on EEG technology has been employed in this context to develop portable synchronous and asynchronous control and communication. Noninvasive EEG-based BCI can be classified as "evoked" (reactive BCI), "spontaneous" (active BCI), or passive BCI. An evoked BCI takes use of a major feature of the EEG, the so-called evoked potential, which displays the brain’s instantaneous automatic reactions to some external stimuli. On the other hand, spontaneous BCI is based on the investigation of EEG events linked with various elements of brain activity connected to mental tasks performed at the user’s discretion [ 30 ]. Finally, the concept of passive BCI has recently been developed in the BCI area, in which the system takes action to perform a command based on the random brain activity of the user while the participant lacks voluntary control over the system [ 41 ]. The use of EEG signal may increase the applicability of individualized brain-controlled rehabilitation equipment. Individually tailored parameters for describing signals are generated from many training trials in the development of EEG-based BCI, which are then utilized to perform online signal decoding [25]. As a result, EEG might reveal the cortical activations associated with loss of balance or balance regulation. Furthermore, when compared to other neuroimaging techniques such as Positron Emission Tomography (PET), functional Magnetic Resonance Imaging (fMRI), or functional Near-Infrared Spectroscopy (fNIRS), EEG provides superior temporal resolution (1 ms), allowing researchers to precisely time-lock neural activity relative to perturbation onset or subsequent movements in order to extract perturbation-related or response-related cortical activations [48]. One disadvantage of utilizing EEG is that it measures the total of electrical potentials from several sources of cerebral activity that correlate to the distinct sensory, motor, or cognitive events. Nevertheless, it is possible to extract Event-Related Potentials (ERP) associated with specific occurrences by averaging across EEG epochs [48]. The ability to predict human movement intention is critical for successful gait restoration. Brain waves are captured, processed, and interpreted to operate an assistive device in a BCI-based rehabilitation system. It is vital for a successful assistive system to detect movement intention as early as possible to allow the system adequate time to adjust to the individual’s needs [20]. Assistive, adaptive, and rehabilitative BCI applications for older adults and elderly patients should be created to help with household duties, strengthen connections with family members, and improve cognitive and motor abilities. Many BCI applications have been developed in the last decade to assist the elderly keeping them healthy, with high quality of life, and a feeling of well-being [30]. The main goal of research in the field of Brain-computer interfaces is to gain control of a computer or machine purely by utilizing users’ thoughts. While there has been a major emphasis on utilizing BCI to operate assistive devices, BCI can improve Human-Machine Interaction (HMI) [ 10 ]. Thus, an effective algorithm for detecting near falls based on ata can contribute to the development of fall detection systems by improving their performance, time response, and quality in terms of accuracy. 1.3. Dissertation Goals and Research Questions 9 1.3 DISSERTATION GOALS AND RESEARCH QUESTIONS The present work aims to decode events of imbalance in human gait from the collection of brain signals through the use of the EEG. A protocol for data collection was designed based on good practices found in the scientific literature. The data were collected with different subjects, wearing an EEG cap with 16 channels and submitted to slip-like perturbations while walking on a treadmill in a controlled environment. EEG data were processed to create relevant features, through feature engineering, which were then used to detect loss of balance during walking on a treadmill through the implementation of artificial intelligence models. Because it was necessary to identify the most used pre-processing and artifact removal methods and algorithms, as well as identify the most relevant brain rhythms. Furthermore, the performance of different Artificial Neural Network (ANN) models, such as Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), Gated Recurrent Units (GRU), among others, were evaluated and compared in order to find a benchmark model that can identify the perturbations, being robust enough to be implemented and integrated into different robotic platforms. The present dissertation focused on pursuing the following towards the outlined solution: • Goal 1: Gather information about human brain signals in response to ADL activities and loss of balance, by performing a state-of-the-art review, and understanding the most effective and significant paradigms, techniques, and devices. This will be presented in Chapter 2. Key Performance Indicators (KPIs): i) identify BCI technologies applied in the real world; ii) EEG signals related to BCI application; iii) EEG variables (namely latency, peak amplitude, and time to peak) regarding brain signals in response to slip events evidenced in literature [48]; • Goal 2: Gather knowledge about methods used currently for decoding ADL activities, from data acquisition to Artificial Intelligence models by performing a literature review. This method will be described in Chapter 3. KPIs: i) systematic review for article published; ii) data pre-processed tools used to extract relevant information regarding ADLs; iii) Artificial Neural Network architecture applied to decode brain signal. • Goal 3: Delineate protocols to obtain data from multiple subjects in the laboratory, while performing slip-like perturbations wearing the EEG. This was then used to train and evaluate the best AI-based model. This topic will be addressed in Chapter 4. KPIs: i) definition of an experimental protocol for data acquisition. • Goal 4: Data analysis to identify patterns in the EEG signals and channels most related to the loss of balance caused by slip-like perturbations in different subjects walking on treadmill at different speeds. - Chapter 5. KPIs: i) identify EEG channels where a pattern can be found; ii) detection of signals components in response to slip-like perturbations which positive potential (P1) that peaks around 30–90 ms after perturbation onset and a negative potential (N1) that peaks around 90–160 ms [48]. • Goal 5: Exploit ANN models capable of recognizing slip-perturbation from a dataset with data collected during trials performed in a laboratory setting. Chapter 6shows the obtained results. KPIs: i) obtain an average accuracy higher than 87.6% ˘ 4.2%, and F-Score higher than 87.2% ˘ 5 [ 35 ]; ii) obtain accuracy above 80% for minimal electrode layout (1 channel) [10]. 1.4. Scientific Contributions 10 Consequently, in this dissertation the following Research Questions are expected to be answered: • RQ 1: What are the most relevant signals and features that can help recognize the loss of balance provoked by slip-like perturbations? The answer is included in Chapter 5. • RQ 2: What are the most relevant channels to decode the slip-like perturbations during walking? The answer is included in Chapters 5and 3. • RQ 3: What is the best Deep Learning model to implement for the detection of slip-like perturbation during walking? The answer is included in Chapter 3. 1.4 SCIENTIFIC CONTRIBUTIONS The main contributions of this dissertation to the current state-of-the-art in this field are: • A systematic literature review of brain signals decoding related to daily activities and loss of balance provoked by slip-like perturbations. • Validation of data pre-processing methodologies described in the literature for slip-perturbation during walking on a treadmill. • Creation of a segmented and feature-selected dataset from the raw data collected along slip-perturbation experiments performed in the laboratory, which allows more robust and validated training and testing of AI-based models. • Creation of Deep Learning classification algorithms capable of detecting loss of balance provoked by slip-like perturbations. • Selection through computational performance analysis, of the most relevant features and DL models to be tested in a real-time application. 1.5 DISSERTATION OUTLINE The remainder of this dissertation is organized in 6 chapters: Chapter 2contains a review of the literature regarding brain-computer interfaces, which devices are used for data acquisition with a focus on EEG signal and its paradigms. In addition, in this chapter data pre-processing techniques and the AI-based models applied for the decoding brain signal in general BCI applications are presented. Chapter 3includes a start-of-the-art review about decoding human brain signals strategies to identify different ADL and induced loss of balance in a static position. This analysis will include the experimental protocol previously developed to collect the data, steps related to the data pre-processing, classification applying Deep Learning algorithms, and discussion about the results obtained from the review. 1.5. Dissertation Outline 11 In Chapter 4, the project conceptual design of the dissertation is introduced and shows the dissertation’s main phases. In Chapter 5, the methodology used to obtain the PEP signals involved in the trials developed at EEGLAB will be presented. This Chapter also shows the results which will be addressed in the next chapter. Chapter 6demonstrates the process of developing and evaluating DL-based models for slip-like perturbations, as well as selecting the classifier that best fits the loss balance recognition. Finally, Chapter 7concludes the dissertation, presenting a short analysis of the work developed as well as the answers to the RQs. In addition, it includes the directions for future research improvements to be developed. 2.1. EEG-Based Brain Computer Interfaces 18 Figure 3: ERP scalp maps showing the topography of perturbation-evoked N1. Taken from [48] PEPs can be contaminated by movement artifacts owing to whole-body perturbations, in addition to other neural and non-neural artifacts such as eye blinks, eye movements, EMG,ECG, and line noise artifacts, which are of special concern. It is worth noting, however, that perturbed head motions do not appear to mask the early components of PEPs. This is most likely owing to the timing of events, with the perturbed head movement beginning after the early components of the PEPs [48]. 2.1.5.1 Components The first component is a brief positive wave that lasts between 30 and 90 milliseconds. This is followed by a negative peak at 90-160 ms and a last, late response (P2 and N2) at 200-400 ms. These PEPs are frequently identified by averaging event-related waveforms from several trials. If PEP could be identified in a single trial and used in a BCI system, balance disturbances could be detected far sooner in a fall detection system based on EEG. This would provide considerably more time to implement preventative actions [35]. Figure 4: Waveform and terminology of Perturbation-evoked Potentials from a single normal subject. Taken from [48] 2.1. EEG-Based Brain Computer Interfaces 19 The most commonly researched properties of PEP components are the onset latency, peak latency, and peak amplitude. The peak amplitude of the PEP components is measured with regard to a pre-perturbation baseline period or as the voltage difference between the component peak and the perturbation onset. In Varghese et al. [ 48 ] studies, the latencies of the PEP components are assessed in relation to perturbation onset (i.e., time = 0 (Figure 5)). The peak latencies of the various PEP components vary between studies in the healthy adult population due to changes in perturbation release timing, the metric employed to quantify perturbation initiation or the time point picked by different researchers as the start of timing for the PEPs. For example, perturbations delivered at various stages of the gait cycle may result in variable PEP latencies. Furthermore, the peak amplitude of the PEP components varies with the kind of perturbation utilized and the starting acceleration of the perturbation in the healthy adult population. 2.1.5.2 PEP P1 The first component of the PEP is the P1. Varghese et al. [48] paper summarizes that the peak amplitude of the P1 is small, usually ranging between 0.2 and 7 µV. However, mean amplitudes as great as 12.7 µV were reported for the P1 evoked by sudden platform toe-up (ankle perturbations) tilt, measured from the P1 onset to the peak of P1. Because of its modest amplitude and/or unpredictability in the mechanical initiation of disturbance relative to the trigger, the P1 is not always discernible from background EEG signals. When the disturbance is quick, the P1 is easily detected; but, when the perturbation is sluggish, it might be indistinguishable from the baseline EEG activity [10]. Evidence from previous works [ 10 , 48 ] indicates that P1 is the first unspecific cortical response to perturbations and that P1 features are associated with various parameters such as age, stance width, and height. The same studies indicate that the P1 is most likely generated by somatosensory input. P1 latency is controlled by the subject’s starting postural state (stance or gait), height, age, and diseases, whereas P1 amplitude is determined by the beginning stance width (narrow or wide). It was discovered that when the perturbation is created during stance and gait circumstances, the latency of the P1 peak is considerably delayed in gait situations. With increasing stance width, the amplitude of the P1 rises. Varghese et al. [ 48 ] reported the impact of the base of support by inducing sudden platform toe-up tilts with three different initial stance widths (0 cm, 3 cm, and 30 cm) and discovered that the amplitude of the P1 increased with a wider stance. 2.1.5.3 PEP N1 The PEP N1 is broadly distributed in the frontal, central, and parietal regions. The amplitude of N1 varies from 0.8 to 80 µV depending on the variables varied in the tests and is greatest at either the FCz or CZ electrode [6,35]. Unlike other PEP components, the N1 is reproducible across trials and subjects, regardless of the direction of perturbation or the many parameters that impact the latency and amplitude of N1. Even with various types of perturbations (stance perturbations versus sitting perturbations) and balancing reactions, the PEP N1 occurs bilaterally with no significant changes in magnitude across left and right hemispheres (upper limb vs lower limb) [48]. 2.1. EEG-Based Brain Computer Interfaces 20 (A) ERP-image plot of single-trial EEG epochs across all subjects. (B) Event-related spectral perturbation (ERSP) and (C) inter-trial coherence (ITC) plots at the FCz electrode site. The vertical dashed line at time = 0 represents the perturbation onset. The bottom trace (D) shows the perturbation-evoked N1 averaged across all subjects. Figure 5: ERP the scalp maps, time series, and time–frequency plots of PEP N1 averaged from a single-subject EEG data. Taken from [48] 2.1. EEG-Based Brain Computer Interfaces 21 It has been proposed in Varghese et al. [ 48 ] that the N1 is associated with error detection and is independent of sensory and motor processes linked with compensatory postural responses. They discovered that the amplitudes of the N1 were smaller during predictable perturbations compared to unexpected situations. A substantial N1 potential emerged, however, when an unpredictable perturbation was applied following a sequence of predictable perturbations. After observing similar N1 peak latency and amplitude for stance and seated perturbations, the authors [ 10 , 48 ] proposed that the N1 response is driven by the perturbation and is independent of specific sensory inputs that code the perturbation characteristics and motor elements of the balance reactions. Despite these differences in the role of the N1, it is acceptable to assume that the N1 plays a role in compensating postural responses. However, due to the lack of information on previous reports, further studies are needed to understand the significance of the N1, especially whether it plays a role in detecting postural instability and/or generating postural responses. The PEP N1 is the outcome of averaging across many trials. As such, it is simply a summary measure of cortical activations linked to balancing regulation, and averaging may result in the loss of significant information about the underlying neuronal assemblies involved in producing the PEP N1. The participation of distinct frequency bands during the PEP N1 has been linked to sensorimotor and cognitive processes connected with reactive balance management. Furthermore, concurrent theta, alpha, and beta frequency phase synchronization during this phase of the response to balance perturbation demonstrates the participation of many concurrent cognitive processes in balance regulation, and that their combined activity leads to the emergence of the PEP N1 [48]. The PEP N1 is the most extensively researched component of the PEP. Several environmental factors impact the PEP N1 latency, amplitude, and psychological factors such as initial postural state (stance or gait), learning effects or adaptation, perturbation amplitude and duration, the direction of perturbation (forward or backward), perturbation mode (unilateral or bilateral), mechanical triggering mode of perturbation (self-induced or externally induced), the base of support (stance width), concurrent cognitive task (dual tasking) and postural threat. As mentioned by Varghese et al. [ 48 ]PEPs related to both stance and gait perturbations using treadmill accelerations indicated delayed latencies and lower amplitudes in the PEP N1 during the walking test compared to the standing condition. The authors hypothesized that the lower amplitude of the PEP N1 explains neurons’ failure to react to perturbation stimuli caused by active movement during gait. In contrast to the differences in amplitudes and latencies between stance and gait, no difference in N1 amplitude and latency was observed between seated perturbations that elicited upper limb compensatory postural responses and stance perturbations that elicited lower limb compensatory responses. Also, the authors found that changes in sensory qualities (sensory from sitting vs stance perturbations) and motor responses (upper limb versus lower limb) required to recover balance have no effect on the spatiotemporal aspects of the PEP N1. 2.1. EEG-Based Brain Computer Interfaces 22 ERP-scalp maps (A) and ERP time series (B) of PEP N1 at the FCz electrode site averaged from a single normal subject. Time = 0 represents the perturbation onset. (C) Event-related spectral perturbation plot at the FCz electrode from a single normal subject. Figure 6: ERP plots. Taken from [49] 2.1.5.4 PEP P2 and N2 The PEP N1 is followed by the P2 and N2 (known as late PEPs). In contrast to the PEP N1, the P2 and N2 are not identified in all PEP research. Varghese et al. [ 48 ] cite that early PEP experiments, in particular, were unable to uncover any long latency components (longer than 180 ms). One possible explanation is that these lengthy delay components are associated with task-specific behavior and their timing is not fixed, depending on the task [ 48 ]. As a result, late PEP are analyzed in a wider time window after the N1 response, typically 200-400 ms after the start of the perturbation [ 10 ]. The N2 amplitude fluctuates between 27 and 28 µV and is greatest at the CZ electrode. The P2 peak amplitude spans between 22 and 44 µV and is greatest at the Cz or CPz electrodes. The starting stance width influences the magnitude of the P2 [48]. As a result, the authors [ 48 ] hypothesized that late PEP could indicate cognitive processes linked with task demands or cognitive processing of the balance disturbance rather than sensorimotor processing involved with balance response regulation. The same authors discovered that a concurrent cognitive task performed during 2.2. Decoding 23 the stance perturbations had no effect on the amplitude of the late PEP and argued that they indicate a sort of orienting response that does not demand the attentional resources required in a dual-task paradigm. The amplitude of P2 but not the latency is also modified by the initial postural position (stance or sitting), with stance perturbations producing much greater P2 amplitudes than seated perturbations. This discrepancy was attributed to differences in the latter phases of balance control between standing and sitting circumstances [10]. To summarize, late PEP are unlikely to represent cortical sensorimotor processing related to postural response regulation, and if attention does explain variation in late PEP, it is more likely driven by attention changes toward new perturbation events. 2.2 DECODING As mentioned in the previous subsection, to build a BCI system, five components are generally needed: signal acquisition during a specific experimental paradigm, preprocessing, feature extraction, classification (detection), translation of the classification result to commands (BCI applications), which one of them will be detailed on next subsections. For quick and accurate processing and analysis of brain data, researchers have developed many open-source software packages and toolboxes such as BCI20001, EEGLab, FieldTrip, and Brainstorm. These software packages are based on advanced signal and image processing methods and artificial intelligence programs for performing sensor or source-level analyses. Prediction of human movement intention is highly significant for successful gait rehabilitation. In a BCI-based rehabilitation system, the brain waves are extracted, processed, and translated to control an assistive device. For an effective assistive system, it is critical to detect the movement intention as early as possible to provide the system with enough time to adapt to the requirement of the individual [16]. 2.2.1 Data Preprocessing and Feature Extraction 2.2.1.1 Data Preprocessing Pre-processing is a time-consuming procedure that is used to eliminate any undesirable components present in the EEG signal. Good preprocessing improves signal quality, which leads to improved feature separation and classification performance. The basic approaches to reducing artifacts in the observed EEG are simply low, high, and bandpass filters. However, these are only effective when the signal’s frequency bands do not overlap. When there is spectral overlap and artifacts are recorded with the EEG, various artifact removal techniques such as adaptive filtering, Artifact Subspace Reconstruction ASR, Common Average Reference CAR, and Independent Component Analysis ICA decomposition. Besides, in some cases to increase computational efficiency and sync EEG frequency with other devices, for example, EMG, the signal is downsampled. As mentioned in the previous paragraph for data preprocessing, firstly, a series of filters are applied to the signal to extract accurate information. The raw signal is collected from the electrode, and stored in a circular 2.2. Decoding 24 Figure 7: Performance of ASR in removing eye movement artifacts and unnatural high amplitude noise. buffer for easier data stream handling. The notch filters, type of filter, can be applied to the raw signal to diminish power line noise and its harmonic interference at 50Hz. The filtered signal is then filtered with a 2nd or 4th-order Butterworth filter can be the low-pass, high-pass, or band-pass filter to extract the information [ 12 ]. If needed the filters can be combined to obtain better signals. In addition, an approach to remove artifacts is to perform amplitude threshold rejection removing all trials with an amplitude that exceeds a specific pre-defined voltage. The rejection threshold can be adjusted based on the value of the standard deviation (STD) [ 10 ]. Other approaches can be used in process of removing unwanted artifacts from EEG signal and depending on the application they can be carried out in two steps. Artifact Subspace Reconstruction is a non-stationary method that uses sliding window principal components analysis (PCA) to remove unusual large-amplitude noise or artifacts. The usage of ASR increases data stationarity and makes the data suitable for independent component analysis operation. In [ 16 ] studies, ASR can be used for two purposes: bad channel rejection (flat channels, channels with a large amount of noise may be removed based on their standard deviation, and channels, which are poorly correlated with other channels), and removal of short-time high-amplitude artifacts in continuous data. The value is chosen in such a way that it was small enough to remove activities from artifacts and eye-related components and large enough to retain signals from brain-related components. According to Varghese et al. [ 49 ], ICA is performed on epoched EEG data to remove eye blinks, eye movements, whole-body movements, muscle artifacts, heart signals, and line noise artifacts. Identification and rejection of artifact-independent components (IC) is based on the visual inspection of topographic maps, power spectra, and time domain activity of each IC [49]. Lastly, the common average reference which is spatial filtering can be applied to the raw input signals to improve the signal-to-noise ratio. For every sample time, CAR subtracts the mean value of all electrodes, which minimizes the uncorrelated random noise with a zero mean through the averaging process [25]. 2.2. Decoding 25 2.2.1.2 Feature Extraction After the line noise and artifact removal phase, the most discriminative and non-redundant information within the EEG is extracted through different feature extraction techniques. Time-domain, frequency domain, time-frequency domain, and spatial domain are the popular types of feature extraction techniques in EEG-based BCIs. The features can be extracted from all channels or from specific channels depending on preprocessing results. In Jochumsen and Niazi [ 23 ], the template matching feature, an average of the epochs for the specific channel was used, and the cross-correlation is calculated with each epoch. The feature was the cross-correlation with zero time lag. The time-domain features were the mean amplitude of 0.5-second non-overlapping windows (i.e.: four features were extracted from each channel). The frequency-domain features were power spectral density estimated for the entire epoch in 1 Hz bins from 6–30 Hz. A 1-second Hamming window with a 0.5-second overlap was used. For time-series applied data segmentation to divide the data into time intervals [ 16 ] where the signals are segmented in epochs locked to the onset of each class [ 32 ]. Using a sliding window with shift and overlapping is possible to extract the signal amplitude in specific channels or components that will be used as input to the classifier [51]. Besides, windows of different time lengths can be chosen to correspond to a specific class. If the number of features is too high compared to the number of samples, there is a high chance of having redundant and noisy features in the feature set. That is why a feature selection method is necessary to remove redundant features. In Hasan et al. [ 16 ] work, the absolute value of the standardized u-statistic of a two-sample unpaired Wilcoxon test (also known as the Mann-Whitney test) was chosen to be the criterion to select distinctive and informative features. To further reduce the number of features, the average of the absolute values of the cross-correlation coefficient between the candidate feature and all previously selected features were calculated, and features that were highly correlated with the features already picked were less likely to be included in the output list. This procedure ensured the formation of a reduced and more distinctive set of features for successful classification. Dimensionality reduction techniques improve the performance of classifiers, as they reduce the number of features to mitigate the curse of dimensionality. These techniques can be categorized into feature selection (e.g. F-score) selects a subset of original features and feature extraction which derives new features from original features. The first category does not change the nature of the features, while in the second category, mathematical transformations are used to reduce the number of features [37]. 2.2.2 Classification - AI Algorithms To operate a BCI system, the individual must generate distinct brain activity patterns that the system can recognize and convert into commands. It should be noted that either regression or classification techniques might be used to achieve the stated goal. However, the use of classification algorithms is the most prevalent strategy as the degrees of freedom of most BCI systems are low and output commands are reduced to just a few options (classes). The classification step’s design requires selecting one or more classification algorithms from a large number of options. 2.2. Decoding 26 The capability to achieve reliable automated categorization of EEG data is a critical step toward making EEG more practical in many applications and less reliant on skilled specialists. It is worth mentioning that, despite major progress in traditional BCI systems over the last two decades, the biggest challenge still confronts significant problems in EEG categorization. Recently, the availability of large EEG data sets has led to the use of Deep Learning (DL) architectures, particularly to extract relevant information from signals that were previously impossible to obtain using conventional approaches, and has demonstrated success in addressing the aforementioned challenges. 2.2.2.1 Convolutional Neural Network Convolutional Neural Networks (CNN) is a class of artificial neural networks (ANNs) that specializes in spatial information exploration. CNNs have seen a lot of successful applications in many different domains such as reaching human-level performance in image recognition problems as well as different natural language processing tasks Motivated by the success of these CNN architectures in these various domains, researchers have started adopting them for time series analysis [17]. CNN has at least one convolutional layer, which uses a convolution operator to map the input to output [ 43 ]. ACNN’s three major layers are convolution, pooling, and fully connected as shown in Fig. 8. On a dataset of huge registers of various classifications, CNNs outperform the prior highest-computing techniques. CNNs are feed-forward neural networks that may be used for image processing, pattern recognition, and classification. The convolution layer filters the input data, such as kernels with trainable parameters, creating the next layer in the network also called the feature map. The feature map is then down-sampled using the pooling layer to reduce the dimension and consequently the computational complexity and overfitting. These settings enabled the learning of many network features while keeping the number of traceable parameters low. After several alternating convolution and pooling operations, the final feature map is unrolled to a fully connected hidden layer to generate the output [34]. Figure 8: Example of CNN Architecture. Taken from [38] 2.2.2.2 Recurrent Neural Network Recurrent Neural Network (RNN) architecture is a sophisticated Deep Learning classification algorithm that is designed exclusively for sequential data. This type of DL architecture can assess the entire logical sequence of the incoming data. These logical sequences are content-rich and have a complicated temporal interaction with one another. The basic principle of RNN is that the current network’s hidden state retains the prior input information, 2.2. Decoding 27 which is then used for the next current network. Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) are two popular RNN designs that has received a lot of attention and success [43]. 2.2.2.3 Long Short-Term Memory Long Short Term Memory is a more advanced variant of the standard RNN in which three gates (forget gate, update gate, and output gate) are added to regulate the information to store and pass while avoiding the vanishing gradient problem that is generally associated with training a conventional RNN. Each gate in the cell is in charge of a distinct duty. The forgetting gate’s function is to remove undesired information from the prior state and output from the top hidden layer. The update gate updates new components to the state, while the cell filters the current state and finds desired and unwanted information to guarantee that the output gate selects the critical information generated by it. The data to be output is selected by the output layer, which is handled by the filtered input and cellular state, i.e. the output layer decides which information will be output and processed by the cell state. Figure 9: Example of LSTM Architecture. Adapted from [40] 2.2.2.4 Gated Recurrent Unit The Gated Recurrent Unit (GRU) is another RNN enhancement, and similar to LSTM,GRU uses gating mechanisms to control and manage the flow of information between cells in the neural network. GRU contains two gates (a reset gate and an update gate) that manage how information is retained and sent between nodes. The update gate functions similarly to an LSTM’s forget and input gates, it determines what information to discard and what fresh information to include, and another gate used to select how much past information to forget is the reset gate. Although prior empirical assessments have not revealed a clear winner between GRU and LSTM, it is thought that GRU may be a superior model when dealing with fewer data points to generalize on, due to the fewer parameters in contrast to LSTM [43]. 3.2. Experimental Protocol and Setup 34 steadily for 15 mins where the decoder is calibrated for the next BCI control [ 47 ]. Chisari et al. [ 39 ] and [ 40 ] experiment with the participants walking on a treadmill at two distinct velocities, 2.5 km/h and 3.5 km/h. For each walking velocity, two acquisitions of 10 minutes each were performed. Healthy participants in [ 11 ] study attended one experimental session in which they were asked to either walk in a passive or in an active mode at a speed of 1.5Km/h wearing an exoskeleton, as soon the treadmill started, participants walked for 49s, after which the treadmill was stopped. The treadmill had a delay of at least 7s to come up to a stable speed and 7s to slow down and stop completely. Table 1: Experimental Protocol - Procedure Article Participants Age (avg) Task Treadmill Speed (km/h) Trials Time per trial (min) [43], [47] 8 Healthy 24.5 sit-to-stand; stand-to-sit 1.6 3 20 [32] 8 Healthy 24.5 walking 1.6 3 20 [39], [40] 11 Healthy 30 walking 2.5 / 3.5 2 for each speed 10 for each trial [22] 10 Healthy 25 resting; walking intention; walking 50 0.5 [11] 10 Healthy 32 walking; stand 1.5 7 active walking; 7 passive walking 1 [23] 13 Healthy 24 stand-to-sit; sitto-stand; walking; step up; side step; back step; 50 for each task 180 [37] 27 Healthy 24 walking [46] 5 Healthy 24.5 walking 1.6 [45] 5 Healthy 8 stand-to-sit; sitto-stand; walking; stop; 1.6 20 [51] 1 Healthy; 1 Non-Healthy walking; turn left; turn right; stop 20 [38] 3 Healthy; 8 Non-Healthy walking 9 1 [12] 4 Healthy 23 stand-to-sit; sit-tostand 78 2 [27] 20 Healthy 22 step forward; step up; step back 10 4 [16] 7 Healthy; 2 Non-Healthy 32.6 walking; stop 145 175 Six of these 16 manuscripts performed gait movements during overground locomotion. The experiment paradigm in [ 22 ] consisted of three states: a resting state of approximately 10 s, an intention state, and 10 s exoskeleton walking. Prasad et al. [ 51 ] is composed of 2 tasks, the first task was a four-class, single-session task in which the subjects performed different movements (i.e.: walking forward, turning right, turning left and stopping, 3.2. Experimental Protocol and Setup 35 following the marked path on the ground, and the second task, subjects only executed walking and stop motions according to audible beep instructions). Eguren et al. [ 45 ] perform four different types of locomotion tasks (i.e.: sitting and standing, start and stop overground walking). In the sitting and standing task, the participants completed 20 sit-to-stand and stand-to-sit transitions which were initiated by a visual cue placed in front of the subjects. The session began with the participant standing quietly in an upright posture for 15 seconds. After the quiet standing period, the participant began the self-initiated sit-to-stand and stand-to-sit transitions. The waiting period between the transitions was approximately 10 seconds. The participants were verbally informed when 20 transitions were completed. At the start and stop of the overground walking session, the participants completed 20 walk-to-stand and stand-to-walk transitions. The session began with the participants standing still for 15 seconds. After the stationary standing period, the participant held the standing position for a period ranging randomly from 10 to 15 seconds before the stand-to-walk transition (indicated by a green light). The participants walked on a 10-meter walkway and were instructed to stop walking at the onset of a red light. The walk-to-stop transition completed a single trial and the participant returned to the starting position. The process continued until 20 trials were completed. Jochumsen et al. [ 23 ] performed 6 tasks which were: (1) stand-to-sit; the subject was standing in front of a chair (height of seat: 45 cm) and had to sit on that, (2) sit-to-stand; the subject was sitting on the chair and had to stand up, (3) walking; the subject had to walk three strides (starting with the right leg), (4) step up; the subject had to step up to a plateau (height: 16 cm) starting with the right foot, (5) side step; the subject took one step to the right side, and (6) back step; the subject took one step back starting with the right foot. Each run consisted of six blocks where each block was 10 movements of the same movement type, (i.e., after each run 10 movement trials were performed of each movement type). After each run, a resting recording of two minutes was performed (i.e., five recordings in total) while the subjects were standing relaxed and focused on a point on the wall four meters away. A clock was counting down to three seconds, and the subject had to initiate the movement task at this point. The movement trials were separated by 15 s. The subjects were instructed not to blink or do any facial movements during the 3-second countdown and while the movement task was performed. The experiment was performed in an electrically shielded room, and it lasted approximately three hours. The single trial performed in [ 38 ] experiment is composed of trial start, auditory cue, start walking, stop walking, and trial end. It took about 1 minute for each trial. Each subject was asked to walk into the room after the cue sign. They walked at their natural pace while looking at the marked dot to minimize ocular and head movement artifacts. Tan Abdullah Al-Mamun et al. [ 37 ] the subjects had to perform free walking along an approximately 21-meter corridor. Lastly, in the experiment [ 32 ] for sit-to-stand and stand-to-sit, a video stimulus that lasted for 4s to 5s and showed either the sit-to-stand or stand-to-sit video task, was presented to guide the participants to avoid the ambiguity of the instructions. The protocol began with a sitting posture, followed by 5 repeated trials of sit-to-stand and stand-to-sit tasks alternatively. 3.2. Experimental Protocol and Setup 36 Figure 12: The 10/20international system of electrode positions for EEG. Taken from [29] 3.2.2 EEG Setup According to the articles reviewed in this study, all the EEG data were collected and labeled in accordance with the extended 10-20 international system (see an example in Fig. 12). Under the 10/20 system, the skull is divided into six areas from nasion to inion with interval rates of 10%, 20%, 20%, 20%, 20%, and 10% (Fp: frontopolar, F: frontal, C: central, P: parietal, and O: occipital, respectively), and also divided into the same ratios from left to right preauricular points (T3: temporal, C3: central, Cz, C4, and T5, respectively). Four (4) papers describe the use of a 64-channel active EEG electrode system in which 4 channels were used as EOG sensors to capture and remove eye-related artifacts using adaptive filtering algorithms. The sampling frequency in 3 of 4 papers mentioned was set to 100 Hz [ 43 , 47 , 46 ]. Ground (GND) and reference (REF) channels were placed on the left and right earlobe (A1 and A2), respectively. T7 and T8 channels were moved to FCz and AFz, respectively. FT9, FT10, TP9, and TP10 were used as EOG to capture eye blinks and eye movements. This modification aims to improve the decoding accuracy based on two main reasons: 1) GND and REF channels in the standard setup were close to the motor cortex and 2) EOG sensors were required in the artifact removal algorithm [43,46]. Eguren et al. [45] study a 64-channel was used to record wirelessly at 1000 Hz from the face and scalp. Channels TP9, PO9, PO10, and TP10 were removed from the cap and used for EOG to capture blinks and eye movements; however, these data were excluded from all analyses in this experiment. EEG data were recorded with a custom signal pre-amplifying active electrode cap and a 64channel EEG amplifier with a sampling rate of 2048 Hz/channel (bandwidth DC - 1024 Hz), however EEG was resampled at 1024 Hz before further preprocessing [ 39 , 40 ]. Jochumsen et al. [ 23 ] also recorded using a 64-channel sampled with 1200 Hz whereas [51] the sampling frequency was 100Hz. Nienhuis et al. [ 11 ] electrical signals from 62 electrodes were recorded at a 500Hz sampling rate, while Jeong et al. [ 22 ] and Park et al. [ 38 ] used 32-channel wireless EEG data system, however in the first paper the authors digitized the frequency at a sampling of 1000 Hz and second paper at 500Hz. In addition, in contrast with the rest of the articles presented in this study, [ 51 ]EEG (64 channels) was recorded by combining two 32-channel amplifiers (sampled at 100 Hz). 3.3. EEG Data Acquisition and Processing 37 From whole papers, only [ 32 ] obtained EEG signals using 11 passive electrodes with a sampling rate set to 1200 Hz, and the reference and ground electrodes placed on the left and the right earlobes, respectively. EOG signals were acquired from 2 passive electrodes positioned under and next to the outer canthus of the right eye. The system in this paper and in other 5 papers was set up to record EMG signals simultaneously throughout the experiment which is often used to modulate the gait pattern and some application to identify the onset of the movement. EMG electrodes were placed on Tibialis Anterior (TA), Vastus Medialis (VM) and Biceps Femoris (BF) of each leg [ 40 , 39 ], Rectus Femoris (RF), TA, and Gastrocnemius Lateralis (GL) of two lower limbs [ 32 ], Gastroc Medialis (GM), Semitendinosus and Vastus Lateralis (VL) [ 11 ], two bipolar Ag/AgCl electrodes on the TA and BF muscles of the right leg (these muscles are known as fast activation muscles for walking)[ 22 ] and in [ 16 ] study The EMG channel was placed at the mid-belly of right leg TA muscle with the reference electrode placed on the bony surface of the right knee. TA was chosen because it is one of the muscles which activates the earliest during a gait cycle. Table 2shows EEG details from 16 studies. Table 2: Experimental Protocol - Sensor Detail Article EEG Channels EEG Frequency Additional Sensors [43], [47], [46] 64 100Hz EOG [39], [40] 64 2048Hz EMG [23] 64 1200Hz [45] 64 1000Hz EOG [37] 62 1000Hz [11] 62 500Hz EMG [22] 32 1000Hz EMG [38] 32 500Hz [51] 64 100Hz [32] 11 1200Hz EOG and EMG [12] 8 250Hz EMG [27] 19 200Hz [16] 8 500Hz EMG 3.3 E E G D ATA A C Q U I S I T I O N A N D P R O C E S S I N G The EEG signals, typically ranging in the amplitude of microvolts, are captured from active electrodes and amplified through an EEG amplifier and digitized. Initially, the signals were then filtered using the following filter types: i) 4th order Butterworth in 5 papers, ii) 2nd order Butterworth filter in 4 papers, iii) Finite Impulse Response (FIR) in 2 papers, and iv) Zero phase 24th Chebyshev type II in 1 paper. In addition, different classes of the filter were applied: i) highpass filter from 0.1 Hz to 10 Hz was used, based on [ 40 , 16 ] the value of 1 Hz is applied to eliminate DC power supply bias from the signal and according to [ 22 ] value of 10 Hz is applied to remove delta (1-4 Hz), theta (4-8 Hz) band from the signals; ii) bandpass filter from 0.1 Hz to 50 Hz (average) was applied in the majority of the papers, which frequency range is the most suited for gait decoding, allowing to EEG spectrum 3.3. EEG Data Acquisition and Processing 38 Figure 13: Brain Waves Frequencies. Adapted from [13] be divided, as shown in Fig. 13 into delta (1-4 Hz), theta (4-8 Hz), mu (8-12 Hz), beta (12-30 Hz) and gamma (30-50 Hz) bands [37]. In Prasad et al. [ 51 ] the acquired data were filtered in the 0.1–2Hz range using a second-order Butterworth filter, and standardized by channel by subtracting the mean and dividing by the standard deviation (z-score) [ 46 , 45 , 51 ]. Despite taking as many experimental precautions to reduce interference of movement artifacts as possible, EEG data during walking may generally contain movement and other artifacts, either physiological or non-physiological. In particular, gait-locked artifacts overlap in time and frequency with brain activity [21]. As a consequence, when decoding gait activity from EEG signals special care should be taken to avoid exploiting task-related artifacts instead of neural correlates to decode the task itself. To minimize this risk, several preprocessing procedures have been proposed in the literature to reject movement artifacts, most of them based on independent component analysis (ICA) [ 24 ]. ICA was employed to remove the independent components (IC) (Figure 14) representing artifacts by visual inspection [ 38 ] to project the data from the scalp channels domain to the IC domain. A custom 50 Hz comb notch filter with no real poles has been used to remove the power line interference [39]. Bad channels, indicated as a standard deviation greater than 1000 µV or kurtosis of more than five standard deviations from the mean can be rejected [ 47 ], in another hand an automatic rejection of bad channels can also be performed on EEGLAB toolbox in Matlab [ 3 ] based on the probability and Kurtosis statistics of the distribution of the entire signal in each channel using a threshold of 90% [39]. After that, a common average reference (CAR) spatial filtering on the remaining signals is applied to improve the signal-to-noise ratio. A CAR filter is applied to the chunk of EEG data inside each window and the band (1-8 Hz) was extracted with a 4th-order zero-lag Butterworth filter [ 40 ]. For every sample time, CAR subtracts the mean value of all electrodes, which minimizes the uncorrelated random noise with a zero mean through the averaging process [ 25 ]. Epochs containing highamplitude (above 100 µV) and irregular artifacts were manually removed from the data by visual inspection [39,40], however, epochs were rejected if they exceeded ˘150 µV [23]. 3.3. EEG Data Acquisition and Processing 39 Figure 14: Examples of artifact and brain ICs.Taken from [15] An artifact subspace reconstruction (ASR) was applied to remove high amplitude artifacts (e.g. eye blinks, muscle burst) [ 45 ]. ASR applies principal component analysis to the EEG data in sliding windows and identifies channels that significantly deviate from the baseline data containing minimal movement artifact [ 47 ]. Prasad et al. [51] compared classification accuracies with and without ASR, to assess the potential effects of motion artifacts. Next, Reliable Independent Component Analysis (RELICA) was applied to ensure that classification performance was not affected by movement-related artifacts. The extracted ICs were then clustered across RELICA repetitions for each subject according to their mutual similarity into a number of clusters equal to the number of EEG channels. Within the RELICA framework, a quality index can be associated with each IC based on the compactness of the cluster to which it was associated [ 39 ]. The ICA weights were applied to the EEG signals coming from Step I, projecting the data into the domain of the independent component. Components that belong to stereotypical artifacts (e.g. neck muscles, eye movement) were rejected by back projecting the EEG signals to the original domain using only the components related to brain activity [40]. A summary of some procedures discussed above can be seen in Figure 15, where single-subject EEG data were first preprocessed (green box) by going through two EEG preprocessing stages, namely Step I (cyan box) and Step II (orange box). The first is more conservative and includes line noise, bad channels, and noisy epochs removal. The second applies a more aggressive rejection of artifacts to maximize the reliability of extracted ICs (blue box). ICs are applied to data processing according to the first preprocessing step. Studies from the literature have identified neural correlates of gait in different frequency bands. The most common approach is based on Event-Related Desynchronization (ERD)/Event-Related Synchronization (ERS), which was calculated by normalizing the power in the frequency of interest from the active and passive walking by the corresponding baseline condition [ 11 ] in mu (8-13 Hz) and beta (1420 Hz) known as sensory-motor rhythms [39]. 3.3. EEG Data Acquisition and Processing 40 Figure 15: Schematic workflow of the offline EEG processing and classification.Taken from [39] 3.4. Outcomes 41 Other works exploited Movement Related Cortical Potentials (MRCP). An MRCP comprises two main components: a readiness potential (RP) and a movement monitoring potential (MMP). The RP is a negative cortical potential, which begins approximately 1 2 s before the onset of a voluntary movement activated over the presupplementary motor area (pre-SMA) or the contralateral primary motor cortex (M1). An MMP is a slow positive deflection associated with the outcome of the motor process after the execution of the movement intention. Therefore, owing to its characteristics such as potential spontaneity and early detection of user intentions, an MRCP decodes the process of movement preparation/execution based on a single trial basis [ 22 ]. in delta (<3 Hz) or theta (4–8 Hz) [ 39 ]. The MRCP signals were extracted from the EEG signals recorded during the Motor Execution (ME). The EEG signals were then high-pass filtered at 0.05 Hz (2nd order non-causal Butterworth filter). The notch filter frequency rate was defined at 50 Hz to filter out the electrical noises. Next, the EEG signals were down-sampled from 1200 to 250 Hz [32]. Besides the procedures mentioned above, two papers included other methods in their real-time operations: (1) Ravindran et al. [ 43 ] used an H-infinity algorithm to specifically remove eye blinks, eye motions, amplitude drifts, and recording biases simultaneously. The parameters of the H-infinity algorithms were kept the same as the real-time decoding. Peripheral channels were removed as they typically contain many artifactual components, (2) whereas Nakagame et al. [ 46 ] using data recorded from EOG channels, applied H-infinity filter to remove ocular artifacts and signal drifts which may be the major source ofEEG contamination. EOG and peripheral channels were then removed resulting in 50 channels being retained for designing and validating the Unscented Kalman Filter (UKF) decoder [ 46 ] [ 43 ]. An UKF was implemented as a neural decoder and its parameters were updated during BCI operations by using a closed-loop decoder adaptation (CLDA) to improve the performance in real-time [47]. 3.4 OUTCOMES The results of the majority of papers that used Machine Learning and Deep Learning approaches to train and validate the models to decode EEG signals to classify the types of ADLs will be presented in the next sections with their respective results. The table below summarizes the percentage of data split and the models’ accuracy obtained from the literature review. Table 3: Dataset Split (Train, Validation, and Test) & Accuracy Article Train Validation Test Accuracy [43] 80% 20% 10% 90% [39] 80% 20% 10% 90% [40] 60% 15% 25% 80% [22] 50% shared with train 50% 86% multiple channels and 81% Cz Channel [46] 4 segment of data 1 segment of data 43% [32] 14 trials 1 trial 82.73% [11] 90% 10% 89.9% [38] 7 trials 1 trial 83.4% 3.4. Outcomes 42 [37] CV = 77.8% [12] 75 25 83.3% [27] 70 30 86% [16] 85.65% 3.4.1 Gait Deconding from EEG with LSTM Chisari et al. [ 39 ] adopted an extensive offline preprocessing stage as shown in Fig. 15 to guarantee that the optimization of network hyper-parameters, as well as model performance, were not, or minimally, influenced by artifacts time-locked with stepping frequency. Regarding cortical correlates of gait, different frequency bands of the EEG spectrum ( δ band (1-4 Hz), δ and θ bands (1-8 Hz), from δto µand low βbands (1-16 Hz), only µand low βbands (816 Hz), high βband (24-40 Hz)) were extracted as neurophysiological features to decode walking activity. All the bands including low frequencies (1-3, 18, 1-16 Hz) are characterized by the presence of a MRCP prior to each swing phase. In contrast, the ERDS ranges (8-16, 24-40 Hz) present a desynchronization—smaller absolute signal amplitude—during the swing phase with respect to the stance phase [39]. During the offline evaluation, the effect of LSTM hyper-parameters selection we examined on the decoding performance. Performance was evaluated in terms of accuracy and cross-entropy loss of the network trained on 60% of the whole dataset (training set) in the 1-16 Hz frequency band, since it is the widest band considered in this study, and used to predict 15% of the whole dataset (validation set). To evaluate the performance with different frequency bands, we considered a network with 2 LSTM layers, 250 LSTM units in the first layer and 100 LSTM units in the second layer, trained with constant learning rate of 10 ´ 3 (none learning rate schedule), achieving on average 92.8˘3.1% of accuracy and 0.14˘0.12 of loss in validation [39]. Thus, the three classifiers were trained for each subject with the EEG signals filtered in the 1-8 Hz frequency band, and their performances were tested on the 25% of the completely unseen data in the test set. Chisari et al. [ 39 ] proposed in their study a deep learning-based classifier for decoding of gait events from scalp EEG signals, which their method relies on the memory capabilities of a specific type of recurrent network, namely LSTM, to learn time dependencies within the data. Indeed, the brain activity generated prior to the swing phase of each leg overlaps in time with the stance phase of the contro-lateral leg (Wagner et al 2014). This fact may prevent clear discrimination of the dynamics of each leg, as shown by the ineffective performance of memory-less classifiers in the decoding of right and left gait events separately with respect to the decoding of the two legs together. 3.4.2 Gait Intention Using Spatio-Spectral CNN This study was conducted entirely online. The accuracy is the average of all test sets from k-fold cross-validation. The subaverage is the average accuracy of each subject group and the total average is the average of all subjects. The classification accuracy using spatio-spectral CNN model was 83.4% on gait/stand state recognition. In another 3.4. Outcomes 43 hand, the accuracy for the gait and stand intention recognition reached 77.3% and 77.7%. The findings suggest that selecting the data’s characteristics is critical while building a CNN architecture. Park and Park [ 38 ] used only 19 EEG channels from the participants, which was insufficient to reflect spatial features. A deep learning model with improved spatial and temporal features of EEG data should increase performance. 3.4.3 Real-Time EGG-based BCI The paper results analyzed below were performed using online preprocessing and offline decoding combination. Nakagome et al. [ 43 ] aimed to improve the decoding accuracy and robustness for the lower limb decoding using EEG from algorithm perspectives. Accurate lower limb decoding is important for controlling exoskeletons and neuroprosthetics for better usability and systems. Unscented Kalman Filter UKF showed its superior performance in early convergence with a smaller number of samples and when evaluated from the r-value perspective. On the other hand, UKF showed its vulnerability when evaluated from the R2 score and also when a channel is perturbed. UKF decoders tend to perform better with fewer samples, but with larger tap sizes, other algorithms may outperform UKF decoders. This might be one of the baselines used to determine the number of samples to be utilized in real-time decoding. It was not previously known how sampling frequency affects performance. This study looked at this issue using a sample-by-sample decoding approach and discovered that performance may be improved. Although data can be recorded at the highest sampling frequency, the delta band passed features can technically reduce the sample size to 20Hz if there is a sufficient frequency range for reconstruction. With this approach, [ 43 ] could technically also increase the future prediction time from 1 ms (when 100 Hz) to 5 ms (in 20 Hz) with the same decoding scheme. We also showed that the performance could actually improve. In particular, Luu et al. [ 32 ] found sustained α / µ suppression in the Posterior Parietal Cortex (PPC), and Inferior Parietal Lobe (IPL) regions and significant decreases (ERD) in the β band, indicating increased cortical involvement in the walking task. Besides, the results also revealed a substantial increase in cortical activity in the low frequency ( ∆ bands) in the Anterior Cingulate Cortex (ACC) region, demonstrating the potential benefits of using closed BCI, which employs cortical networks involved in error monitoring and motor learning. In addition, β suppression was observed in the ACC,PPC, and IPL, as well as low γ in the and superior temporal gyrus STG. These findings suggest that the closed-loop BCI system encourages spontaneous control of human gait. The system may also help to better understand cortical dynamics during walking with a closed-loop BCI system. 3.4.4 Decoding Sensorimotor Rhythms Classification performances differentiating walking from baseline for both healthy participants and stroke patients were above 93% and 89%, respectively [ 11 ]. The classifier weights revealed that the key brain signals contributing to this performance were ERD in the mu rhythm, which was more bilaterally distributed, and ERD in the beta and low gamma bands, which were more centro-medially situated, as previously described. In comparison to 4.4. Outcomes 50 4.3.2 Decoding perturbations events Since the development of BCI is one of the currently growing approaches to fall prevention [ 30 ], the second major goal of this dissertation is to classify slip-like perturbations events during walking on the treadmill so that can contribute to the development of a device for fall prevention. A framework script in Python developed by BiRDLab was customized to work with EEG data collected from experimental trials. As previously mentioned, the classification for loss of balance events was based on the quantitative outputs from the previous addressed data analysis. Before feeding the framework with the data, it is necessary to label the segmented data and to establish strategies to split the data, to compare different experimental setups, such as: i) same subject with different speed, ii) same speed with different subjects, and iii) all subjects and all speeds. The preceding described methods are ways to lessen overfitting on a dataset because of its size and the dearth of data on perturbations. Additionally, regarding this section, decoding brain signals using AI-based models to prevent falls due to slipinduced perturbations will be presented in Chapter 6, which purpose is to perform comparative analysis and identify the best hyperparameters and the best fit artificial neural network architecture considering the possibility of decoding brain signals from EEG data, besides the model has good computer performance because perhaps it will contribute for a real-time solution. 4.4 OUTCOMES The aging of the world’s population leads to an increase in the prevalence of neurological illnesses such as dementia, Parkinson’s disease, and cerebrovascular accidents. In addition to other implications, the aforementioned disorders cause people to be less mobile, increasing the likelihood of falls. Aside from the economic ramifications for the world’s healthcare systems, these incidents result in post-fall injuries. For all these reasons and considering the need to develop alternative strategies for fall prevention, this dissertation intends, based on a brain signal analysis, to gather a set of quantitative information, considering EEG data, for training the AI-Based models to identify the slip-like perturbation with the purpose of get the natural human response to these events classifying as perturbation and non-perturbation. Two major phases were selected and presented in this Chapter: (i) slip-induced experimental data analysis which design is based on good practices to collect EEG data and preprocessing which is applied to remove the artifacts and select relevant features, as mentioned on previous Chapter, allowing to detect loss of balance during walking on a treadmill will be addressed in Chapter 5; and ii) decoding perturbations events for fall prevention robotic devices which AI framework, developed by BiRDLab team, was adapted from this project, and selection of the deep learning architecture was based on the literature review presented in Chapter 1 and 2, where DL-based models and their computer performance will be evaluated and discussed in Chapter 6. 4.5. Discussion 51 4.5 DISCUSSION This dissertation aims to gather a set of information useful to the development of wearable robotic devices to fall prevention, based on a comprehensive study of the brain signal response to slip perturbations. In this chapter, an architecture was idealized based on a literature review of collected and processed data for ADLs and its gaps related to running experimental protocols for loss of balance. The literature reviews presented in Chapter 3, in addition to allowing the understanding and variables of interest to be analyzed during the study of experimental data from slip-like perturbations, also allowed for the identification of some gaps in the literature, which will be filled as much as possible during the analysis of experimental data collected at BiRDLab. The experimental data analysis’ outcome will be a fundamental step in identifying the loss of balance signature in EEG the signals. Additionally, the EEG data analysis performed will provide a more comprehensive understanding of the brain signals response to slip-like perturbations considering different subjects and speeds of gait, thus filling the gaps found in the review of the existing literature [ 35 , 10 , 31 ], which results for loss of balance were obtained from experiments where the subjects stood on a platform and postural reactions were applied in two horizontal translations (forward and backward) and two vertical translation (toes up and toes down). This outcome will be achieved in Chapter 5. Finally, the classification for perturbation and non-perturbation using different Artificial Neural Networks Architectures for fall prevention was also a gap determined after the analysis of the existing literature [ 10 , 31 ], which the present dissertation intends to overcome, besides to compare the AI-based models in the face to computer performance to evaluate their effectiveness, which outcomes will be discussed in Chapter 6. In this Chapter research hypotheses that were considered to support the investigation described were also addressed as well as the outcomes expected to achieve in this dissertation. 5 INDUCED LOSS BALANCE DATA ANALYSIS The purpose of this chapter is to present a comprehensive analysis of brain response to slip-like perturbations considering EEG data in different walking speeds and considering variable slip perturbations. This chapter will also address the experimental protocol used for data collection as well as how the data were processed. The primary output of this chapter is quantitative data that may be used to help determine the target requirements for balance loss decoding by analyzing cerebral cortex signals of the response to slips. The methods to set up and run an experimental protocol, data pre-processing, and recognition of brain reactions caused by balance disturbances as well as the data segmentation and data labeling will be addressed in Section 5.2. The main results related to Perturbation Evoked Potential will be analyzed in Section 5.3. In the end, Section 5.4 presents a discussion regarding results from the pipeline deployed in the EEGLAB framework (Matlab). 5.1 INTRODUCTORY INSIGHT As previously mentioned, the development of balance loss decoding requires, as an initial step the analysis of the natural brain signals response of humans to slip events, in order to gather information capable of determining the characteristics of actuation capable of mimicking this response. Therefore, an experimental protocol was designed and conducted by BiRDLab team, to collect brain signals from the cerebral cortex and study these signals in order to identify and obtain the variables of interest in slip-like perturbation responses analysis. In this analysis, a pipeline was deployed in EEGLAB, whose goal is the data cleansing and normalization by applying pre-processing techniques to extract relevant features and evaluating their importance for the effectiveness of a successful brain signal response after a slip perturbation based on literature review (Chapter 3) and later be used in the decoding and classification to predict slip-like perturbations, which one will be presented on next Chapter. 5.2 METHODS The research covered in this chapter sought to understand the brain signals’ reactions to slip-like perturbations by collecting EEG data and processing it through data cleaving and normalization processes. The methods applied to obtain a comprehensive analysis will be addressed in this Section sorted by the following topics: i) the 52 5.2. Methods 53 experimental setup to data acquisition through EEG system inside a controlled environment (Subsection 5.2.1); ii) pre-processing the raw data applying filter and techniques to improve signal-to-noise ratio (Subsection 5.2.2). 5.2.1 Experimental Procedures In order to study the brain signals in response to slip-like perturbations, experiments were conducted at BiRDLab, University of Minho. Initially, in order to proceed with the experiment, the subjects had to present: i) healthy locomotion; ii) total postural balance; iii) over 18 years old; and iv) body mass less than 135 kg. Individuals who: i) had any disease (mainly neurological) or deficit that affected locomotion were excluded; and ii) have recently undergone surgical procedures that affect mobility. Based on the prerequisites three healthy male participants (age: 26 ˘ 5; height: 1.80m ˘ 3.85cm; weight: 71 ˘ 3.3 kg) were selected for the experimental protocol. All the subjects that participated in these protocols presented the right dominance. All participants provided written informed consent and voluntarily accepted to participate in the experimental trials. Each participant performed the qualitative assessment of the preferred foot by completing the Waterloo Footedness Questionnaire. To provide data to better understand the brain signals response due to balance loss events and record the instant of the slip perturbations, two sensor systems were used in this protocol. Wireless EEG Headset (g.NAUTILUS PRO), 16 channels (active wet electrodes) to record brain activity in a controlled environment; and IMUs from Xsens MVN Awinda which is composed of 17 IMUs, however, only one was used to record acceleration caused when the rope is pulled to mimic a slip-like perturbation event when the subject is walking on the treadmill. The data from the brain activities were collected through EEG system at 500Hz, and 1 IMU at 100Hz was used to record the acceleration caused by the action of pulling the rope. Subjects used a safety harness device during the experiments to avoid falls in the event of an irreversible Loss of Balance (LOB). This method consisted of a vest that was rope-attached to a structure in the ceiling. The length of the rope was modified to ensure that there was at least 15 cm between the knees and the treadmill belt. This step was carried out by instructing individuals to elevate their feet, resulting in the application of complete body weight to the harness system. To achieve synchronized data gathering from sensors, Sync Lab Desktop for Windows OS, created by BiRDLab, was utilized to synchronously start and stop data collection from previously listed sensors. The Desktop program sends electrical trigger signals. The former is delivered via Syncbox, a previously designed hardware interface that links to the Xsens and EEG devices via direct USB connection. The following diagram describes the experimental setup that was used to obtain data. 5.2. Methods 54 1) EEG headset. 2) EGG base station to capture the signals from the headset and send them to SyncLab. (3) Xsens IMU to register the acceleration regarding pulling rope. (4) Xsens base station, which establishes communication between the Xsens IMU and Synclab. (5) Sync Box to synchronize the base stations. (6) PC with SyncLab to collect and process the sync data; Figure 17: Experimental setup for balance loss data collection. Subjects were instructed to manage unexpected slip-like disturbances while walking on the treadmill, and they were not informed about the protocol in order to avoid any prior bias in their brain activity response. A concealed rope was hooked up to the subject’s ankle at heel striking time, and a second person manually marked the perturbations to be utilized later to adjust the registers with Xsens IMU data. Because the rope was constantly linked to one of the subjects’ feet during all of the trials, the participants did not know whether a disturbance would occur or not. The experimental protocol was applied to volunteers in two (2) different treadmill velocities (1.6 km/h and 2.5 km/h). The volunteers had to wear the EEG cap while walking on the treadmill, and the protocol was performed for each speed per subject. •1 treadmill inclination (flat – 0º) •time per trial: 20 - 30 minutes •2 situations (perturbation and non-perturbation) According to a literate review of Chapter 3, the volunteers had to walk on the treadmill for 20 to 30 minutes for each speed. For each experimental session (1.6 km/h and 2.5 km/h) where perturbations were delivered, the operator applied a minimum of 30 perturbations at random moments during walking. Both phases (Nonperturbation and perturbation) occurred intercalated within a duration period of the 30s to 60s. 5.2. Methods 55 5.2.2 Data Pre-processing The events (slip-like perturbations) were automatically recorded by Xsens whose magnitude of the acceleration vector was used to indicate the occurrence of provoked perturbation for both speeds. In order to detect any spikes that represent wrong situations such as dropping the rope, or a rope hitting the treadmill, as a solution, it was suggested to use the synclab to record the occurrence of events manually. To synchronize these events, a script was created in Matlab to allow this adjustment and the validation of the records before starting the pre-processing of the data. The result of 1 trial is shown in Fig.18. After the data acquisition, the pre-processing of EEG data was carried out using custom-made scripts (Appendix D) in EEGLAB toolbox [ 3 ] running in MATLAB. The EEG dataset has 19 rows (row 1: time (2 ms for each register); rows 2 - 17:EEG channels; row 18: null; row 19: events related to perturbation), and each column represents the values from brain signals obtained by the electrodes, thus, the number of columns is directly related to the duration of the trial. Firstly, the irrelevant attributes were deleted (rows 1 and 18), and the registers in row 19 were imported as events. Afterward, as described before, it is necessary to check the time recorded in the dataset with the time recorded by Xsens and then adjust the time when the perturbation event occurred. It is important to note that the EEG system frequency is 500Hz and Xsens is 100Hz, and this may affect the register of the event time in milliseconds. Furthermore, the information about channel locations needs to be imported to be used when plotting the graphics and analyzing the channels’ significance. Figure 18: Comparison between the manual label and Xsens IMU acceleration vector magnitude to perform label correction (1 Trial - 40 Perturbations) As EEG signal is highly prone to noise and also is non-stationary, EEG recordings can be understood as a mixture of independent cerebral and non-cerebral sources called artifacts (e.g., ocular, muscular, etc.). These 5.2. Methods 56 artifacts are considered disturbances in brain electrical signals, thus a pre-processing procedure to remove them from the recorded EEGs must be implemented to allow the extraction of valuable event-related information. The pre-processing steps used to process EEG signal are summarized in Fig.19 and the images of the results of the signals obtained after each step of the pre-processing are shown in Appendix A. Figure 19: Pre-processing Framework Pipeline Initially, based on papers related to ADLs, in order to cut off the low-frequency drift the data were high-pass filtered at 0.3 Hz using a 4th-order Butterworth filter, and line noise was attenuated at 50 Hz by using a notch filter. However, analyzing the data after these steps, it was not possible to obtain PEP patterns. Thus, a band-pass filter between 4Hz and 30Hz was applied to the data removing low frequencies (applied in motor imagery or stable movements) and frequencies above 30Hz avoiding the muscular artifacts that belong to the natural walking and postural recovery after a balance loss [35]. The processed EEG signal was re-referenced using a common average as a part of the pre-processing process. CAR is commonly used in EEG, where it is necessary to identify small signal sources in very noisy recordings [ 1 ]. The re-referenced EEG signal was then ready for ICA operation, in which independent components representing 5.3. Outcomes 57 independent EEG source signals were extracted using Infomax [ 6 ]. According to papers [ 31 , 49 , 35 , 48 , 10 ] components centered around CZ and distributed over frontal, central, and parietal areas (FCz, C3, C4, and CPz), have a strong relation to the detection of balance loss. As shown in Appendix AFig.26 the EEG headset does not have frontocentral and centroparietal electrodes, and due to the fact of electrodes are fixed, it was decided to test four (4) approaches: i) use all channels; ii) remove all channels except Fz, C3, Cz, C4, Pz, which ones are closer of central area, iii) Fz and Cz, because during some trials was observed the Fz channel obtained similar responses related to provoked perturbation as presented in literature [31,49,35,48,10], and iv Cz was used. To study the event-related EEG dynamics of continuously recorded data, the data epochs time-locked to events of interest were extracted (for example, data epochs time-locked to onsets of one class of experimental stimuli). Thus, the EEG signals were segmented into epochs time-locked to perturbation onset (epoch range: -2s to +1s), wherein 0 ms is included in the window, corresponding to the first sample at the perturbation onset. In addition, due to the poor quality of the data obtained, one subject was discarded. Thus, two remaining male subjects were kept to analyze data. 5.3 OUTCOMES During the occurrence of a slip-like perturbation, the grand average shows (Fig. 20) a strong negative shift (N1 component of PEPs) starting shortly after perturbation onset and peaking 138 ˘ 3 ms after perturbation onset with an average amplitude of –3.77 ˘ 1.72 µV for treadmill speed = 1.6km/h, and peaked on average 150 ˘ 5 ms after perturbation onset with an amplitude of –5.16 ˘ 2.08 µV for speed = 2.5km/h. The negative peak is followed by a strong positive rebound, the P2 component of PEPs, that peaked on average 226 ˘ 21 ms after perturbation onset with an amplitude of 2.25 ˘ 1.61 µV for treadmill speed = 1.6km/h and peaking 216 ˘ 5 ms after perturbation onset with an average amplitude of 3.76 ˘2.72 µV for speed = 2.5km/h. Furthermore, the P1 component was also detectable in the grand average. P1 started shortly after perturbation and peaking 74 ˘ 4ms after perturbation onset with an average amplitude of 2.35 ˘ 0.7µV for treadmill speed = 1.6km/h and peaked on average 88 ˘ 3 ms after perturbation onset with an amplitude of 2.25 ˘ 0.7 µV for speed = 2.5km/h. On the other hand, the negative peak (N2) which succeeds the later positive component (P2) was not as clearly detected as other components, however, it was still possible to analyze its grand average value and it showed peaking at 308 ˘ 4 ms after perturbation onset with an average amplitude of –0.94 ˘ 0.6 µV for treadmill speed = 1.6km/h, and peaked on average 318 ˘ 17 ms after perturbation onset with an amplitude of –1.3 ˘ 1.39 µV for speed = 2.5km/h. At perturbation onset (0 ms) on average, no visible derivation from the baseline is found. The graphics of grand mean PEP latencies and PEP magnitudes for a representative remaining electrodes (Fz, C3, C4 e Pz) are shown in Appendix Afrom Fig.32 to Fig. 39. One observation is Fz channel produces strong PEP responses as compared to parietal and central-left and central-right cortices. Moreover, it was analyzed during the development of this work that C3 and C4 responses are undetectable in subject 2 at both speeds. 5.3. Outcomes 58 (a) Grand Average PEP for Speed = 1.6km/h (b) Grand Average PEP for Speed = 2.5km/h Figure 20: Grand average of perturbation trials at channel Cz. Black lines show the average for each speed. (a): Average of 70 trials for treadmill speed = 1.6km/h; (b): Average of 69 trials for treadmill speed = 2.5km/h. Time = 0 represents the perturbation onset As depicted in Fig. 21, the grand average on a topographical level for time-points: t = 60-80 ms (P1 peak); t = 130-160 ms (N1 peak) and t = 300-320 ms (P2 peak) and t = 210-230 ms (N2 peak). The peak of the N1 component was centered around Cz and distributed for speeds 1.6km/h and 2.5km/h. Also, the peak of all PEP 5.4. Discussion 59 components is centered around Cz, although, for P2 and N2 peak is highlighted the presence of Pz channel as well for speed = 2.5km/h. Otherwise, the peak magnitude in N1, P2, and N2 components over the C4 channel has opposite directions (signal) in comparison to the Cz channel. (a) P1 Component (Speed 1.6km/h and 2.5km/h) (b) N1 Component (Speed 1.6km/h and 2.5km/h) (c) P2 Component (Speed 1.6km/h and 2.5km/h) (d) N2 Component (Speed 1.6km/h and 2.5km/h) Figure 21: Topographical plots show the scalp distribution of the grand average in PEP responses 5.4 DISCUSSION We were able to elicit PEPs using the paradigm and experimental setup described in the method section of this work. Our findings were in agreement with [ 48 , 35 , 10 , 6 ]. We observed an average amplitude of ´ 3.77 µV (speed = 1.6km/h) and -5.16 µV (speed = 2.5km/h) for the N1 component of PEPs elicited during the two tested protocols. On average, the N1 component peaked with a latency of 138 ˘ 3 ms and 150 ˘ 5 ms relative to perturbation onset. The N1 event was spread around frontal, and central areas with a maximal peak at Cz. These findings related to latency are in accordance with results published in previous studies, however, related to amplitude they are in discordance. According to Ditz et al. [ 10 ] study, they found higher amplitudes ( ´ 28.3 ˘ 14.5 µV), and Ozdemir et al. [31] found amplitudes of ´16 ˘3 µV (Young) and ´11 ˘2 µV (Elderly). The late components of PEPs (P2 and N2), can be mainly found from 200 ms and 300 ms after perturbation onset [ 48 ]. As shown in Fig. 20, we were able, through grand average, to identify those components, however, for subject 1 (green line) the component N2 cannot be distinguishable. On average, the P2 peaked at 226 ˘ 21 ms (speed = 1.6km/h) after perturbation onset with an amplitude of 2.25 ˘ 1.61 µV and peaked at 216 ˘ 5 ms (speed = 2.5km/h) after perturbation onset with an amplitude of 3.76 ˘ 2.72 µV This finding agrees with the result regarding waveform and latency of previously conducted studies involving postural seated perturbations [ 10 ] and balance platform which introduces perturbation [ 35 , 6 ]. In contrast, Ditz et al. study found higher amplitudes in P2 6.2. Methods 66 Table 5: Specifications for the use of the Deep Learning models Hyperparameters Value Epoch Number 100 Batch Size 16, 32, 64 and 128 Hidden Layers 128 Dropout 0.5 Optimizer Adam Learning Rate 0.0005 Loss Function Cross Entropy Loss 6.2.3 Model Building and Evaluation The first step taken for the model’s building and evaluation in the comparative analysis was to select the proper dataset and split it into a train set (training the model), a validation set (evaluate the model during the training process), and test set (evaluate the final model accuracy before deployment). For this division, due to the amount of data and samples obtained during the development of this project, each dataset used in this study has only two distinct subjects, subject 1 with 30 disturbances recorded at a speed of 1.6km/h and 2.5 km/h, and subject 2 with 40 disturbances recorded at a speed of 1.6km/h and 39 disturbances at a speed of 2.5km/h. Furthermore, as no cross-validation techniques were applied, the created feature windows were divided by the following approaches: •Approach 1: Speed of 1.6km/h (Total samples = 70): –40 samples of subject 2 and 2 samples of subject 1 were used as training data (total of 60%) –14 samples of subject 1 were used as validation data (total of 20%) –14 samples of subject 1 were used as test data (total of 20%) •Approach 2: Speed of 2.5m/h (Total samples = 69): –40 samples of subject 2 and 1 sample of subject 1 were used as training data (total of 60%) –14 samples of subject 1 were used as validation data (total of 20%) –14 samples of subject 1 were used as test data (total of 20%) •Approach 3: Subject 1 (Total samples = 60): –42 samples were randomly selected as training data (total of 70%) –9 samples were randomly selected from remaining data as validation data (total of 15%) –The last 9 samples were used as test data (total of 15%) •Approach 4: Subject 4 (Total samples = 79): –55 samples were randomly selected as training data (total of 70%) –12 samples were randomly selected from the remaining data as validation data (total of 15%) 6.2. Methods 67 –The last 12 samples were used as test data (total of 15%) •Approach 5: Subject 4 (Total samples = 139): –97 samples were randomly selected as training data (total of 70%) –21 samples were randomly selected from remaining data as validation data (total of 15%) –The last 21 samples were used as test data (total of 15%) These splitting methodologies are not ideal, however, it is expected that, in approaches 1 and 2, all windows of the certain subjects are placed entirely either in the training data group or in the test data group, causing no activity to be only trained or only tested. In addition, with the separation of data by subject, although there is a probability of overfitting for the training data, we can still independently test the generalization of the model (or increase confidence in the obtained generalization value). In approaches 3 and 4, as the number of samples for speeds 1.6km/h and 2.5km/h are practically the same, and in the combined dataset all subsets of data are aggregated, the dataset was divided randomly. These same methodologies were also applied to the dataset that has independent components. In order to understand deep-learning models the classification of slip-like perturbations was carried out using (4) four Neural Networks architectures mentioned in Section 6.2.2 and presented in Fig. 23. Furthermore, to analyze which would be the most suitable models for the classification of slip-like perturbations, we performed several training and validation using different combinations between 5 datasets (Table 4) and DL-based models. In addition, for each dataset was used 4 subsets of features as input to train and evaluate these deep learning models (a subset of features is summarized in the Table below). The performance of neural networks with the best set of features was analyzed and from the performance, results obtained an initial analysis was made for the best classification model for balance loss recognition. Table 6: Subset of Features per Dataset and Deep Learning-Based Model Channels Subsets Participants Treadmill Speed Channels DL Model S1 All GRU 1.6km/h Fz, C3, Cz, C4, Pz CNN-LSTM 2.5km/h Fz, Cz CNN Cz LSTM S2 All GRU 1.6km/h Fz, C3, Cz, C4, Pz CNN-LSTM 2.5km/h Fz, Cz CNN Cz LSTM 1.6km/h All GRU S1 Fz, C3, Cz, C4, Pz CNN-LSTM S2 Fz, Cz CNN Cz LSTM 2.5km/h All GRU S1 Fz, C3, Cz, C4, Pz CNN-LSTM 6.2. Methods 68 S2 Fz, Cz CNN Cz LSTM All All All GRU Fz, C3, Cz, C4, Pz CNN-LSTM Fz, Cz CNN Cz LSTM Independent Components Subsets Participants Treadmill Speed ICs DL Model S1 All GRU 1.6km/h CNN-LSTM 2.5km/h CNN LSTM S2 All GRU 1.6km/h CNN-LSTM 2.5km/h CNN LSTM 1.6km/h All GRU S1 CNN-LSTM S2 CNN LSTM 2.5km/h All GRU S1 CNN-LSTM S2 CNN LSTM All All All GRU CNN-LSTM CNN LSTM 6.2.4 Model Evaluation Metrics Evaluating a model is the most important task in building an effective AI-based model. Its metrics are used to measure the quality of the model. As a binary classification task, the models used during the development of this work can only produce two results (perturbation or non-perturbation). Because supervised deep learning approach was applied, the models follow the following steps: i) fit a model on training data; ii) test the model on testing data. Once the model’s predictions are obtained from the test data, a comparison between the test data and the true values (the correct labels) is performed. To measure the effectiveness of the classification model, the generated classification outcomes may be compared with the actual values of the provided observation. To extract the metrics a Confusion Matrix is utilized to describe these results. The confusion matrix is a table that summarizes the classification model’s performance in predicting instances from different classes i.e., it is an organized way of mapping the predictions to the original classes to which the data belong. The confusion matrix has two axes: one for the expected label and one for the actual label. 6.2. Methods 69 Figure 24: Confusion Matrix Example for Binary Classes. Taken from [44] • True Positives (TP): the actual value is Positive (P) and predicted is also Positive i.e., the model correctly predicts the positive class. • True negatives (TN): when the actual value is Negative (N) and prediction is also Negative i.e., the model correctly predicts the negative class. • False positives (FP): When the actual is negative but prediction is Positive. Also known as the Type 1 error i.e., the model gives the wrong prediction of the negative class (predicted-positive, actual-negative). • False negatives (FN): When the actual is Positive but the prediction is Negative. Also known as the Type 2 error i.e., the model wrongly predicts the positive class (predicted-negative, actual-positive). The confusion matrix delivers insight not only into the errors being made by the classifier but more importantly the types of errors that are being made. In addition, the confusion matrix not only allows the calculation of the accuracy of a classifier, be it the global or the class-wise accuracy, but also helps compute other important metrics that evaluate ML and DL models. In general, we can get the following quantitative evaluation metrics from this binary class confusion matrix: 6.2.4.1 Accuracy Accuracy is calculated as the number of all correct predictions divided by the total number of the dataset. The best accuracy is 1.0, whereas the worst is 0. Based on equation 1accuracy is calculated as the total number of two correct predictions (TP +TN) divided by the total number of a dataset (P+N). Accuracy “TP `TN TP `TN `FP `FN (1) Accuracy is widely used to evaluate model performance; nevertheless, there are a few drawbacks considered before relying on accuracy excessively. One of these disadvantages is that the model must categorize observations based on an imbalanced dataset in which one class, either true or false, is more prevalent than the other. 6.2. Methods 70 Error costs of positives and negatives are usually different. For instance, one wants to avoid false negatives more than false positives or vice versa. Other basic measures, such as sensitivity and specificity, are more informative than accuracy and error rate in such cases. Figure 25: Illustration of Metrics extracted from Confusion Matrix. Adapted from [36] 6.2.4.2 Recall (Sensitivity) The recall or Sensitivity metric calculates how many of the actual positive instances are able to correctly predict. It is the ratio of true positives to the total number of actual positive values in the data. Recall “TP TP `FN (2) 6.2.4.3 Specificity Specificity is defined as the algorithm’s or model’s ability to predict a genuine negative of each accessible category. It is also known simply as the true negative rate in the literature. Formally, the equation below may be used to compute it. Speci f icity “TN TN `FP (3) 6.3. Results 71 6.2.4.4 Precision Precision metrics inform how correct, or precise, the model’s positive predictions are. In definition, precision is defined as the actual correct prediction divided by the total number of positive predictions as presented in equation 4. Precision “TP TP `FP (4) Precision is an important metric when False Positives are more important than False Negatives 6.2.4.5 F1 Score The F1 score is calculated as a weighted average of precision and recall. As presented in equation 4and 2, there is a false positive and false negative in accuracy and recall, therefore it takes both into account. F1 score is frequently more helpful than accuracy, especially if the class distribution is unequal. Accuracy works best when the cost of false positives and false negatives is comparable. If the cost of false positives and false negatives varies significantly, it is preferable to include both Precision and Recall. F1Score “2ˆ pRecall ˆPrecisionq pRecall `Precisionq(5) 6.2.4.6 Matthews Correlation Coefficient Scientific researchers can use a variety of statistical tools to analyze binary classifications and their confusion matrices, depending on the purpose of the investigation. Accuracy and F1 score based using confusion matrices were (and continue to be) among the most often used metrics in binary classification problems. However, when applied to unbalanced datasets, these statistical techniques might possibly provide overoptimistic inflated outcomes [8]. Instead, the Matthews correlation coefficient (MCC) is a more reliable statistical rate that produces a high score only if the prediction performed well in all four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally to the size of positive and negative elements in the dataset. MCC, in fact, calculates the correlation between the real classes and the predicted labels. (worst value = ´ 1; best value = +1) [9]. MCC “pTP ˆTNq ´ pFP ˆFNq apTP `FPq ˆ pFP `FNq ˆ pTN `FPq ˆ pTN `FNq(6) 6.3 RESULTS In the following subsections, the main results obtained in the evaluation process of the different Deep Learning architectures explained in the previous Section will be exposed in Section 6.3.1. Furthermore, the choice of 6.3. Results 72 the best subset of parameters, considering the recognition of slip-like perturbations based on the evaluation metrics, as well as the analysis regarding runtime to train and validate the model, will be depicted in Section 6.3.2. Furthermore, the comparison of the best model results between the channel subset and ICs subset will be presented in Section 6.3.2. 6.3.1 Deep Learning Architecture As mentioned in Sections 6.2.2 and 6.2.3, four architectures of Deep Neural Networks were applied to decode and classify the slip perturbation events. In order to make a comparative analysis of the performance of the Deep Learning models, the inputs for the training of these architectures are the combination of 3 attributes from datasets presented in Table 6. These combinations resulted in 128 test outcomes for channel subsets and 34 test outcomes for the ICs subset to be analyzed. Table 7: Classification Metrics Results for Channels’ Dataset Speed at 1.6 km/h Model ACC F1 Score MCC Precision Recall Specification CNN 88.10% 86.91% 74.08% 87.58% 87.12% 88.10% CNN-LSTM 86.61% 84.99% 71.40% 90.33% 84.21% 86.61% GRU 84.82% 81.96% 67.38% 86.95% 82.05% 84.82% LSTM 84.82% 83.08% 66.97% 87.66% 82.19% 84.82% Speed at 2.5 km/h Model ACC F1 Score MCC Precision Recall Specification CNN 92.86% 92.26% 85.48% 94.19% 91.37% 92.86% CNN-LSTM 90.48% 88.06% 78.65% 92.43% 87.36% 90.48% GRU 92.14% 91.42% 84.05% 93.74% 90.41% 92.14% LSTM 91.07% 90.16% 80.83% 91.78% 89.39% 91.07% Subject 1 Model ACC F1 Score MCC Precision Recall Specification CNN 95.31% 94.39% 89.68% 96.44% 95.65% 95.31% CNN-LSTM 96.35% 96.03% 92.18% 96.54% 95.66% 96.35% GRU 90.63% 90.13% 81.05% 93.48% 89.67% 90.63% LSTM 88.54% 87.49% 75.20% 59.43% 86.94% 88.54% Subject 2 Model ACC F1 Score MCC Precision Recall Specification CNN 93.18% 91.88% 80.74% 91.49% 92.78% 93.18% CNN-LSTM 92.80% 91.44% 83.46% 91.62% 92.47% 92.80% GRU 95.08% 92.82% 87.08% 93.19% 94.05% 95.08% LSTM 93.94% 91.89% 83.28% 91.85% 92.19% 93.94% Combined Dataset (Subjects and Speeds) Model ACC F1 Score MCC Precision Recall Specification CNN 88.10% 86.48% 73.87% 91.03% 86.86% 88.10% CNN-LSTM 87.14% 85.29% 73.84% 90.08% 85.75% 87.14% GRU 86.19% 84.24% 72.80% 90.33% 83.43% 86.19% 6.3. Results 73 LSTM 85.52% 83.45% 69.58% 88.30% 81.84% 85.52% The final results for channel subsets, obtained for each of the tests performed as well as the graphics of the remaining results regarding the evaluation metrics are shown in Appendix B, from Fig.40 to Fig.89. Through direct observation of metrics results in Table 7, the best performance results in the subsets of features sort of by speed (1.6km/h and 2.5km/h) and CAR filter were obtained for the CNN model in both subsets when used 5 channels (Fz, C3, Cz, C4, Pz) as the input (Speed: 2.6km/h, Accuracy: 92.86%, Recall: 91.37%, Specificity: 92.86%, Precision: 94,19%, F1-Score: 92.26%, MCC: 85.48%). In addition, other tests were performed to evaluate the model performance sort of by subjects 1 and 2 as well, and the best performance results regarding DL-Model were different between the subsets. For Subject 1 the CNN-LSTM model was the best and it used all 16 channels as the input (Accuracy: 96.35%, Recall: 95.65%, Specificity: 96.35%, Precision: 96.54%, F1-Score: 96.03%, MCC: 92.18%). Otherwise, for Subject 2 the GRU model was the best and it also used all 16 channels as the input (Accuracy: 95.08%, Recall: 91.93%, Specificity: 83.22%, Precision: 87.80%, F1-Score: 88.62%, MCC: NaN). Complementary to the tests mentioned previously, an evaluation was performed with combined dataset which includes all speeds and subjects to be used as training input and the best performance result was obtained for CNN model when it used all channels as input (Accuracy: 88.10%, Recall: 86.86%, Specificity: 88.0%, Precision: 91.03%, F1-Score: 86.48%, MCC: NaN). From the tests carried out with the artificial neural networks, besides training the model and running the tests with a different number of channels, as shown in Appendix Bin Figs. 44,54,64,74 and 84, it can be seen that the best results regarding performance evaluation metrics are obtained using the following hyperparameters: a) Batch Size: 16 (Subject 1 and combined data), 32 (2.5km/h), 64 (Subject 1), 128 (1.6km/h); b) Overlap: 50 (2.5km/h and combined data), 80 (1.6km/h, Subject 1 and Subject 2). Table 8: Classification Metrics Results for ICs’ Dataset Speed at 1.6 km/h Model ACC F1 Score MCC Precision Recall Specification CNN 76.79% 75.49% 51.97% 76.45% 75.97% 76.79% CNN-LSTM 79.29% 78.21% 57.37% 79.07% 78.35% 79.29% GRU 80.00% 78.36% 57.80% 80.46% 77.75% 80.00% LSTM 79.46% 76.15% 54.59% 78.38% 76.39% 79.46% Speed at 2.5 km/h Model ACC F1 Score MCC Precision Recall Specification CNN 85.71% 84.38% 69.77% 85.57% 84.86% 85.71% CNN-LSTM 83.63% 82.52% 66.95% 86.14% 83.29% 83.63% GRU 78.27% 75.86% 53.72% 82.81% 75.15% 78.27% LSTM 79.46% 76.15% 55.37% 80.70% 75.20% 79.46% Subject 1 Model ACC F1 Score MCC Precision Recall Specification CNN 88.54% 86.87% 75.28% 90.11% 85.34% 88.54% CNN-LSTM 90.10% 87.60% 75.95% 89.88% 86.69% 90.10% GRU 89.06% 85.01% 71.56% 88.76% 83.85% 89.06% 6.3. Results 74 LSTM 88.54% 87.18% 74.94% 88.75% 86.23% 88.54% Subject 2 Model ACC F1 Score MCC Precision Recall Specification CNN 91.25% 90.40% 83.49% 93.15% 91.67% 91.25% CNN-LSTM 94.79% 94.46% 89.91% 94.70% 95.27% 94.79% GRU 92.71% 91.99% 84.21% 92.86% 91.36% 92.71% LSTM 92.71% 92.05% 84.27% 92.58% 91.70% 92.71% Combined Dataset (Subjects and Speeds) Model ACC F1 Score MCC Precision Recall Specification CNN 88.10% 84.33% 70.01% 84.97% 85.07% 88.10% CNN-LSTM 89.09% 88.56% 77.78% 88.22% 89.57% 89.09% GRU 87.10% 84.93% 72.64% 8.90% 84.10% 87.10% LSTM 88.49% 86.59% 74.84% 89.83% 85.93% 88.49% Regarding ICs subsets, the final results obtained for each of the tests performed as well as the graphics of the remaining results regarding the evaluation metrics are shown in Appendix C, from Fig.90 to Fig.119. Opposite to channel subsets, which have 4 combinations for channels (16 channels, 5 channels, 2 channels, and 1 channel), the ICs subsets were training with 16 independent components only. Through direct observation of the results in Table 8, the best performance results in the subsets of features sort of by speed (1.6km/h and 2.5km/h) and ICA filter were obtained for the GRU model at speed of 1.6km/h as the input (Accuracy: 80.00%, Recall: 77.75%, Specificity: 80.00%, Precision: 80.10%, F1-Score: 78.36%, MCC: 57.80%) and the CNN model at speed of 2.5km/h as the input (Accuracy: 85.71%, Recall: 84.86%, Specificity: 85.71%, Precision: 84.97%, F1-Score: 84.38%, MCC: 69.77%). In addition, other tests were performed to evaluate the model performance sort of by subjects 1 and 2 as well, and the best performance results regarding DL-Model were different between the subsets. For Subject 1 the CNN-LSTM model was the best (Accuracy: 90.10%, Recall: 81.09%, Specificity: 90.10%, Precision: 80.40%, F1-Score: 79.87%, MCC: NaN). For Subject 2 the CNN-LSTM model was the best (Accuracy: 94.79%, Recall: 95.27%, Specificity: 94.79%, Precision: 94.70%, F1-Score: 94.46%, MCC: 89.91%). Complementary to the tests mentioned previously, an evaluation was performed with combined dataset which includes all speeds and subjects to be used as training input and the best performance result was obtained for CNN-LSTM model was the best (Accuracy: 89.09%, Recall: 89.57%, Specificity: 89.09%, Precision: 88.22%, F1-Score: 88.56%, MCC: 77.78%). From the tests carried out with the artificial neural networks, as shown in Appendix Cin Figs. 91,97,103, 109 and 115, it can be seen that the best results regarding performance evaluation metrics are obtained using the following hyperparameters: a) Batch Size: 16 (Subject 1 and 2.5km/h), 32 (Subject 2), 128 (Subject 2 and combined data); b) Overlap: 50 (1.6km/h and 2.5km/h), 80 (Subject 1, Subject 2, and combined data). 6.3.2 Models and Parameters Evaluation Furthermore, to the value of metrics presented above, a comparative analysis of Deep Learning models related to the frequency (number of times) of the model performed above accuracy of 85%, as well as the inputs used and hyperparameters tested, it was possible to obtain the values to evaluate their relevance to decode brain signal to 6.3. Results 75 predict loss of balance provoked by slip-like perturbations. The final results can be observed in AppendixBthrough the graphics Figs. 45,55,65,75 and 85 and AppendixCthrough the graphics Figs. 92,98,104,110 and 116 7 CONCLUSION This dissertation presents the process of developing brain signal decoding in response to slip-like perturbation. The current state-of-art concern that falls among the elderly is a severe danger that can result in death or non-fatal effects, as well as a significant sociological and economic impact on families and society. Furthermore, even if a fall does not result in an injury, people can develop a fear of falling, which is a big issue that affects both their mental and physical health. Slipping appears to be the most common cause of falls. Slip-related falls typically have even worse effects than being held responsible for a major share of hip fractures in the elderly, highlighting the need to address this issue as soon as possible. As a result, effective solutions to the aforementioned problem are vital, and any effort to reduce or avoid a fall can save many lives. As a result, the development of wearable robotic systems for preventing slip-like falls arises as a solution to this problem. The fundamental function of these devices is to identify a loss of balance scenario and generate an actuation capable of restoring a proper biomechanical posture to allow the patient to resume a steady walking stride after a slip-like disturbance. To intervene in this problem, the study of brain signals in response to slip-like perturbations appears to be a common preliminary step of the previous approach. Thus, the first step of the work was developing theoretical knowledge in the fields of EEG signals, BCI applications, and ANN, including the research for the most current studies related to decoding brain signals to ADLs. Furthermore, the extent of the literature included guidance and manuals related to frameworks, which assisted the understanding of the concepts, and guidelines from these tools. During the process of developing the requirements applied to the case study (slip-like perturbation), the absence of detailed and specific guidance for creating those requirements was confirmed as a considerable difficulty for the process, mainly during the data analysis phase, which led to another state-of-the-art related to the brain signal response to the loss of balance. A new study was conducted in order to perceive balance perturbations in brain signals which are always accompanied by a specific cortical activation (PEP). However, it is aggravated due to the reduced amount of papers related to ANN architecture to decode those signals. Considering the previously referred limitations found in the scientific literature, the present dissertation seeks to, firstly, present a comprehensive analysis of the brain signal response to slip-like perturbations previously collected at BirdLab. Through the brain signal analysis performed in Chapter 3it was possible to elicit PEPs, and the findings were in agreement with related papers. When the perturbation is delivered, it results in peaks in the N1 component. In addition, it was observed the maximum peak in central areas, mainly at Cz. 82 83 Regarding the deep learning results attained in this dissertation for slip-like perturbation recognition, a comparative analysis was performed by using different deep learning models and architecture parameters, which allows the most relevant combination to classify non-perturbation and perturbation events. The best classification algorithm for the deep learning model was the CNN classifier. The performance obtained for this algorithm was: Accuracy > 88.09%; F1-Score > 86.48%; MCC > 74.07%; Precision > 87.58%; Recall > 86.52%; and Specificity > 88.09%; Furthermore, the training runtime was same for all models, except to CNN-LSTM which runtime spent was higher than other models. In general, all deep learning outcomes for the same classification problem were good (accuracy > 84%), and their potential was demonstrated, indicating that they could be a good option, however, it is important to collect more data. The results from this study will be useful to design future studies for testing on those populations. Overall, the findings suggest that the EEG signals contain short-latency neural information related to an incoming fall, which may be useful for developing brain-machine interface systems for fall prevention. In short, the investigation work developed in the scope of this dissertation enabled for answering the RQs outlined in Chapter 1. • RQ 1: What are the most relevant signals and features that can help recognize the loss of balance provoked by slip-like perturbations? This RQ is addressed in Chapter 5. This study identified the presence of PEPs components in brain signals in response to slip-like perturbation and how changes in PEPs are related to changes in the actual dynamic postural response. For example, whether there is any difference in the onset, latency, or amplitude of ERPs that are correlated to particular elements of a balanced response. Compared to ERPs in other modalities such as visual, auditory, and memory-related ERPs, PEPs are the least explored ERPs even though they possess many diagnostic and research applications to assist in understanding the cortical control of balance [ 48 ]. It was found that all the components of the PEP were preserved and that the latency of the P1 and N1 waves preceded. This suggests the P1 and N1 components are viable signals for fall prediction. Compared to P1, N1 is a significantly larger component distributed across the central, frontal, and parietal channels at a latency of 100–150 ms, thus it can consider relevant features that can help recognize the loss of balance. Furthermore, evidence of the potential involvement of the cortex during reactive balance control in humans has come from event-related potentials (ERPs) that are time-locked to periods of instability. An unexpected postural perturbation evokes a large negative potential (termed as the perturbation-evoked potential N1 (PEP N1) that peaks 100–200 ms after the perturbation onset and is widely distributed over frontal, central, and parietal •RQ 2: What are the most relevant channels to decode the slip-like perturbations during walking? Since the component is distributed over frontal, central, and parietal areas with the peak amplitude located at Cz, the removal of channels that are far away from Cz does not change the classification accuracy and suggests that no additional discriminating information can be found in these channels [ 10 ]. The most relevant channels that can decode the slip-like perturbations during walking, based on analysis of brain signals in response to loss of balance events in Chapter 5and the results of performance metrics related to 84 DL-based models in Chapter 6, were the Fz, C3, Cz, C4, and Pz (5 channels). This hypothesis is supported by the high accuracies of well above 85% (peak) and the minimum runtime spent to train the models as presented in Appendix B. In addition, the minimal layout that uses only the Cz electrode obtained high accuracies of well above 80% (peak) as listed below, however, as per results regarding runtime in Appendix Bthis layout spent much time in comparison to 5 channels and 2 layouts is still a much-discussed topic and without a definitive answer – Speed at 1.6 km/h: Model = CNN-LSTM, ACC = 85.00%, F1S = 82.10%, MCC = 71.40%, PREC = 90.03%, REC = 82.38% and SPEC = 85.00%; – Speed at 2.5 km/h: Model = CNN, ACC = 91.07%, F1S = 90.40%, MCC = 81.06%, PREC = 90.34%, REC = 90.73% and SPEC = 91.07%; – Subject 1: Model = CNN, ACC = 88.02%, F1S = 86.37%, MCC = 73.96%, PREC = 86.88%, REC = 87.13% and SPEC = 88.02%; –Subject 2: Model = CNN-LSTM, ACC = 86.36%, F1S = 84.77%, MCC = 72.85%, PREC = 89.59%, REC = 83.55% and SPEC = 86.36%; – Combined: Model = LSTM, ACC = 81.90%, F1S = 79.76%, MCC = 63.05%, PREC = 84.55%, REC = 78.94% and SPEC = 81.90%. • RQ 3: What is the best Deep Learning model to implement for the detection of slip-like perturbation during walking? From the work results presented in Chapter 6, the best Deep Learning algorithm for slip-like perturbation events recognition was the CNN architecture described in Chapters 2and 6. When testing the trained model with unseen data it was achieved the following results related to five (5) data subsets, which were study objects of this dissertation: – Speed at 1.6 km/h: ACC = 88.10%, F1S = 86.91%, MCC = 74.08%, PREC = 87.58%, REC = 87.12% and SPEC = 88.10%; – Speed at 2.5 km/h: ACC = 92.86%, F1S = 92.26%, MCC = 85.48%, PREC = 94.19%, REC = 91.37% and SPEC = 92.86%; – Subject 1: ACC = 95.31%, F1S = 94.39%, MCC = 89.68%, PREC = 96.44%, REC = 95.65% and SPEC = 95.31%; – Subject 2: ACC = 93.18%, F1S = 91.88%, MCC = 80.74%, PREC = 91.49%, REC = 92.78% and SPEC = 93.18%; – Combined: ACC = 88.10%, F1S = 86.48%, MCC = 73.87%, PREC = 91.03%, REC = 86.86% and SPEC = 88.10%. In addition, through direct observation of the graphs in Appendix Bin Figures 46,56,66,76 and 86, the mean, standard deviation and maximum of the runtime spent to train the ANN architectures, the CNN model also presented the best performance spending less time to be trained, as described below: 7.1. Future Work 85 –Speed at 1.6 km/h: AVG = 163.75s, SD = 59.54s and MAX = 315s; –Speed at 2.5 km/h: AVG = 162.56s, SD = 61.43s and MAX = 311s; –Subject 1: AVG = 151.53s, SD = 58.96s and MAX = 292s; –Subject 2: AVG = 176.88s, SD = 75.11s and MAX = 351s; –Combined: AVG = 261.59s, SD = 134.40s and MAX = 578s. 7.1 F U T U R E W O R K In summary, significant components in PEPs were recognized as early as 70-155 ms after the onset of a mechanical external disturbance, and the perturbations were decoded using a CNN model from a single channel. Furthermore, it has been proved that the model was driven mostly by important components in the PEP to infer predictions, rather than by artifacts. However, it should be noted that these results were obtained using data from able-bodied young adults, and only the data collected from two participants were used in the process of analyzing brain signals in response to slip-like perturbations, as well as training and validating the artificial intelligence model. Future research into the repeatability and reproducibility of the findings in more susceptible groups, such as those over 65 and people with other gait disorders, such as chronic stroke, is needed. In addition, due to the fact PEP frequency analysis has only been performed in healthy people, future studies should analysis at the frequency properties of PEPs in older individuals and patient groups to see if abnormal frequency modulations in a specific frequency band contribute to the abnormalities identified in the time domain [48]. Regarding the experimental protocol, as initially the recognition of ADLs was the object of study of this dissertation, the new experimental protocol needs to be developed to carry out data collection experiments for the different daily activities listed in the literature review. Complementarily, as a suggestion for future research, it recommends that the steps of the EEGLAB framework used in the pre-processing phase be applied and tested with the data acquired from this new experimental protocol, whose aim is to validate whether the methods used for the recognition of the PEP also apply to the recognition of ADLs. As mentioned in this dissertation, non-invasive hBCI technologies have been studied and applied for attempting to improve classification accuracy. In addition, Electroencephalography (EEG) uses fast temporal resolution and is most widely utilized in combination with other brain/non-brain signal acquisition modalities. Thus, the suggestion for future work is to combine EEG and EMG signals in hBCI to detect muscular movement which acts as an artifact in EEG signals, resulting in the false detection of brain signals, and analyze EMG activation patterns during slip-like perturbations to verify the latency related to postural recovery after perturbation events. Due to the fact cross-validation was not applied to the dataset and the size of it is smaller in comparison with the data related to the papers in Chapters 2and 3. As a result, it is envisaged that the results produced would be suboptimal, thus, attempts to enhance the design should be incorporated in future works in order to discover how deep learning approaches operate best in the context of identifying slip-like perturbations. Furthermore, the use of cross-validated grid search to identify the optimal hyperparameters for ANN architecture, demands future studies to investigate this method in order to obtain the optimal model to decode the brain signal. 7.1. Future Work 86 Although other online BCI systems do not suffer severely from the use of a window-based classification technique, a BCI that is designed to detect loss of balance control operates under strong time constraints [ 10 ]. Future studies should address the delay imposed by the window-based categorization approach and determine if the duration of that delay is problematic in real-world scenarios. Overall, the findings suggest that the EEG signals contain short-latency neural information related to an incoming fall, which may be useful for developing brain-machine interface systems for fall prevention. Despite comparable results with the literature, continued development of the tested neural networks and the usage of raw inertial data as input to the networks should be processed to be implemented. Ablation studies should be conducted to determine the impact of the various stages of each architecture on fall event recognition. BIBLIOGRAPHY [1] Ludwig Kip A, Miriani Rachel M, Langhals Nicholas B, Joseph Michael D, Anderson David J, and Kipke Daryl R. Using a common average reference to improve cortical neuron recordings from microelectrode arrays. The Journal of Neurophysiology, 101, 2009. [2] N. Van Dijk T. Van der Hooft A. C. Scheffer, M. J. Schuurmans and S. E. de Rooij. Fear of falling: Measurement strategy, prevalence, risk factors and consequences among older persons. Age and Ageing, 37(1):19–24, 2008. [3] S. Makeig A. Delorme. Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. Journal of Neuroscience Methods, 2003. [4] L. W. Forrester A. Presacco and title = "Decoding Intra-Limb and Inter-Limb Kinematics During Treadmill Walking From Scalp Electroencephalographic (EEG) Signals J. L. Contreras-Vidal". IEEE Xplore, 20(2):1–3, 2012. [5] Ali Agga, Ahmed Abbou, Moussa Labbadi, Yassine El Houm, and Imane Hammou Ou Ali. Cnn-lstm: An efficient hybrid deep learning architecture for predicting short-term photovoltaic power production. Electric Power Systems Research, 208:107908, 2022. [6] Didier Allexandre, Armand Hoxha, H. Vikram Shenoy, Soha Saleh, S. Easter Selvan, and Guang H. Yue. Altered cortical and postural response to balance perturbation in traumatic brain injury – an eeg pilot study. 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 1543–1546, 2019. [7] A. Singla E. Singla M. Isaksson B. S. Rupal, S. Rafique and G. S. Virk. Lower-limb exoskeletons: Research trends and regulatory guidelines in medical and non-medical applications. International Journal of Advanced Robotic Systems, pages 1–2, 2017. [8] Chicco Davide and Jurman Giuseppe. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics, 21, 2020. [9] Chicco Davide, Tötsch Niklas, and Jurman Giuseppe. The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BMC Genomics, 41, 2021. [10] Jonas C Ditz, Andreas Schwarz, and Gernot R Müller-Putz. Perturbation-evoked potentials can be classified from single-trial EEG. Journal of Neural Engineering, 17(3):036008, may 2020. 87 BIBLIOGRAPHY 88 [11] B. Nienhuis J. Duysens P. Desain N. Keijsers J. Farquhar E. García-Cossio1, M. Severens. Decoding sensorimotor rhythms during robotic-assisted treadmill walking for brain computer interface (bci) applications. Plos One, 2016. [12] ZiYing Fang, Neo Darren, Jue Wang, YiCheng Zheng, and Lam Yvonne Y. H. An integrated fall prevention system with single-channel eeg and emg sensor. 2021 4th International Conference on Circuits, Systems and Simulation (ICCSS), pages 183–189, 2021. [13] Sleep Foundation. Alpha waves and sleep. Accessed: 01.04.2023. [14] Van Houdt Greg, Mosquera Carlos, and Nápoles Gonzalo. A review on the long short-term memory model. Artificial Intelligence Review, 53, 12 2020. [15] K. Watanabe H. Yokoyama, N. Kaneko and K. Nakazawa. Neural decoding of gait phases during motor imagery and improvement of the decoding accuracy by concurrent action observation. Journal of Neural Engineering, 2021. [16] S. M. Shafiul Hasan, Masudur R. Siddiquee, Roozbeh Atri, Rodrigo Ramon, J. Sebastian Marquez, and Ou Bai. Prediction of gait intention from pre-movement eeg signals: a feasibility study. Journal of NeuroEngineering and Rehabilitation, 17, 2020. [17] Ismail Fawaz Hassan, Forestier Germain, Weber Jonathan, Idoumghar Lhassane, and Muller Pierre-Alain. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33, 07 2019. [18] Yongtian He, David Eguren, José M Azorín, Robert G Grossman, Trieu Phat Luu, and Jose L Contreras-Vidal. Brain–machine interfaces for controlling lower-limb powered robotic systems. Journal of Neural Engineering, 15(2):021004, 2018. [19] Priyadarshini Ishaani and Cotton Chase a. A novel lstm–cnn–grid search-based deep neural network for sentiment analysis. The Journal of Supercomputing, 17, 2021. [20] Pang Ivan, Okubo Yoshiro, Sturnieks Daina, Lord Stephen R., and Brodie Matthew A. Detection of near falls using wearable devices: A systematic reviews. Journal of Geriatric Physical Therapy, 42:48–56, 2019. [21] K. L. Snyder J. E. Kline, H. J. Huang and D. P. Ferris. Isolating gait-related movement artifacts in electroencephalography during human walking. Journal of Neuro Engineering, 2015. [22] C. Guan J. H. Jeong, N. S. Kwak and S. W. Lee. Decoding movement-related cortical potentials based on subject-dependent and section-wise spectral filtering. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(3), 2020. [23] M. Jochumsen and I. K. Niazi. Detection and classification of single-trial movement-related cortical potentials associated with functional lower limb movements. Journal of Neural Engineering, 2020. BIBLIOGRAPHY 89 [24] H. J. Huang K. L. Snyder, J. E. Kline and D. P. Ferris. Independent component analysis of gait-related movement artifact recorded using eeg electrodes during treadmill walking. Frontiers in Neuroscience, 2015. [25] L. Perroud R. Chavarriaga Millán J. del R. K. Lee, D. Liu. A brain-controlled exoskeleton with cascaded event-related desynchronization classifiers. Robotics and Autonomous Systems, 90:15–23, 2017. [26] A. P. P. A Majeed R. M. Musa A. F. Ab. Nasir B. S. Bari M. Rashid, N. Sulaiman and S. Khatun. Current status, challenges, and possible solutions of eeg-based brain-computer interface: A comprehensive review. Frontiers in Neurorobotics, 2020. [27] Luis Mercado, Lucero Alvarado, Griselda Quiroz-Compean, Rebeca Romo-Vazquez, Hugo Vélez-Pérez, M.A. Platas-Garza, Andrés A. González-Garrido, J.E. Gómez-Correa, J. Alejandro Morales, Angel RodriguezLiñan, Luis Torres-Treviño, and José M. Azorín. Decoding the torque of lower limb joints from eeg recordings of pre-gait movements using a machine learning scheme. Neurocomputing, 446:118–129, 2021. [28] José del R. Millán. The human-computer connection : an overview of brain-computer interfaces. Metode Science Studies Journal-Annual Review, 9:135, 2018. [29] Teli Mohammad. Dimensionality reduction and classification of time embedded eeg signals. 01 2023. [30] Belkacem Abdelkader Nasreddine, Jamil Nuraini, Palmer Jason A., Ouhbi Sofia, and Chen Chao. Brain computer interfaces for improving the quality of life of older adults and elderly patients. Frontiers in Neuroscience, 14, 2020. [31] Recep A. Ozdemir, Jose L. Contreras-Vidal, and William H. Paloski. Cortical control of upright stance in elderly. Mechanisms of Ageing and Development, 169:19–31, 2018. [32] N. Mingchinda P. Leelaarporn N. Kunaseth S. Tammajarung P. Manoonpong S. C. Mukhopadhyay R. Chaisaen, P. Autthasan and T. Wilaiprasitporn. Decoding eeg rhythms during action observation, motor imagery, and execution for standing and sitting. IEEE Sensors Journal, 20, 2020. [33] O. Lennon E. Formaggio N. Sutaj G. Dazzi C. Venturin I. Bonini R. Ortner H. A. C. Bazo L. Tonin S. Tortora S. Masiero R. Di Marco, M.Rubega and A. Del Felice. Experimental protocol to assess neuromuscular plasticity induced by an exoskeleton training session. MPDI, 2021. [34] M. K. Al-Hares R. Kolaghassi and K. Sirlantzis. Systematic review of intelligent algorithms in gait analysis and prediction for lower limb robotic systems. IEEE Access, 9, 2021. [35] Akshay Sujatha Ravindran, Christopher A Malaya, Isaac John, Gerard E Francisco, Charles Layne, and Jose L Contreras-Vidal. Decoding neural activity preceding balance loss during standing with a lower-limb exoskeleton using an interpretable deep learning model. Journal of Neural Engineering, 19(3):036015, may 2022. [36] Abhijeet Rojatkar. Precision, recall, sensitivity and specificity. Accessed: 24.10.2022. BIBLIOGRAPHY 90 [37] K. C. Tan Abdullah Al-Mamun N. Thakor A. Bezerianos S. K. Goh, H. A. Abbass and J. Li. Spatio–spectral representation learning for electroencephalographic gait-pattern classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29(9), 2018. [38] and; H. Kim S. Park; F. C. Park; J. Cho. Eeg-based gait state and gait intention recognition using spatiospectral convolutional neural network. 7th International Winter Conference on Brain-Computer Interface (BCI), 2019. [39] C. Chisari S. Micera S. Tortora, S. Ghidoni and F. Artoni. Deep learning-based bci for gait decoding from eeg with lstm recurrent neural network. Journal of Neural Engineering, 2020. [40] C. Chisari S. Micera E. Menegatti S. Tortora, L. Tonin and F. Artoni. Hybrid human-machine interface for gait decoding through bayesian fusion of eeg and emg classifiers. Frontiers in Neurorobotics, pages 1–3, 2020. [41] Jalilpour Shayan and Müller-Putz Gernot. Toward passive bci: asynchronous decoding of neural responses to directionand angle-specific perturbations during a simulated cockpit scenario. Scientific Reports, 12(6802), 2022. [42] Sur Shravani and Sinha V. K. Event-related potential: An overview. Industry Psychiatry Journal, 18:70–73, 2009. [43] Y. He A. S. Ravindran S.Nakagome, T.P. Luu and J. L. Contreras-Vidal. An empirical comparison of neural networks and machine learning algorithms for eeg gait decoding. Nature, 2020. [44] Anuganti Suresh. What is a confusion matrix? Accessed: 24.10.2022. [45] D. Eguren M. Cestari T. P. Luu, Member and J. L. Contreras-Vidal. Eeg-based neural decoding of gait in developing children. International Conference on Systems, Man and Cybernetics (SMC), 2019. [46] S. Nakagame J. Gorges K. Nathan T. P. Luu, Y. He and J. L. Contreras-Vidal. Unscented kalman filter for neural decoding of human treadmill walking from non-invasive electroencephalography. 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2016. [47] Y. He T. P. Luu, S. Nakagome and J. L. Contreras-Vidal. Real-time eeg-based brain-computer interface to a virtual avatar enhances cortical involvement in human treadmill walking. Nature, 2017. [48] Jessy Parokaran Varghese, Robert E. McIlroy, and Michael Barnett-Cowan. Perturbation-evoked potentials: Significance and application in balance control research. Neuroscience & Biobehavioral Reviews, 83:267– 280, 2017. [49] Jessy Parokaran Varghese, William R. Staines, and William E. McIlroy. Activity in functional cortical networks temporally associated with postural instability. Neuroscience, 401:43–58, 2019. [50] WHO. Who global report on falls prevention in older age. Accessed: 01.02.2022. BIBLIOGRAPHY 91 [51] S. Prasad A. Kilicarslan Y. Zhang, 2 and J. L. Contreras-Vidal. Multiple kernel based region importance learning for neural classification of gait states from eeg signals. Frontiers in Neuroscience, 2017. A.3. Plots from Grand Average of Perturbations over Fz, C3, C4 and Pz electrodes 98 Figure 33: Fz - Grand Average PEP for (Speed = 2.5km/h) Figure 34: C3 - Grand Average PEP for (Speed = 1.6km/h) A.3. Plots from Grand Average of Perturbations over Fz, C3, C4 and Pz electrodes 99 Figure 35: C3 - Grand Average PEP for (Speed = 2.5km/h) Figure 36: C4 - Grand Average PEP for (Speed = 1.6km/h) A.3. Plots from Grand Average of Perturbations over Fz, C3, C4 and Pz electrodes 100 Figure 37: C4 - Grand Average PEP for (Speed = 2.5km/h) Figure 38: Pz - Grand Average PEP for (Speed = 1.6km/h) A.3. Plots from Grand Average of Perturbations over Fz, C3, C4 and Pz electrodes 101 Figure 39: Pz - Grand Average PEP for (Speed = 2.5km/h) A.4. Subdatasets from EEGLAB 102 A.4 S U B DATA S E T S F R O M E E G L A B Table 12: Dataset extracted from EEGLAB Dataset from EEGLAB Month Treadmill Speed Channels/ICA Epochs Qty Data_16_ago_CAR_Channels_ERP August 1.6km Channels 30 Data_25_ago_CAR_Channels_ERP August 2.5km Channels 40 Data_16_set_CAR_Channels_ERP September 1.6km Channels 30 Data_25_set_CAR_Channels_ERP September 2.5km Channels 39 Data_16_ago_CAR_ICA_ERP August 1.6km ICA 30 Data_25_ago_CAR_ICA_ERP August 2.5km ICA 40 Data_16_set_CAR_ICA_ERP September 1.6km ICA 30 Data_25_set_CAR_ICA_ERP September 2.5km ICA 39 B SLIP-LIKE PERTURBATION CLASSIFICATION METRICS FOR DEEP LEARNING MODELS - CHANNELS B.1 OUTCOMES OF ENTIRE DATASET (N O F I LT E R ): Figure 40: All subjects and speeds - Table filter by CNN model. 103 B.1. Outcomes of Entire Dataset (no filter): 104 Figure 41: All subjects and speeds - Table filter by LSTM model. Figure 42: All subjects and speeds - Table filter by GRU model. B.1. Outcomes of Entire Dataset (no filter): 105 Figure 43: All subjects and speeds - Table filter by CNN-LSTM model. Figure 44: All subjects and speeds - Classification Metrics Graphics sort of by (a) Models, (b) Channels, (c) Batch and (d) Overlap B.1. Outcomes of Entire Dataset (no filter): 106 Figure 45: All subjects and speeds - Accuracy greater 80% and 85% sort of by Model, Channels, Batch and Overlap. Overall accuracy greater 80% and 85%. Overall MCC greater 70% and 80%. Figure 46: All subjects and speeds - Runtime sort of by Model. B.1. Outcomes of Entire Dataset (no filter): 107 Figure 47: All subjects and speeds - Runtime sort of by Channel. Figure 48: All subjects and speeds - Runtime sort of by Batch Size. B.2. Outcomes filtered Speed = 1.6km/h 114 Figure 59: Runtime sort of by Overlap. Dataset filtered by speed = 1.6km/h. B.3. Outcomes filtered by Speed = 2.5km/h 115 B.3 OUTCOMES FILTERED BY SPEED = 2.5K M /H Figure 60: Table filter by CNN model (Speed = 2.5km/h). B.3. Outcomes filtered by Speed = 2.5km/h 116 Figure 61: Table filter by LSTM model (Speed = 2.5km/h). Figure 62: Table filter by GRU model (Speed = 2.5km/h). B.3. Outcomes filtered by Speed = 2.5km/h 117 Figure 63: Table filter by CNN-LSTM model (Speed = 2.5km/h). Figure 64: Classification Metrics Graphics sort of by (a) Models, (b) Channels, (c) Batch and (d) Overlap. Dataset filtered by speed = 2.5km/h. B.3. Outcomes filtered by Speed = 2.5km/h 118 Figure 65: Accuracy greater 80% and 85% sort of by Model, Channels, Batch and Overlap. Overall accuracy greater 80% and 85%. Overall MCC greater 70% and 80%. Dataset filtered by speed = 2.5km/h. Figure 66: Runtime sort of by Model. Dataset filtered by speed = 2.5km/h. B.3. Outcomes filtered by Speed = 2.5km/h 119 Figure 67: Runtime sort of by Channels. Dataset filtered by speed = 2.5km/h. Figure 68: Runtime sort of by Batch. Dataset filtered by speed = 2.5km/h. B.3. Outcomes filtered by Speed = 2.5km/h 120 Figure 69: Runtime sort of by Overlap. Dataset filtered by speed = 2.5km/h. B.4. Outcomes filtered by Subject 1 121 B.4 OUTCOMES FILTERED BY SUBJECT 1 Figure 70: Table filter by CNN model (Subject 1). B.4. Outcomes filtered by Subject 1 122 Figure 71: Table filter by LSTM model (Subject 1). Figure 72: Table filter by GRU model (Subject 1). B.4. Outcomes filtered by Subject 1 123 Figure 73: Table filter by CNN-LSTM model (Subject 1). Figure 74: Classification Metrics Graphics sort of by (a) Models, (b) Channels, (c) Batch and (d) Overlap. Dataset filtered by Subject 1. B.5. Outcomes filtered by Subject 2 130 Figure 85: Accuracy greater 80% and 85% sort of by Model, Channels, Batch and Overlap. Overall accuracy greater 80% and 85%. Overall MCC greater 70% and 80%. Dataset filtered by subject 2. Figure 86: Runtime sort of by Model. Dataset filtered by Subject 2. B.5. Outcomes filtered by Subject 2 131 Figure 87: Runtime sort of by Channels. Dataset filtered by Subject 2. Figure 88: Runtime sort of by Batch. Dataset filtered by Subject 2. B.5. Outcomes filtered by Subject 2 132 Figure 89: Runtime sort of by Overlap. Dataset filtered by Subject 2. C SLIP-LIKE PERTURBATION CLASSIFICATION METRICS FOR DEEP LEARNING MODELS - ICS C.1 OUTCOMES OF ENTIRE DATASET (N O F I LT E R ): Figure 90: Classification metrics results of all subjects and speeds - Entire Dataset 133 C.1. Outcomes of Entire Dataset (no filter): 134 Figure 91: All subjects and speeds - Classification Metrics Graphics sort of by (a) Models, (b) Batch and (c) Overlap Figure 92: All subjects and speeds - Accuracy greater 80% and 85% sort of by Model, Batch and Overlap. Overall accuracy greater 80% and 85%. Overall MCC greater 70% and 80%. C.1. Outcomes of Entire Dataset (no filter): 135 Figure 93: All subjects and speeds - Runtime sort of by Model. Figure 94: All subjects and speeds - Runtime sort of by Batch. C.2. Outcomes filtered Speed = 1.6km/h 136 Figure 95: All subjects and speeds - Runtime sort of by Overlap. C.2 OUTCOMES FILTERED SPEED = 1.6K M /H Figure 96: Classification metrics results of dataset filtered by 1.6km/h (Speed) C.2. Outcomes filtered Speed = 1.6km/h 137 Figure 97: Classification Metrics Graphics sort of by (a) Models, (b) Batch and (c) Overlap. Dataset filtered by speed = 1.6km/h. Figure 98: Accuracy greater 80% and 85% sort of by Model, Batch and Overlap. Overall accuracy greater 80% and 85%. Overall MCC greater 70% and 80%. Dataset filtered by speed = 1.6km/h. C.2. Outcomes filtered Speed = 1.6km/h 138 Figure 99: Runtime sort of by Model. Dataset filtered by speed = 1.6km/h. Figure 100: Runtime sort of by Batch. Dataset filtered by speed = 1.6km/h. C.3. Outcomes filtered by Speed = 2.5km/h 139 Figure 101: Runtime sort of by Overlap. Dataset filtered by speed = 1.6km/h. C.3 OUTCOMES FILTERED BY SPEED = 2.5K M /H Figure 102: Classification metrics results of dataset filtered by 2.5km/h (Speed)