Scientific Advances in STEM: From Professor to Students

Edited by Scientific Advances in STEM From Professor to Students Yadir Torres Hernández, Manuel Felix Angel, Ana María Beltrán Custodio and Francisco Garcia Moreno Printed Edition of the Special Issue Published in Metals, Polymers, Foods, Sustainability and Sensors www.mdpi.com/topics Scientific Advances in STEM: From Professor to Students Scientific Advances in STEM: From Professor to Students Editors Yadir Torres Hern´andez Manuel Felix Angel Ana Mar´ıa Beltr´an Custodio Francisco Garcia Moreno MDPI •Basel •Beijing •Wuhan •Barcelona •Belgrade •Manchester •Tokyo •Cluj •Tianjin Editors Yadir Torres Hern´ andez University of Seville (US) Spain Manuel Felix Angel University of Sevilla (US) Spain Ana Mar´ ıa Beltr´ an Custodio University of Seville (US) Spain Francisco Garcia Moreno Helmholtz-Centre Berlin Germany Editorial Office MDPI St. Alban-Anlage 66 4052 Basel, Switzerland This is a reprint of articles from the Topical Collection published online in the open access journals Metals (ISSN 2075-4701) (available at: https://www.mdpi.com/journal/metals), Polymers (ISSN 2073-4360) (available at: https://www.mdpi.com/journal/polymers), Foods (ISSN 2304-8158) (available at: https://www.mdpi.com/journal/foods), Sustainability (ISSN 2071-1050) (available at: https://www.mdpi.com/journal/sustainability) and Sensors (ISSN 1424-8220) (available at: https: //www.mdpi.com/journal/sensors). For citation purposes, cite each article independently as indicated on the article page online and as indicated below: LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. Journal Name Year,Volume Number, Page Range. ISBN 978-3-0365-1775-9 (Hbk) ISBN 978-3-0365-1776-6 (PDF) Cover image courtesy of Sergio Mu˜ noz Moreno (University of Seville (Spain)). © 2021 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications. The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND. Contents About the Editors .............................................. ix Preface to ”Scientific Advances in STEM: From Professor to Students” .............. xi Lourdes Duran-Lopez, Juan P. Dominguez-Morales, Antonio Rios-Navarro, Daniel Gutierrez-Galan, Angel Jimenez-Fernandez, Saturnino Vicente-Diaz and Alejandro Linares-Barranco Performance Evaluation of Deep Learning-Based Prostate Cancer Screening Methods in Histopathological Images: Measuring the Impact of the Model’s Complexity on Its Processing Speed Reprinted from: Sensors 2021,21, 1122, doi:10.3390/s21041122 .................... 1 Antonio Parejo, Sebasti´an Garc´ıa, Enrique Personal, Juan Ignacio Guerrero, Antonio Garc´ıa and Carlos Leon OpenADR and Agreement Audit Architecture for a Complete Cycle of a Flexibility Solution Reprinted from: Sensors 2021,21, 1204, doi:10.3390/s21041204 .................... 15 Mar´ıa Dolores Borr´as-Talavera, Juan Carlos Bravo and C´esar ´ Alvarez-Arroyo Instantaneous Disturbance Index for Power Distribution Networks Reprinted from: Sensors 2021,21, 1348, doi:10.3390/s21041348 .................... 37 Francisco Luna-Perej´on, Luis Mu ˜noz-Saavedra, Javier Civit-Masot, Anton Civit and Manuel Dom´ınguez-Morales AnkFall—Falls, Falling Risks and Daily-Life Activities Dataset with an Ankle-Placed Accelerometer and Training Using Recurrent Neural Networks Reprinted from: Sensors 2021,21, 1889, doi:10.3390/s21051889 .................... 55 Fan Zhao, Sidi Shao, Lin Zhang and Zhiquan Wen A Straightforward and Efficient Instance-Aware Curved Text Detector Reprinted from: Sensors 2021,21, 1945, doi:10.3390/s21061945 .................... 75 Shivani Choudhary, Deborah Herdt, Erik Spoor, Jose´ Fernando Garc´ıa Molina, Marcel Nachtmann and Matthias R¨adle Incremental Learning in Modelling Process Analysis Technology (PAT)—An Important Tool in the Measuring and Control Circuit on the Way to the Smart Factory Reprinted from: Sensors 2021,21, 3144, doi:10.3390/s21093144 .................... 93 Jia-Rong Ou, Shu-Le Deng and Jin-Gang Yu WS-RCNN: Learning to Score Proposals for Weakly Supervised Instance Segmentation Reprinted from: Sensors 2021,10, 3475, doi:10.3390/s21103475 ....................105 C.Fern´andez-Ponce,J.M.M´anuel,R.Fern´andez-Cisnal,E.F´elix,J.Beato-L´opez,J.P.Mu˜noz- Miranda,A.M.Beltr´an,A.J.Santos,F.M.Morales,M.P.Yeste,O.Bomati-Miguel, R. Litr´an and F. Garc´ıa-C´ozar Superficial Characteristics and Functionalization Effectiveness of Non-Toxic Glutathione-Capped Magnetic, Fluorescent, Metallic and Hybrid Nanoparticles for Biomedical Applications Reprinted from: Metals 2021,11, 383, doi:10.3390/met11030383 ....................123 v Andrea Mesa-Restrepo, Ana Civantos, Jean Paul Allain, Edwin Pati ˜no, Juan Fernando Alzate, Norman Balc´azar, Robinson Montes, Juan Jos´e Pav´on, Jose´ Antonio Rodr´ıguez-Ortiz and Yadir Torres Synergistic Effect of rhBMP-2 Protein and Nanotextured Titanium Alloy Surface to Improve Osteogenic Implant Properties Reprinted from: Metals 2021,11, 464, doi:10.3390/met11030464 ....................147 Fernando Ancio, Esperanza Rodriguez-Mayorga and Beatriz Hortigon Analysis of the Main Aspects Affecting Bonding in Stainless Steel Rebars Embedded in a Hydraulic Medium Reprinted from: Metals 2021,11, 786, doi:10.3390/met11050786 ....................165 Hailong Wang, Wenping Deng, Tao Zhang, Jianhua Yao and Sujuan Wang Development of Elastoplastic-Damage Model of AlFeSi Phase for Aluminum Alloy 6061 Reprinted from: Metals 2021,11, 954, doi:10.3390/met11060954 ....................187 Mar´ıa Alonso-Gonz´alez, Manuel Felix, Antonio Guerrero and Alberto Romero Effects of Mould Temperature on Rice Bran-Based Bioplastics Obtained by Injection Moulding Reprinted from: Polymers 2021,13, 398, doi:10.3390/polym13030398 .................201 Mercedes Jim´enez-Rosado, V´ıctor Perez-Puyana, Pablo S´anchez-Cid, Antonio Guerrero and Alberto Romero Incorporation of ZnO Nanoparticles into Soy Protein-Based Bioplastics to Improve Their Functional Properties Reprinted from: Polymers 2021,13, 486, doi:10.3390/polym13040486 .................213 Estefan´ıa ´ Alvarez-Castillo, Carlos Bengoechea and Antonio Guerrero Strengthening of Porcine Plasma Protein Superabsorbent Materials through a Solubilization-Freeze-Drying Process Reprinted from: Polymers 2021,13, 772, doi:10.3390/polym13050772 .................227 Arturo Tagliabue, Fernando Eblagon and Frank Clemens Analysis of Styrene-Butadiene Based Thermoplastic Magnetorheological Elastomers with Surface-Treated Iron Particles Reprinted from: Polymers 2021,13, 1597, doi:10.3390/polym13101597 ................239 Raquel Ca˜nete, Sonia L´opez and M. Estela Peralta KEYme: Multifunctional Smart Toy for Children with Autism Spectrum Disorder Reprinted from: Sustainability 2021,13, 4010, doi:10.3390/su13074010 ................251 Guodong Li, Zhibin Liu, Wenjia Tang, Dongpo He and Wei Shan Experimental and Numerical Study on the Flexural Performance of Assembled Steel-Wood Composite Slab Reprinted from: Sustainability 2021,13, 3814, doi:10.3390/su13073814 ................281 Eduardo Garc´ıa Villena, Silvia Pueyo-Villa, Irene Delgado Noya, Kilian Tutusaus Pifarr´e, Roberto Ru´ız Salces and Alina Pascual Barrera Instrumentalization of a Model for the Evaluation of the Level of Satisfaction of Graduates under an E-Learning Methodology: A Case Analysis Oriented to Postgraduate Studies in the Environmental Field Reprinted from: Sustainability 2021,13, 5112, doi:10.3390/su13095112 ................299 vi Shujing An, Dou Niu, Ting Wang, Binkai Han, Changfen He, Xiaolin Yang, Haoqiang Sun, Ke Zhao, Jiefang Kang and Xiaochang Xue Total Saponins Isolated from Corni Fructus via Ultrasonic Microwave-Assisted Extraction Attenuate Diabetes in Mice Reprinted from: Foods 2021,10, 670, doi:10.3390/foods10030670 ...................319 Rub´en Llinares, Pablo Ram´ırez, Jose´ Antonio Carmona, Luis Alfonso Trujillo-Cayado and Jose´ Mu ˜noz Assessment of Fennel Oil Microfluidized Nanoemulsions Stabilization by Advanced Performance Xanthan Gum Reprinted from: Foods 2021,10, 693, doi:10.3390/foods10040693 ...................335 vii sensors Article Performance Evaluation of Deep Learning-Based Prostate Cancer Screening Methods in Histopathological Images: Measuring the Impact of the Model’s Complexity on Its Processing Speed Lourdes Duran-Lopez 1,2,3,4,*, Juan P. Dominguez-Morales 1,2,4, Antonio Rios-Navarro 1,2,4, Daniel Gutierrez-Galan 1,2,4, Angel Jimenez-Fernandez 1,2,4, Saturnino Vicente-Diaz 1,2,4 and Alejandro Linares-Barranco 1,2,3,4   Citation: Duran-Lopez, L.; Dominguez-Morales, J.P.; Rios-Navarro, A.; Gutierrez-Galan, D.; Jimenez-Fernandez, A.; Vicente-Diaz, S.; Linares-Barranco, A. Performance Evaluation of Deep Learning-Based Prostate Cancer Screening Methods in Histopathological Images: Measuring the Impact of the Model’s Complexity on Its Processing Speed. Sensors 2021, 21, 1122. https://doi.org/10.3390/ s21041122 Academic Editors: Yadir Torres Hernández, Manuel Félix Ángel, Ana María Beltrán Custodio and Francisco Garcia Moreno Received: 13 January 2021 Accepted: 1 February 2021 Published: 5 February 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1Robotics and Tech. of Computers Lab, Universidad de Sevilla, 41012 Seville, Spain; [email protected] (J.P.D.-M.); [email protected] (A.R.-N.); [email protected] (D.G.-G.); [email protected] (A.J.-F.); [email protected] (S.V.-D.); [email protected] (A.L.-B.) 2Escuela Técnica Superior de Ingeniería Informática (ETSII), Universidad de Sevilla, 41012 Seville, Spain 3Escuela Politécnica Superior, Universidad de Sevilla, 41012 Seville, Spain 4Smart Computer Systems Research and Engineering Lab (SCORE), Research Institute of Computer Engineering (I3US), Universidad de Sevilla, 41012 Seville, Spain *Correspondence: [email protected] Abstract: Prostate cancer (PCa) is the second most frequently diagnosed cancer among men worldwide, with almost 1.3 million new cases and 360,000 deaths in 2018. As it has been estimated, its mortality will double by 2040, mostly in countries with limited resources. These numbers suggest that recent trends in deep learning-based computer-aided diagnosis could play an important role, serving as screening methods for PCa detection. These algorithms have already been used with histopathological images in many works, in which authors tend to focus on achieving high accuracy results for classifying between malignant and normal cases. These results are commonly obtained by training very deep and complex convolutional neural networks, which require high computing power and resources not only in this process, but also in the inference step. As the number of cases rises in regions with limited resources, reducing prediction time becomes more important. In this work, we measured the performance of current state-of-the-art models for PCa detection with a novel benchmark and compared the results with PROMETEO, a custom architecture that we proposed. The results of the comprehensive comparison show that using dedicated models for specific applications could be of great importance in the future. Keywords: deep learning; convolutional neural networks; artificial intelligence; prostate cancer; performance evaluation; benchmark 1. Introduction Prostate cancer (PCa) is the second most common cancer and the fifth leading cause of cancer death in men (GLOBOCAN [ 1 ]). In 2018, almost 1.3 million cases and around 360,000 deaths worldwide were registered due to this malignancy. According to the World Health Organization ( WHO ), there will be an increase of prostate cancer ( PCa ) cases worldwide, with 1,017,712 new cases being estimated for 2040. Most of these cases will be registered in Africa, Latin America, the Caribbean and Asia, and appear to be related to an increased life expectancy [2]. To diagnose PCa , digital rectal examination (DRE) is the primary test for the initial clinical assessment of the prostate. Then, prostate-specific antigen ( PSA ) is used in a screening method for the investigation of an abnormal prostatic nodule found in a digital rectal examination ( DRE ). Finally, in the case of abnormal DRE and elevated PSA results, Sensors 2021,21, 1122. https://doi.org/10.3390/s21041122 https://www.mdpi.com/journal/sensors 1 Sensors 2021,21, 1122 trans-rectal ultrasound-guided biopsy is performed to obtain samples of the prostate tissue [ 3 ]. Then, these tissue samples are scanned, resulting on gigapixel-resolution images called whole-slide images (WSIs), which are then analyzed and diagnosed by pathologists. Due to the high increment of new cases, and thanks to the impacts of artificial intelligence ( AI ) in recent years [ 4 , 5 ], several computer-aided diagnosis ( CAD ) systems have been developed to speed up the process of PCa diagnosis. A computer-aided diagnosis (CAD) system is an automatic or semi-automatic algorithm whose purpose is to assist doctors in the interpretation of medical images in order to provide a second opinion in the diagnosis. Among the different AI algorithms, deep learning (DL) has become very popular in recent years, and convolutional neural networks (CNNs) particularly [ 6 ]. They have been applied in several fields in medical image analysis, such as in disorder classification [ 7 ], lesion/tumor classification [ 8 ], disease recognition [ 9 ] and image construction/enhancement [10], among others. Deep learning (DL) algorithms have also been applied to other medical image analysis fields such as histopathology, in which whole-slide images ( WSI s) are used. Since it is not possible for a convolutional neural network ( CNN ) to work with a whole WSI as input due to its large size, a common approach is to divide this image into small subimages called patches. This procedure has been widely used in order to develop CAD systems in this field. Recently, many researchers have investigated the application of CAD systems to the diagnosis of PCa in WSI s. Ström et al. [ 11 ] developed a deep learning ( DL )-based CAD system to perform a binary classification distinguishing between malignant and normal tissue. The classification was performed using an ensemble of 30 widely used InceptionV3 models [ 12 ] pretrained on ImageNet. They achieved areas under the curve (AUC) of 0.997 and 0.986 on the validation and test subsets, respectively. For areas detected as malignant, the authors trained another ensemble of 30 InceptionV3 CNN s in order to discriminate between different PCa Gleason grading system ( GGS ) scores, achieving a mean pairwise kappa of 0.62 at slide level. Campanella et al. [ 13 ] presented a CAD system to detect malignant areas in WSI s. The classification was performed with the well-known ResNet34 model [ 14 ] together with a recurrent neural network ( RNN ) for tumor/normal classification. achieving an area under curve ( AUC ) of 0.986 at slide level. In a previous study [ 15 ], we proposed a CAD system, in which we focused on performing a patch-level classification of histopathological images between normal and malignant tissue. The proposed architecture, called PROMETEO, consisted of four convolution stages (convolution, batch normalization, activation and pooling layers) and three fully connected layers. The network achieved 99.98% accuracy, 99.98% F1 score and 0.999 AUC on a separate test set at patch level after training the network with a 3-fold cross-validation method. These previous works achieved competitive results in terms of accuracy, precision and other commonly-used evaluation metrics. However, to the best of our knowledge, most state-of-the-art works do not focus on prioritizing the speed of the CAD system as an important factor. Many of them used very complex, well-known networks to train and test, without taking into account the computational cost and the time required to perform the whole process. Since these algorithms are not intended to replace pathologists but to assist them in their task, in some cases it is better to prioritize the speed of the analysis, sacrificing some precision so that the expert has a faster and more dynamic response from the system. In this paper, a novel benchmark was designed in order to measure the processing and prediction time of a CNN architecture for a PCa screening task. First, the proposed benchmark was run for the PROMETEO architecture on different computing platforms in order to measure the impacts that their hardware components have on the WSI processing time. Then, using the personal computer ( PC ) configuration that achieved the best performance, the benchmark was run with different state-of-the-art CNN models, comparing them in terms of average prediction time both at patch level and at slide level, and also reporting the slowdown when compared to PROMETEO. 2 Sensors 2021,21, 1122 The rest of the paper is structured as follows: Section 2introduces the materials and methods used in this work, including the dataset (Section 2.1), the CNN models ( Section 2.2 ) and the benchmark proposed (Section 2.3). Then, the results obtained are presented in Section 3, which are divided in two different experiments: first, the performance of a proposed CNN model is evaluated in different platforms, and then it is compared to state- of-the-art, widely-known CNN architectures. Sections 4and 5present the discussion and the conclusions of this work, respectively. 2. Materials and Methods 2.1. Dataset In this work, a dataset with WSI s obtained from three different hospitals was used. These cases consisted of different Hematoxylin and Eosin-stained slides globally diagnosed as either normal or malignant. From Virgen de Valme Hospital (Sevilla, Spain), 27 normal and 70 malignant cases obtained by means of needle core biopsy were digitized into WSI s. Clínic Barcelona Hospital (Barcelona, Spain) provided 100 normal and 129 malignant WSIs, also obtained by means of needle core biopsy. Finally, from Puerta del mar Hospital (Cádiz, Spain), 65 malignant (26 obtained from needle core biopsy and 39 from incisional biopsy) and 79 (33 obtained from needle core biopsy and 46 from incisional biopsy) WSI s were obtained. Table 1summa- rizes the WSIs considered in the dataset. Table 1. Dataset summary. Hospital No. of WSIs Normal Malignant Total Virgen de Valme Hospital 27 70 97 Clínic Hospital 100 129 229 Puerta del Mar Hospital 79 65 144 2.2. CNN Models Different CNN s models were considered in this work in order to compare their performance by using the benchmark proposed in Section 2.3. Three different architectures from state-of-the-art DL -based PCa detection works were compared, along with other wellknown CNN architectures. The first one is the custom CNN model, called PROMETEO, which we proposed in [ 15 ], where we also demonstrated that applying stain-normalization algorithms to the patches in order to reduce color variability could improve the generalization of the model when predicting new unseen images from different hospitals and scanners. The second CNN architecture that was considered in this work is the wellknown ResNet34 model [ 14 ], which was used by Campanella et al. in [ 13 ]. The third one is InceptionV3, introduced in [12], which was used by Ström et al. [11]. Apart from these three CNN models, other widely-known architectures were evaluated with the same benchmark, comparing their performance in terms of execution time with the rest of the networks for the same task. These were VGG16 and VGG19 [ 16 ], MobileNet [17], DenseNet121 [18], Xception [19] and ResNet101 [14]. 2.3. Benchmark In this work, a novel benchmark was designed in order to measure and compare the performances of different CNN models and platforms on a PCa screening task. In order to make the benchmark feasible to be shared with other researchers so that it could be run in different computers, a reduced set of WSI s were chosen from the dataset presented in Section 2.1. Since the total amount of WSI s of the dataset represent more than 300 gigabytes (GB) hard drive space, only 40 of them were considered, building up a benchmark of around 50 GB, which is much more shareable. These 40 WSI s were randomly selected, 3 Sensors 2021,21, 1122 considering all the three different hospitals and scanners, and thus representing well the diversity of the dataset in this benchmark. The benchmark performs a set of processing steps which are detailed next (see Figure 1 ). First, as it was introduced in Section 1, since it is not possible for a CNN to use a whole WSI as input due to its large size, these images are divided into small subimages called patches (100 × 100 pixels at 10 × magnification in this case), which are read from each WSI . This process is called "read," and apart from extracting the patches from the input WSI , those corresponding to background are discarded (identified as D in the figure). Then, in the scoring step, a score is given to each patch depending on three factors: the amount of tissue that it contains, the percentage of pixels that are within Hematoxylin and Eosin’s hue range and the dispersion of the saturation and brightness channels. This score allows discarding patches corresponding to unwanted areas, such as pen marks, external agents and patches with small amounts of tissue, among others. In Figure 1, discarded patches in this step are highlighted in red, while those that pass the scoring filter are highlighted in green. The third step, called stain normalization, performs a color normalization of the patch based on Reinhard’s stain-normalization algorithm [ 20 , 21 ]in order to reduce color variability between samples. In prediction, which is the last step of the process, each of the patches are used as input to a trained CNN , which classifies them as either malignant or normal tissue. Deeper insights into these steps are given in [ 15 ]. When the execution of the benchmark finishes, it reports both the hardware and system information of the computer used to run the benchmark, and the results of the execution. These results consist of the mean execution time and standard deviation for each of the four processes (read, scoring, stain normalization and prediction) shown in Figure 1and presented in [15], both at patch level and at WSI level. WĂƚĐŚĞƐ ĐŽŶƐŝĚĞƌĞĚ     ZĞŝŶŚĂƌĚ ŶŽƌŵĂůŝnjĂƚŝŽŶ    EE EŽƌŵĂů ĂŶĐĞƌ /ŶƉƵƚ ZĞĂĚ ^ĐŽƌŝŶŐ ^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ WƌĞĚŝĐƚŝŽŶ KƵƚƉƵƚ   Figure 1. Block diagram detailing each of the steps considered for processing a whole-slide image (WSI) in the proposed benchmark. 3. Results The CNN -based PROMETEO architecture described in Section 2.2 was proposed and evaluated in terms of accuracy and many other evaluation metrics in [ 15 ]. In this work, we evaluated that model in terms of performance and execution time per patch and WSI. First, the same architecture was tested in different platforms using the benchmark proposed in Section 2.3. These results allow us to measure and quantify the impacts of different components in the whole processing and prediction process, which is useful for designing an edge-computing prostate cancer detection system. Then, the benchmark was used to evaluate the performances of different state-of-the-art CNN architectures on the computing platform that achieved the best results on the first experiment. Fourteen different PC configurations were used to evaluate the performance of the PROMETEO architecture introduced in Section 2.2. The hardware specifications (central processing unit ( CPU ) and graphics processing unit ( GPU )) of these computers are listed in Table A1 of Appendix A. In Figure 2, the average patch processing time is shown for each of the fourteen configurations, where the mean time for the steps performed when processing a patch (see Section 2.3) is reported. As it can be seen, the step that requires more time is the prediction in most of the cases, but it is highly reduced in configurations consisting of a GPU. 4 Sensors 2021,21, 1122 Ϭ ϬϬϬϱ ϬϬϭ ϬϬϭϱ ϬϬϮ ϬϬϮϱ ϬϬϯ ϬϬϯϱ ϬϬϰ ϬϬϰϱ &', / : <>DE dŝŵĞ;ƐͿ ǀĞƌĂŐĞƉƌŽĐĞƐƐŝŶŐƚŝŵĞƉĞƌƉĂƚĐŚ WĂƚĐŚZĞĂĚ WĂƚĐŚ^ĐŽƌĞ WĂƚĐŚ^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ WĂƚĐŚWƌĞĚŝĐƚŝŽŶ Figure 2. PROMETEO average patch processing time (in seconds) per step for each of the hardware configurations detailed in Table A1. Figure 3depicts the average and standard deviation of the execution time needed per WSI when running the benchmark on the fourteen different PC configurations. As in Figure 2, each of the steps considered in the whole process is shown. As it can be seen, reading the whole WSI patch by patch is the step that involves the longest amount of time in most of the devices (mainly in those configurations with no GPU ). This might seem contradictory considering Figure 2, but it is important to mention that, in that step, all patches from a WSI are read and analyzed, but not all of them are processed in the following steps. Unwanted areas, such as background regions with no tissue, are discarded before being scored. Then, only those which are not background and pass the scoring step are stain normalized and predicted by the CNN. Ϭ ϭϬ ϮϬ ϯϬ ϰϬ ϱϬ ϲϬ ϳϬ &', / : <>DE dŝŵĞ;ƐͿ ǀĞƌĂŐĞƉƌŽĐĞƐƐŝŶŐƚŝŵĞƉĞƌt^/ t^/ZĞĂĚ t^/^ĐŽƌĞ t^/^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ t^/WƌĞĚŝĐƚŝŽŶ Figure 3. PROMETEO average WSI processing time (in seconds) and standard deviation per step for each of the hardware configurations detailed in Table A1. 3.1. PROMETEO Evaluation The sum of the average execution time of the four preprocessing steps for each WSI was computed and it can be seen in Figure 4. The best case (device M) takes 22.56 ±5.67 s on average to perform the whole process per WSI , where the prediction step only represents 4.20 ±1.73 s. 5 Sensors 2021,21, 1122 Ϭ ϮϬ ϰϬ ϲϬ ϴϬ ϭϬϬ ϭϮϬ ϭϰϬ &', / : <>DE dŝŵĞ;ƐͿ ǀĞƌĂŐĞƉƌŽĐĞƐƐŝŶŐƚŝŵĞƉĞƌt^/ t^/ZĞĂĚ t^/^ĐŽƌĞ t^/^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ t^/WƌĞĚŝĐƚŝŽŶ Figure 4. PROMETEO average WSI processing time (in seconds) and standard deviation of the hardware configurations detailed in Table A1. The execution times obtained and used for generating the plots presented in this subsection are detailed in Table A2 of Appendix A. 3.2. Performance Comparison for Different State-of-the-Art Models After evaluating the PROMETEO architecture using the benchmark designed for this work with different PC s, the same network was compared to other widely-known architectures. For this purpose, the same computer (device M) was used in order to perform a fair comparison. The same benchmark that was used in the previous evaluation (see Section 3.1) was executed in computer M (see Table A1) for each of the CNN architectures mentioned in Section 2.2. The CNN s considered are PROMETEO [ 15 ], ResNet34 and ResNet101 [ 14 ], InceptionV3 [ 12 ], VGG16 and VGG19 [ 16 ], MobileNet [ 17 ], DenseNet121 [ 18 ] and Xception [19]. The average patch processing time per preprocessing step can be seen in Figure 5for each of the architectures mentioned. Since the architecture does not have an effect on the first three steps (reading the patch from the WSI , scoring it in order to discard unwanted patches, and normalizing it), the times needed to process them are similar across all the different cases reported in the figure. This does not happen with the prediction time, which directly depends on the complexity of the network. Ϭ ϬϬϬϱ ϬϬϭ ϬϬϭϱ ϬϬϮ ϬϬϮϱ ϬϬϯ ϬϬϯϱ ϬϬϰ ϬϬϰϱ ϬϬϱ WZKDdK ZĞƐEĞƚϯϰ /ŶĐĞƉƚŝŽŶsϯ s''ϭϲ s''ϭϵ DŽďŝůĞEĞƚ ĞŶƐĞEĞƚϭϮϭ yĐĞƉƚŝŽŶ ZĞƐEĞƚϭϬϭ dŝŵĞ;ƐͿ ǀĞƌĂŐĞƉƌŽĐĞƐƐŝŶŐƚŝŵĞƉĞƌƉĂƚĐŚ WĂƚĐŚZĞĂĚ WĂƚĐŚ^ĐŽƌĞ WĂƚĐŚ^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ WĂƚĐŚWƌĞĚŝĐƚŝŽŶ Figure 5. Average patch processing time (in seconds) per step for each of the CNN architectures using computer M (see Table A1). Figure 6reports the combined processing time that device M takes to compute a WSI on average, together with its corresponding standard deviation. The same case explained in Section 3.1, where the WSI reading step takes much longer than the patch reading step in relation to the rest of the subprocesses, can also be observed in this figure. It is important to 6 Sensors 2021,21, 1122 mention that the model proposed by the authors is faster than the rest in terms of prediction time, with a total of 22.56 ±5.67 s per WSI on average. Ϭ ϭϬ ϮϬ ϯϬ ϰϬ ϱϬ ϲϬ ϳϬ ϴϬ ϵϬ ϭϬϬ WZKDdK ZĞƐEĞƚϯϰ /ŶĐĞƉƚŝŽŶsϯ s''ϭϲ s''ϭϵ DŽďŝůĞEĞƚ ĞŶƐĞEĞƚϭϮϭ yĐĞƉƚŝŽŶ ZĞƐEĞƚϭϬϭ dŝŵĞ;ƐͿ ǀĞƌĂŐĞƉƌŽĐĞƐƐŝŶŐƚŝŵĞƉĞƌt^/ t^/ZĞĂĚ t^/^ĐŽƌĞ t^/^ƚĂŝŶEŽƌŵĂůŝnjĂƚŝŽŶ t^/WƌĞĚŝĐƚŝŽŶ Figure 6. Average WSI processing time (in seconds) and standard deviation for each of the CNN architectures using computer M (see Table A1). Table 2presents a summary of the results obtained for each architecture, focusing on the prediction process, which is the only one affected when changing the CNN architecture. Moreover, the number of trainable parameters and the slowdown are also reported. The latter is calculated by dividing the average prediction time per WSI of the corresponding CNN by that obtained with PROMETEO. This way, the improvement in terms of prediction time between PROMETEO and the rest of the architectures considered can be clearly seen. The proposed model predicts 2.55 × faster than the CNN used in [ 13 ] and 11.68 × faster than the one used in [ 11 ]. It is also important to mention that, in the latter, the authors did not use only an InceptionV3 model, but an ensemble of 30 of them. In this case, the figures and tables only report the execution times for a single network. When compared to other different widely-known architectures, PROMETEO is between 7.41×and 12.50×faster. Table 2. Average patch and WSI prediction time, slowdown and number of trainable parameters for each of the CNN architectures considered in this work. Model Avg. Prediction Time (patch) Avg. Prediction Time (WSI) Slowdown * Trainable Parameters PROMETEO 3.054 ±4.845 ms 4.201 ±1.739 s 1×1,107,010 ResNet34 8.982 ±10.086 ms 10.712 ±3.134 s 2.55×21,800,107 InceptionV3 41.301 ±44.282 ms 49.076 ±14.353 s 11.68×23,851,784 VGG16 28.664 ±9.241 ms 34.921 ±10.160 s 8.31×138,357,544 VGG19 29.931 ±9.305 ms 36.250 ±10.536 s 8.63×143,667,240 MobileNet 25.689 ±10.986 ms 31.110 ±9.030 s 7.41×4,253,864 DenseNet121 42.489 ±16.859 ms 51.483 ±14.945 s 12.25×8,062,504 Xception 34.050 ±11.789 ms 41.764 ±12.175 s 9.94×22,910,480 ResNet101 43.287 ±14.679 ms 52.517 ±15.266 s 12.50×44,707,176 * Calculated by using the average prediction time per WSI and taking the PROMETEO architecture as reference. A slowdown of A × means that model B is A times slower than PROMETEO. The execution times obtained and used for generating the plots presented in this subsection are detailed in Table A3 of Appendix B. 4. Discussion In order to design a fast edge-computing platform for PCa detection, an evaluation of a proposed CNN was performed. This allowed us to compare different hardware 7 Sensors 2021,21, 1122 components and configurations and measure the impacts of them when processing WSI s. Apart from the figures presented in Section 3.1, two specific cases are highlighted in Figure 7 . Figure 7a shows the impact that the frequency of the CPU has on the whole process when using the same computer. As it can be seen, the four processing steps clearly benefit when a faster CPU is used. On the other hand, Figure 7b compares two cases where the same configuration is used, except for the GPU , which was removed in one of them. As expected, the GPU highly accelerated the prediction time (by around three times in this case). Therefore, in order to build a low-cost edge-computing platform for PCa diagnosis, this analysis could be useful and should be taken into account in order to prioritize in which component the funds should be invested. As it was explained, all patches from a WSI have to be read, but not all of them have to be predicted, since the majority of them correspond to background and are discarded first. Therefore, the CPU has a higher impact than the GPU in the whole process. Ϭ ϭϬ ϮϬ ϯϬ ϰϬ ϱϬ ϲϬ  dŝŵĞ;ƐͿ ^ĂŵĞĐŽŵƉƵƚĞƌ;ŶŽ'WhͿ ĐŽ;ϭϮ',njͿǀƐŽŽƐƚŵŽĚĞ;Ϯϲ',njͿ t^/ZĞĂĚ t^/^ĐŽƌĞ t^/^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ t^/WƌĞĚŝĐƚŝŽŶ (a) Ϭ Ϯ ϰ ϲ ϴ ϭϬ ϭϮ ϭϰ ϭϲ ϭϴ 'D dŝŵĞ;ƐͿ ^ĂŵĞĐŽŵƉƵƚĞƌŶŽ'WhǀƐ'Wh t^/ZĞĂĚ t^/^ĐŽƌĞ t^/^ƚĂŝŶŶŽƌŵĂůŝnjĂƚŝŽŶ t^/WƌĞĚŝĐƚŝŽŶ (b) Figure 7. Impacts of the CPU and the GPU in the different WSI processing steps. ( a ) Same PC , different CPU frequency. Left: 1.2 GHz; right: 2.6 GHz. (b) Same PC. Left: without using GPU; right: using GPU. When comparing PROMETEO to other state-of-the-art CNN models, the former achieved the fastest prediction time, being from 2.55 times up to 12.50 times faster than any of the rest. Although the results in terms of accuracy and other commonly-used metrics in DL algorithms cannot be compared since the authors in [ 11 , 13 , 15 ] used different datasets, all of them reported state-of-the-art results for PCa detection. In [ 15 ], the authors compared PROMETEO to many of the models used in this work in terms of accuracy when using the same dataset for training and testing the CNN , showing that similar results were obtained. The use of transfer learning in CNN s for medical image analysis has become a commonplace technique, and most of the current research focuses on using this approach for avoiding the problem of having to design, train and validate a custom CNN model for a specific task. This has proved to achieve state-of-the-art results in many different fields and has also accelerated the process of training a custom CNN from scratch [ 22 ]. However, when using this technique, very deep CNN s are commonly considered, which, as presented in this work, leads to a higher computational cost when predicting an input image, and therefore, a slower processing time. Some specific tasks could benefit from designing shallower custom CNN models from scratch, such as DL -based PCa screening, providing a faster response to the pathologists in order to help them in this laborious process. With the increases in the number of cases and the mortality produced by PCa , this factor could become even more relevant in the future. As an alternative, cloud computing has provided powerful computational resources to big data processing and machine learning models [ 23 ]. Recent works have focused on accelerating CNN -based medical image processing tasks by using cloud solutions. While it is true that processing images using GPU s and tensor processing units ( TPU s) in the cloud is faster than in any local edge-computing device, there is an aspect that is not commonly taken into account when stating this fact: the time required to upload the image to the cloud. This depends on many factors and it is not easy to predict. Moreover, when digitizing 8 Sensors 2021,21, 1122 histological images, scanners store them in a local hard drive using around 1 GB for each of them. As an example, with an upload speed of 300 Mbps, it would take more than 27 s in ideal conditions just for uploading the WSI to the cloud, which is more than the time it would take to fully process the image on a local platform. To design a fast, low-cost, edge-computing platform, both the hardware components considered and the CNN model design have to be taken into account. Optimizing these two aspects led to achieving a very short WSI processing time when compared to current DL -based solutions without penalizing the performance of the system in terms of accuracy. In the next future, the authors would like to build a custom bare-bones approach based on the evaluations achieved in this work and test it in some of the hospitals that collaborated with us in this project. 5. Conclusions In this work, we have presented a comprehensive evaluation of the performance of PROMETEO, a previously-proposed DL -based CNN architecture for PCa detection in histopathological images, which achieved 99.98% accuracy, 99.98% F1 score and 0.999 AUC on a separate test set at patch level. Our proposed model outperforms other widely-used state-of-the-art CNN architectures such as ResNet34, InceptionV3, VGG16, VGG19, MobileNet, DenseNet121, Xception and ResNet101 in terms of prediction time. PROMETEO takes 22.56 s to predict a WSI on average, including the preprocessing steps needed, using an Intel ® Core ™ i7-8700K (Intel, Santa Clara, CA, USA) and an NVIDIA ® GeForce ™ GTX 1080 Ti (NVIDIA, Santa Clara, CA, USA). If we focus only on the prediction time, PROMETEO is between 2.55 and 12.50 times faster than any of the other architectures considered. The promising results obtained suggest that edge-computing platforms and custom CNN designs could play important roles in the future for AI -based medical image analysis, being able to aid pathologists in their laborious tasks speed-wise. Author Contributions: Conceptualization, L.D.-L. and J.P.D.-M.; methodology, L.D.-L.; software, L.D.-L. and J.P.D.-M.; validation, L.D.-L., J.P.D.-M., A.R.-N., D.G.-G. and A.J.-F.; formal analysis, L.D.- L. and J.P.D.-M.; investigation, L.D.-L. and J.P.D.-M.; resources, L.D.-L., J.P.D.-M., A.R.-N., D.G.-G., A.J.-F., S.V.-D. and A.L.-B.; data curation, L.D.-L. and J.P.D.-M.; writing—original draft preparation, L.D.-L. and J.P.D.-M.; writing—review and editing, L.D.-L., J.P.D.-M., A.R.-N., D.G.-G., A.J.-F., S.V.-D. and A.L.-B.; visualization, L.D.-L.; supervision, J.P.D.-M., S.V.-D. and A.L.-B.; project administration, S.V.-D. and A.L.-B.; funding acquisition, S.V.-D. and A.L.-B. All authors have read and agreed to the published version of the manuscript. Funding: This work was partially supported by the Spanish grant (with support from the European Regional Development Fund) MIND-ROB (PID2019-105556GB-C33), the EU H2020 project CHISTERA SMALL (PCI2019-111841-2) and by the Andalusian Regional Project PAIDI2020 (with FEDER support) PROMETEO (AT17_5410_USE). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Acknowledgments: We would like to than Gabriel Jimenez-Moreno and Luis Muñoz-Saavedra for executing the benchmark on their computers and reporting to us its performance. We would also like to thank Antonio Felix Conde-Martin and the Pathological Anatomy Unit of Virgen de Valme Hospital in Seville (Spain) for their support in the PROMETEO project, together with VITRO S.A., along with providing us with annotated WSI s from the same hospital. We would finally like to thank Puerta del Mar Hospital (Cádiz, Spain) and Clínic Barcelona Hospital (Barcelona, Spain) for providing us with diagnosed WSIs from different patients. Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. 9 Sensors 2021,21, 1122 Abbreviations The following abbreviations are used in this manuscript: AI Artificial Intelligence AUC Area Under Curve CAD Computer-Aided Diagnosis CNN Convolutional Neural Network CPU Central Processing Unit DL Deep Learning DRE Digital Rectal Examination GB Gigabyte GGS Gleason Grading System GPU Graphic Processing Unit H&E Hematoxylin and Eosin PC Personal Computer PCa Prostate Cancer PSA Prostate-Specific Antigen RNN Recurrent Neural Network TPU Tensor Processing Unit WHO World Health Organization WSI Whole-Slide Image Appendix A. PROMETEO Evaluation Table A1. Hardware specifications (CPU and GPU) of the different computers used in the PROMETEO evaluation. Device CPU GPU AIntel®Core™ i7-8850U @ 1.80 GHz 4 cores, 8 threads - BIntel®Core™ i9-7900X @ 3.30 GHz 10 cores, 20 threads - CIntel®Core™ i7-6700HQ @ 1.20 GHz 4 cores, 8 threads - DIntel®Core™ i7-6700HQ @ 2.60 GHz 4 cores, 8 threads - EIntel®Core™ i5-6500 @ 3.20 GHz 4 cores, 4 threads - FIntel®Core™ i7-4770K @ 3.50 GHz 4 cores, 8 threads - GIntel®Core™ i7-8700K @ 3.70 GHz 6 cores, 12 threads - HIntel®Core™ i7-4970 @ 3.60 GHz 4 cores, 8 threads - IIntel®Core™ i9-7900X @ 3.30 GHz 10 cores, 20 threads NVIDIA®GeForce™ GTX 1080 Ti 11 GB GDDR5X JAMD®Ryzen™ 9 3900X @ 4.20 GHz 12 cores, 24 threads NVIDIA®GeForce™ GTX 1080 Ti 11 GB GDDR5X KIntel®Core™ i5-6500 @ 3.20 GHz 4 cores, 4 threads NVIDIA®GeForce™ GT 730 2 GB GDDR5 LIntel®Core™ i7-4770K @ 3.50 GHz 4 cores, 8 threads NVIDIA®GeForce™ GTX 1080 Ti 11 GB GDDR5X MIntel®Core™ i7-8700K @ 3.70 GHz 6 cores, 12 threads NVIDIA®GeForce™ GTX 1080 Ti 11 GB GDDR5X NIntel®Core™ i7-4970 @ 3.60 GHz 4 cores, 8 threads NVIDIA®GeForce™ RTX 2060 6 GB GDDR6 10 Sensors 2021,21, 1204 have experienced in recent years, it is worth highlighting the use of Advanced Metering Infrastructure (AMI) [ 18 , 19 ] and in particular the deployment by the DSOs of customers’ smart meters (SM), which following the European directives are fully deployed in most of the European countries [ 20 ]. The AMI is composed by two main functions, which are the Automatic Metering Reading (AMR) and Automatic Metering Management (AMM). Both functions are usually integrated in the Smart Metering architecture, which is used for the monitoring of customer consumption and other elements of the power system. In Europe, the deployment of smart metering for gas and electricity started with the publication of the European Directive 2009/72/EC, by pointing out the advantages of such systems, and recommending their deployment [21]. Finally, communication and control of the “internal” elements of a building, as was mentioned before, is solved. There are many technologies and possibilities to integrate sensors, meters, and central control for facilities (even in the case of industrial wide areas, university campuses, etc.). Notwithstanding, communication between the DSO and the user is currently under development and research, mainly since this user-DSO integration is not still fully extended. This final connection between buildings and the distribution network is sometimes referred as “the last mile” of connection, as can be seen in Ref. [22]. Specifically, the work described in this paper is focused on this “last mile” environment. The objective is to define a complete system architecture for prosumers, providing a fluid communication channel with the aggregator/operator and capability to participate in flexibility services, including the audit mechanism. This proposal has been deployed on a testbed, which is called “DERBis lab” here, which is made up of a DR architecture, a Building Management System (BMS) to manage and operate the different loads, and an AMI solution to validate the compliance with the agreements between the different parties. The communication channel with the DSO/aggregator uses the OpenADR standard. Additionally, the smart metering infrastructure is based on the PRIME standard. In this paper, the whole process of DR planning is analyzed from the point of view of the customer, including the use of their own sensing and control systems to check the flexibility services availability. The architecture and process steps are described, analyzed, and finally tested using real data and information from the BMS and the smart metering devices. It is remarkable that the proposed architecture has been designed considering the same type of smart meters that are already deployed in most countries, which perform hourly consumption measurements. This restriction has been applied in order to avoid the need of deploying new metering systems for the present application, which would lead to higher costs. Before describing the proposed solution, a brief explanation about the state of the art in the field of flexibility and DSM implementation and AMI solutions is carried out in Section 2. Section 3contains the description of architecture and its parts. Then, Section 4 details the implementation of the proposed architecture for the DERBis lab. Section 5shows three study cases of flexibility services. Finally, the conclusions are shown in Section 6. 2. State of the Art The previous section outlines how the inclusion of flexibility programs could potentially bring multiple benefits for the management of the power system. The next step is to discuss which of the possible architectures is the most convenient one for the inclusion of prosumers, and how their possible actions should be categorized for their scheduling and dispatching. The tools for power management in the level of unit commitment are classified in different ways depending on the zone. In Ref. [ 23 ], J. Ortega, D. Watts, and H. Ren review how the operating reserves are categorized in USA, Chilean, and German markets. They conclude that five categories are used in the case of America, while three of them are established in Europe. From the list of European categories, the one that can be considered closer to the flexibility from the customer side is the Tertiary Reserve. This reserve, which operates 17 Sensors 2021,21, 1204 a maximum number of times in 15 min, is manually activated. Its use is a response to a disbalance in the control area and/or bigger congestion problems [ 23 ]. This description corresponds to their use in the context of system operations, but the same idea can be applied in congestions at the DSO level, even considering longer times (hours) for dispatched DSM events. In the context of American power markets, these services can fit into the category of Supplemental Reserves. There are, in fact, some examples of North American distribution companies that have already included DSM collaboration for their customers. This is the case of the companies San Diego Gas & Electric (SD-G&E) [ 24 ] or Pacific Gas & Electric (PG&E) [ 25 ], whose DR programs are focused on what is called a “Capacity Bidding Program”, where a customer commits, under contract, to reduce their power a certain quantity when required. The maximum number of requests is also established in the agreement. Other approaches propose the organization of flexibility and ancillary services under a specific market where TSOs and DSOs could buy those services that they need from prosumers and aggregators [26]. According to [ 6 ], DSM techniques can be categorized according to their objective. The categories are direct load reduction, load scheduling, pricing schemes, based on optimization types, and home energy management. Fortunately, in recent years some technology and protocols have emerged to solve these connections. Not only for DR, of course, but for managing the global Smart Grid environment, where DR is usually included. This has been a natural response to the rising environmental concern around the world, particularly in the U.S., Europe, and Asia [27]. A review of DR is presented in Ref. [ 28 ]. Specifically, this last paper studies the advantages of Automatic Demand Response (ADR) for DSM. ADR is the implementation of DR using Information and Communication Technologies (ICTs) without needing manual interventions. Samad et al. states that ADR is critical technology in DR. Additionally, in order to implement ADR, standard communication protocols such as OpenADR [ 29 , 30 ] are essential to allow interoperability as the authors state. OpenADR is a protocol to allow remote ADR procedures. Despite this protocol being very “general”, it offers many possibilities for running DSM and DR programs [ 31 ] in a variety of possible architectures. Some examples of these options are exposed in Refs. [ 32 , 33 ], putting special emphasis on the different architectures and connections that could be used. Another good point of this protocol is that it includes functions for dynamic pricing, which are used for example in Ref. [ 34 ] as part of more complex Smart Grid managing system. This is not the only service that can be included in the DR category. Another example of an ancillary service is peak saving [ 35 ]. The use of DR is not limited to the domestic/residential level, and examples at the industrial level can be found. An example is [ 36 ], where a service of DR is applied to refrigerated warehouses. The participation of a customer in DR implies two steps: the event dispatching, and the audit of the actuation during such an event. This means that the distribution company must check if the customer accomplished the requirements of consumption during the established time. Thanks to the network modernization drive by international directives (i.e., European directive 2009/72/EC [ 21 ] in which the energy smart meter deployment program was established for the first time), this audit can be done using the same systems that are usually used to establish the power bill, which are the customer’ smart meters. As [ 28 ] states, although ADR can be conducted without them, smart meters can enhance the implementation of a DR program, as they enable more elaborate means of compensating for DR participation by establishing baselines and comparing demand-side performance against those baselines. An example of use of OpenADR and smart metering can be seen in Ref. [37]. There are many types of solutions for the deployment of smart metering infrastructures, one of the most important characteristics being the method of communication. While non-wired solutions have many advantages, the Power Line Communication (PLC) 18 Sensors 2021,21, 1204 solutions are most extended for electricity metering, as they use the power lines as communication channels. A general structure of a PLC smart metering infrastructure in the power system can be seen in Figure 1. Figure 1. PLC (Power Line Communication)-based smart metering infrastructure [38]. As mentioned in the previous section, the deployment of smart metering is relatively advanced in Europe thanks to the impulse provided by legislative powers. This is very convenient for the future of ADR, as it can serve as auditory infrastructure. In fact, the smart metering deployment is very advanced not only for the electric sector, but also for water and gas utilities [39]. However, currently, the smart meter architectures are traditionally operating with aggregated measurements of one hour. Therefore, if DSOs want to use this system for auditing compliance with flexibility agreements, the offers and actions must match these time intervals. This limitation would be reduced if this granularity became a quarter of an hour, a possible period for this type of network, and more in line with flexibility services. The conditions of the audit should be based on the expected or estimated consumption of the customer during the period of the event, which will be compared to the actual one to check if the power was effectively reduced or not. In the case of buildings, this estimation can be done using multiple techniques, based on statistical, or even artificial, intelligent methods [40,41]. The review of the state of the art regarding the functionalities and applications of DSM, DR, and Smart Metering has shown that their use can bring multiple benefits to the power system. Nevertheless, their deployment brings about various technical complications, such as the integration and coordination of the different systems and actors. The next section proposes an architecture pointing out the requirements of each of its parts for DSM/DR implementation. 3. Proposed Architecture In this section, the proposed architecture is described. This architecture includes the complete cycle of DSM, including dispatching and auditing. The general structure is the one shown in Figure 2. 19 Sensors 2021,21, 1204 Figure 2. Structure and communication of DSO and customer systems for DSM/DR and auditing. As can be seen in the Figure 2, the DSM cycle is split into two parts: dispatching and auditing. The dispatching is done through the OpenADR protocol while the auditing is done through the AMI based on PRIME standard. The system in charge of managing the customer flexibility services is the BMS. The proposed architecture aims to take advantage of the already existing metering system for the audit process, not requiring any remarkable economic effort. The automation systems would require an update to fulfill the requirements, but there is currently a rising tendency to install BMSs in big and medium size customers and domestic automatization systems, so it is expected that these systems will become relatively common and its integration with the proposed approach will be easy. Finally, it is true that it is the DSMs architecture which requires bigger changes for its integration in the DSO control systems. Notwithstanding, it is a clear interest by the DSO. Proof of that is that various DSOs are already exploring the use of this kind of techniques, citing as examples the initiatives of San Diego Gas & Electric Company [24] and Pacific Gas & Electric Company [25]. Therefore, three main parts can be distinguished from the customer side, the DSO-user communications (DRMS-BMS) through the OpenADR protocol, the BMS in the customer side, and the smart metering system to audit the DSM operations. This section attempts to analyze, in detail, the required characteristics of the three parts. Moreover, the Section 3.4 explains how the whole process of DSM dispatching and auditing works. 3.1. OpenADR Standard As previously stated, a crucial aspect of flexibility (and, more specifically, DSM) and, therefore, congestion management is the communication between service providers (or customers) and the DSO. In this sense, the development of OpenADR standard is becoming an important tool to communicate grid flexibility actions. OpenADR came up in 2002 at the Demand Response Research Center, which is part of the Lawrence Berkeley National Laboratory (LBNL) in California [ 30 ]. In 2009, OpenADR specification (v1.0) specification was donated to different standardization organizations including the Organization for the Advancement of Structured Information Standards (OASIS) turning into an open-based specification. In 2010, the OpenADR Alliance was created by industry stakeholders to support the development, testing, deployment, and standardization of this protocol. The OpenADR 2.0 standard, which was released in its first version (2.0a) in 2011, was created using a profile from the Energy Interoperation 20 Sensors 2021,21, 1204 version 1.0 standard from OASIS, which is an information model to enable collaborative and transactive use of energy. Currently, it is in version 2.0b since 2015. The main difference of this version with the previous one (2.0a) is that the current version includes four services inherited from the Energy Interoperation (see below for the description of the services) while the previous one only includes one service (the EiEvent) and it is limited to just one signal. OpenADR is an open specification designed to facilitate and automate DR communication [ 42 ]. It is a communication data model, but it also defines transport and security mechanisms. The protocol is based on eXtensible Mark-up Language (XML). As transport mechanisms, OpenADR uses Extensible Messaging and Presence Protocol (XMPP) or Hypertext Transfer Protocol (HTTP) using REpresentational State Transfer (REST). The protocol defines two kinds of actors or nodes inherited from the OASIS Energy Interoperation, the Virtual Top Node (VTN), and the Virtual End Node (VEN). Communication is hierarchical, always between a VTN and one or more VENs. Thus, no communication between the same kind of nodes could take place on an OpenADR schema. Nevertheless, a node can be at the same time a VEN and a VTN of another hierarchically lower OpenADR stage. In an OpenADR schema, the VTN is responsible for communicating the DSM conditions and other relevant data to other entities (VENs). In this sense, the VEN is responsible for controlling and managing the electrical resources at the final consumer level or aggregating different resources and/or clients. Thus, the VTN will normally be at the DSO side and the VEN will be at the customer side. The VENs can be either physical devices or a cloud-based platform. Physical devices can have direct control of the customer assets based on the flexibility commands coming from the DSO. Cloud-based platforms can be used by Flexibility Service Providers (FSP) or Aggregators to gather different assets from different locations and owners in one single endpoint making the interaction easier for the DSO. As previously mentioned, the current regulation does not allow the DSO to own DER assets or to communicate flexibility signals to final customers in Europe. Nevertheless, flexibility signals to a customer using a third party like an aggregator are allowed. The current version of OpenADR defines four main communication services (see Figure 3) inherited from OASIS Energy Interoperation: • EiRegisterParty: Used to identify and enable communications between a VTN and a VEN. • EiEvent: Used by the VTN to request DR operations to one or more VENs. In the OpenADR terminology, the process of requesting a DSM is called event. Events can be accepted or rejected by the VEN. • EiReport: Used to share data. Both the VTN and the VEN are able to report information to each other, e.g., the VEN could report metering information to the VTN. • EiOpt: Used to communicate availability or unavailability, e.g., a VEN can communicate the period of time in which it will not accept any events coming from the VTN. In HTTP, VENs can communicate either in pull mode or push mode. In push mode the VEN acts as a client and the VTN acts as a server, the VEN periodically polls the VTN to see if it wants to communicate anything. In push mode, both the VTN and the VEN can act either as server or as client, thus, communication can be asynchronous, and no polling is needed. In a XMPP schema, due to its nature, communication is asynchronous, and no polling is needed. Additionally, OpenADR defines a fifth service: the oadrPoll, which is only used by the VEN in the HTTP pull mode to periodically poll the VTN. VTN Development There exist multiple implementations of VTN software [ 43 ], but they do not have an external interface to enable interoperability with other Smart Grid systems like a Demand Response Management System (DRMS) or a Distributed Energy Resources Management System (DERMS). Therefore, a VTN has been developed by the authors to fulfil these 21 Sensors 2021,21, 1204 requirements. The architecture is shown in Figure 4. The VTN architecture is divided in four main modules. From bottom to top: • Application Layer and Schema Validation: It handles the OpenADR HTTP services. The VTN is pull and push mode compatible. In addition, the compliance of the messages with the OpenADR protocol is also checked. •OpenADR Core: It implements the OpenADR 2.0b services. • Memory Manager: The memory manager is split into a Cache Memory manager which stores frequently requested data in RAM and a Persistence Memory Manager, which manages the connection with a relational database. This split allows the reduction of the database access, increasing the response times of the system. • API: It provides an interface to other systems to use the VTN services. It is based on REST and JavaScript Object Notation (JSON). Figure 3. OpenADR communication services. Figure 4. Architecture of the developed VTN with the different layers. 22 Sensors 2021,21, 1204 The data models used in the VTN infrastructure, besides OpenADR, also obey the Common Information Model (CIM) from the International Electrotechnical Commission (IEC). OpenADR is harmonized with this standard, which means that it is interoperable with Distribution Management (IEC 61968) and Energy Management (IEC 61970). This, together with the API, makes the VTN interoperable with other Smart Grid systems. Thanks to the API, the VTN acts as a gateway for other Smart Grid systems. In this sense, the VTN does not take any real actions by itself. These actions are done by means of the API interface. Additionally, the layer-based architecture makes the system easily upgradable to other OpenADR versions, even the EI. 3.2. Building Management System A BMS is a system that centralizes the information and control of a building or area. Their possibilities go from the mere monitorization of the consumption or comfort conditions to the advanced control algorithms and automation (e.g., [ 44 ] that propose a local Energy Consumption Scheduling for HVAC control that which compensates for possible errors between weather forecast and real local conditions). In this sense, most of the BMS control approaches are focusing on getting improvements in energy efficiency [ 45 ], obtaining the optimal use of renewable resources [ 46 ], or are based on static pricing schemes [47]. Beyond these local or static approaches, in the context of their inclusion as part of a DSM cycle, they should also accomplish certain tasks to permit the participation of the customer resources in flexibility services. Firstly, it must be aware of “when” and “how much” the resources can provide power reduction (or extra generation) if required. Secondly, it should include any type of consumption prediction, at least in a horizon of one day, to calculate which is the effective “margin of flexibility” that the building can offer. Finally, it must count with the communication systems that are used for the flexibility market and the DSO. This point is related to what was previously exposed regarding OpenADR. Thus, the BMS needs a communication endpoint with the DSO as means of an OpenADR VEN. 3.3. Smart Metering and Audit Process A smart metering system is made up of smart meters (usually one for each client) and a data concentrator, which receives the measurements and sends the information to the DSO central systems. The communication can be performed in multiple ways, but one of the most popular ones is the PLC [ 48 ]. This communication is done through the power lines, so it does not require extra elements for the deployment of smart metering. About the protocol for the PLC, some of the most extended ones in Europe are: • PRIME: Powerline Intelligent Metering Evolution (PRIME). Defined by the “Prime Alliance”. •Meters&More: Defined by the “Meters and More Association”. •G3PLC: Supported by the G3PLC Alliance. PRIME will be the chosen protocol for the present proposal, it being one of the most extensive in Spain. Notwithstanding, any of them could be valid from the point of view of required functionality. Thus, as was commented before, the consumption measurement period for billing is typically hourly in Europe. This measurement interval could initially be considered adequate for this application, although it would be advisable to increase the resolution to a quarter of an hour, in order to better adapt to the requirements of flexibility services. 3.4. Complete Flexibility Cycle Process Figure 5presents a resume of the whole process of flexibility estimation, participation in the market, event receiving and execution, and the audit. 23 Sensors 2021,21, 1204 Figure 5. Complete flexibility cycle. (1) Flexibility offers calculation [BMS]; (2) Send offers to market [BMS-Market]; (3) Services selection [DSO-Market]; (4) Service request [DSO-BMS]; (5) Load control for requested service [BMS-Loads]; (6) Auditory of the requested service [DSO]. At the start of the day (or before, depending on the flexibility program), the BMS performs an estimation of their own consumption (and generation, if any), and also considers which devices are curtailable (Stage 1 of the Figure 5). The flexibility offer is derived from these values, which is the description of the possible load reduction for each of the hours of the day (Stage 2). This offer is sent to the flexibility market, where all the offers of all the customers under the market are listed (Stage 3). The DSO selects those offers that are needed for the flexibility planning, and their selection is communicated to the affected customers using an OpenADR event (Stage 4). These events can be sent along the day but should be done some time before the moment when the requested event will start (e.g., at least one hour before). This restriction is usually described in the flexibility program. When the time of the event starts, the BMS will manage the loads to keep the requested power constraints according to the event characteristics, until the event finishes (Stage 5). At the end of the current day (or at the end of the settlement cycle), the DSO will have retrieved the data from the smart meters and make the audit for the billing adjustment, depending on whether the customer had correctly achieved the request of the event (Stage 6). In the next Section, the implementation of these systems in a real laboratory is detailed. 4. Laboratory Systems To demonstrate the correct operation of the system and the benefits of OpenADR in flexibility operations and congestion management a use case has been conducted. This use case has been conducted using the accommodations at the “DERBis” located at the Escuela Politécnica Superior of the University of Seville. The DERBis is a demonstrator of Smart Grid, DERs and Building Management technologies. 4.1. DERBis Description The classroom has 23 seats (22 for students and 1 for the professor), each one with a PC. This space is located at the Escuela Politécnica Superior and it is mainly used for teaching purposes for the MSc degree in Intelligent Systems, Energy, and Transport. Nevertheless, this classroom has become a living lab for different Smart Grid and Building Management technologies. This facility makes it possible to train students in the different possibilities of these technologies and how to manage them. Additionally, this space is used by researchers to test different scenarios and use cases related to flexibility. 24 Sensors 2021,21, 1204 The complete overview of the different devices and systems deployed in the lab is shown in Figure 6. Figure 6. DERBis systems overview. DRMS: Demand Response Management System; VTN: Virtual Top Node; VEN: Virtual End Node; BMS: Building Management System. On the one hand, the current Smart Grid deployments in the lab are an AMI based on smart meters and a physical OpenADR VEN. On the other hand, the Building Management deployments are based on different controllers with Modbus communications, it being possible to control different systems, such as: The Heat, Ventilation and Air Conditioning (HVAC) modules, the light circuits, and the user’s loads (wall plugs). Additionally, a centralized system manages all the resources of the lab. This system acts as a BMS that controls the room assets based on different rules and constraints. Figure 7shows the devices inside the room. Thanks to the controllable loads, and based on the energy demand prediction, the lab can provide flexibility services despite not counting with generation resources, simply reducing the load when it is required. 4.1.1. Smart Grid Technologies on the DERBis Living Lab Additionally, a physically OpenADR VEN has been installed: an EISSBox 3.0 [ 49 ] (see Figure 7). This VEN is fully OpenADR 2.0b compatible and it is currently connected to the VTN developed by the authors. Through this VEN, OpenADR commands can be sent to the lab simulating DSO actions. Also, the consumption of the lab can be obtained from the VEN by means of the report service of the OpenADR protocol. This VEN has four relay outputs that can be configured to respond to OpenADR events. Also, the VEN has four pulse inputs for energy metering purposes. Two of the relays of the VEN are connected to an input Modbus device in order to read their state. Two of the pulse inputs of the VEN are connected to the pulse outputs of the Modbus energy meter. 25 Sensors 2021,21, 1204 Figure 7. DERBis control and measurement devices. The smart meters of the DERBis use PLC communications based on PRIME protocol. All smart meters communicate with a PLC concentrator (CIRCUTOR SGE-PL1000) which receives the measurements and can send different commands to the meters. These commands are related to the management and configuration of each smart meter, which can be done separately. A more detailed description of the AMI devices can be found in [ 38 ], where their characteristics, structure, functions, and possibilities for teaching purposes are pointed out. The Smart Meters are distributed through different loads and locations of the lab (student benches, HVAC devices, etc.). In addition, one Smart Meter is installed at the head of the lab, measuring the overall room consumption. The schema of the smart metering system can be seen in the Figure 8. Figure 8. Smart metering system. 26 Sensors 2021,21, 1204 6. Conclusions The inclusion of DSM over the power system undoubtedly constitutes a powerful tool to improve the management capacity and flexibility of distribution networks. However, the use of these kinds of services implies an agreement between the two parts, i.e., the customers and the DSO, which could be performed under multiple existing terms, principles and methods. In this paper, a possible architecture for DSM dispatching and audit is proposed and tested. In this architecture, all the controllable devices are connected to a BMS. As it was exposed, the BMS established communication to the DSO systems using the standard OpenADR, providing a channel to inform about availability, scheduling, prices or any needed data. The audit function depends on the DSO smart meter infrastructure that provides remote information about the hourly consumption, this data being used to check the behavior of the loads in question during the period of the dispatched DSM events. The accomplishing or not of such events will therefore be evaluated using information from the smart meters to perform the audit process. Likewise, this hourly information is enough for this purpose, although its extension to a quarter hour is considered interesting for the future to increase network flexibility. As Proof of Concept the architecture has been tested on a real testbed located in a laboratory of the Escuela Politécnica Superior of the University of Seville. In this sense, as can be seen in its results, the proposed architecture clearly shows the possibilities of use of controllable loads under the established DSM contract conditions. The principal difficulty from the customer side would be the requirement of the OpenADR VEN device, which would not be such a big problem in the case of buildings that count with a BMS. Thus, thanks to taking advantage of the available power meter infrastructure that is used for the electricity billing, the audit process can be done by the DSO in a secure and appropriate manner, showing the adequacy of the AMI for this task, as this system makes easier the integration of flexibility services. It can also be concluded that the described laboratory constitutes a complete platform where some of the most important smart grid related technologies can be evaluated, integrated and taught. Author Contributions: Conceptualization: E.P., J.I.G., and A.G.; methodology: A.P., S.G. and A.G.; investigation: A.P., S.G. and E.P.; software: S.G., E.P., and J.I.G.; validation: A.P., S.G., E.P. and J.I.G.; formal analysis: A.P., S.G. and E.P.; writing, original draft preparation: A.P., S.G., E.P., and J.I.G.; writing, review and editing: A.G. and C.L.; supervision: A.G. and C.L.; project administration: C.L. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the Enel-Endesa Company, in the project “GRID Flexibility & Resilience Project” (number P020-19/E24), and by the “Ministerio de Ciencia, Innovación y Universidades”, Government of Spain under the project “Bigdata Analitycs e Instrumentación Cyberfísica para Soporte de Operaciones de Distribución en la Smart Grid”, number RTI2018-094917-B-I00. Sebastián García is also supported by this project. Moreover, Antonio Parejo is supported by the scholarship “Formación de Profesorado Universitario (FPU)”, Grant Number FPU16/03522 from the “Ministerio de Educación y Formación Profesional”, Government of Spain. Data Availability Statement: The data presented in this study are available on request from the corresponding author. The data are not publicly available due to project privacy issues. Acknowledgments: The authors would also like to thank the Enel-Endesa Company, the “Ministerio de Ciencia, Innovación y Universidades” and the “Ministerio de Educación y Formación Profesional”, Government of Spain, for their funding and support. Furthermore, the authors want to thank Alejandro Gallardo Soto and Julian Rodríguez Galán for their help and support in the implementation of the control system in the laboratory. Conflicts of Interest: The authors declare no conflict of interest. 33 Sensors 2021,21, 1204 References 1. Gerard, H.; Puente, E.I.R.; Six, D. Coordination between transmission and distribution system operators in the electricity sector: A conceptual framework. Util. Policy 2018,50, 40–48. [CrossRef] 2. Mohammed, N.A.; Al-Bazi, A. Management of renewable energy production and distribution planning using agent-based modelling. Renew. Energy 2021,164, 509–520. [CrossRef] 3. Martinez-Anido, C.B.; Botor, B.; Florita, A.R.; Draxl, C.; Lu, S.; Hamann, H.F.; Hodge, B.-M. The value of day-ahead solar power forecasting improvement. Solar Energy 2016,129, 192–203. [CrossRef] 4. Valentine, K.; Temple, W.; Thomas, R.J.; Zhang, K.M. Relationship between wind power, electric vehicles and charger infrastructure in a two-settlement energy market. Int. J. Electr. Power Energy Syst. 2016,82, 225–232. [CrossRef] 5. Romero-Ruiz, J.; Pérez-Ruiz, J.; Martin, S.; Aguado, J.A.; la Torre, S.D. Probabilistic congestion management using EVs in a smart grid with intermittent renewable generation. Electr. Power Syst. Res. 2016,137, 155–162. [CrossRef] 6. Ahmad, S.; Ahmad, A.; Naeem, M.; Ejaz, W.; Kim, H. A Compendium of Performance Metrics, Pricing Schemes, Optimization Objectives, and Solution Methodologies of Demand Side Management for the Smart Grid. Energies 2018,11, 2801. [CrossRef] 7. Malik, S.; Harish, V.S.K.V. Integration of automated Demand Response and Energy Efficiency to enable a smart grid infrastructure. In Proceedings of the 2019 2nd International Conference on Power Energy, Environment and Intelligent Control (PEEIC), Greater Noida, India, 18–19 October 2019; IEEE: New York City, NY, USA, 2019. 8. Ghatikar, G.; Zuber, J.; Koch, E.; Bienert, R. Smart grid and customer transactions: The unrealized benefits of conformance. In Proceedings of the 2014 IEEE Green Energy and Systems Conference (IGESC), Long Beach, CA, USA, 24 November 2014; IEEE: New York City, NY, USA, 2014. 9. Huh, J.-H.; Seo, Y.-S. Understanding Edge Computing: Engineering Evolution with Artificial Intelligence. IEEE Access 2019 ,7, 164229–164245. [CrossRef] 10. Giraldo-Soto, C.; Erkoreka, A.; Mora, L.; Uriarte, I.; Portillo, L.D. Monitoring System Analysis for Evaluating a Building’s Envelope Energy Performance through Estimation of Its Heat Loss Coefficient. Sensors 2018,18, 2360. [CrossRef][PubMed] 11. Parejo, A.; Sanchez-Squella, A.; Barraza, R.; Yanine, F.; Barrueto-Guzman, A.; Leon, C. Design and Simulation of an Energy Homeostaticity System for Electric and Thermal Power Management in a Building with Smart Microgrid. Energies 2019 ,12, 1806. [CrossRef] 12. Cheng, Z.; Li, Z.; Liang, J.; Si, J.; Dong, L.; Gao, J. Distributed coordination control strategy for multiple residential solar PV systems in distribution networks. Int. J. Electr. Power Energy Syst. 2020,117, 105660. [CrossRef] 13. Papavasiliou, A.; Mezghani, I. Coordination Schemes for the Integration of Transmission and Distribution System Operations. In Proceedings of the 2018 Power Systems Computation Conference (PSCC), Dublin, Ireland, 11–15 June 2018; IEEE: New York City, NY, USA, 2018. 14. Migliavacca, G.; Rossi, M.; Gerard, H.; Dzamarija, M.; Horsmanheimo, S.; Madina, C.; Kockar, I.; Leclecq, G.; Marroquín, M.; Svendsen, H.G. TSO-DSO coordination and market architectures for an integrated ancillary services acquisition: The view of the SmartNet project. In Proceedings of the CIGRE Session 2018, Paris, France, 27–31 August 2018. 15. European Union. Directive (EU) 2019/ 944 of The European Parliament and of The Council—of 5 June 2019—on Common Rules for the Internal Market for Electricity and Amending Directive 2012/27/EU.75; European Commission: Brussels, Belgium, June 2019. 16. Alonso, J.I.G.; Personal, E.; García, S.; Parejo, A.; Rossi, M.; García, A.; Delfino, F.; Pérez, R.; León, C. Flexibility Services Based on OpenADR Protocol for DSO Level. Sensors 2020,20, 6266. [CrossRef] 17. Yanine, F.; Sanchez-Squella, A.; Barrueto, A.; Sahoo, S.K.; Parejo, A.; Shah, D.; Cordova, F.M. Homeostaticity of energy systems: How to engineer grid flexibility and why should electric utilities care. Period. Eng. Nat. Sci. 2019,7, 474. [CrossRef] 18. Mak, S.T.; So, E. Integration of PMU, SCADA, AMI to accomplish expanded functional capabilities of Smart Grid. In Proceedings of the 29th Conference on Precision Electromagnetic Measurements (CPEM 2014), Rio de Janeiro, Brazil, 24–29 August 2014; IEEE: New York City, NY, USA, 2014. 19. Huh, J.-H. Smart Grid Test Bed Using OPNET and Power Line Communication; IGI Global: Hershey, PA, USA, 2018. 20. European Comission. Benchmarking Smart Metering Deployment in the EU-28. Final Report. March 2020. Available online: https: //op.europa.eu/en/publication-detail/-/publication/b397ef73-698f-11ea-b735-01aa75ed71a1 (accessed on 28 January 2021). 21. European Parliamen. Directive 2009/72/EC of the European Parliament and of the Council of 13 July 2009 Concerning Common Rules for the Internal Market in Electricity and Repealing Directive 2003/54/EC 2009; European Parliamen: Brussels, Belgium, 2009. 22. Cui, T.; Carr, J.; Brissette, A.; Ragaini, E. Connecting the Last Mile: Demand Response in Smart Buildings. Energy Procedia 2017 , 111, 720–729. [CrossRef] 23. Ren, H.; Ortega, J.; Casimis, D.W. Review of Operating Reserves and Day-Ahead Unit Commitment Considering Variable Renewable Energies: International Experience. IEEE Lat. Am. Trans. 2017,15, 2126–2136. [CrossRef] 24. San Diego Gas & Electric Company. Schedule CBP: Capacity Bidding Program; San Diego Gas & Electric Company: San Diego, CA, USA, April 2019. 25. Pacific Gas & Electric Company. Electric Schedule E-CBP: Capacity Bidding Program; Pacific Gas & Electric Company: San Francisco, CA, USA, July 2018. 26. Guerrero, J.I.; Personal, E.; Caro, S.G.; Parejo, A.; Rossi, M.; Garcia, A.; Sanchez, R.P.; Leon, C. Evaluating Distribution System Operators: Automated Demand Response and Distributed Energy Resources in the Flexibility4Chile Project. IEEE Power Energy Mag. 2020,18, 64–75. [CrossRef] 34 Sensors 2021,21, 1204 27. Samad, T.; Koch, E. Automated demand response for energy efficiency and emissions reduction. In PES T&D 2012; IEEE: New York City, NY, USA, 2012. 28. Samad, T.; Koch, E.; Stluka, P. Automated Demand Response for Smart Buildings and Microgrids: The State of the Practice and Research Challenges. Proc. IEEE 2016,104, 726–744. [CrossRef] 29. OpenADR Alliance. OpenADR 2.0 Profile Specification B Profile. 2013. Available online: https://www.openadr.org/specification (accessed on 26 November 2020). 30. Kiliccote, S.; Dudley, J.H.; Piette, M.A. Northwest Open Automated Demand Response Technology Demonstration Project; (No. LBNL- 2573E); Lawrence Berkeley National Lab. (LBNL): Berkeley, CA, USA, 2009. 31. McParland, C. OpenADR open source toolkit: Developing open source software for the Smart Grid. In Proceedings of the 2011 IEEE Power and Energy Society General Meeting, Detroit, MI, USA, 24–28 July 2011; IEEE: New York City, NY, USA, 2011. 32. Herberg, U.; Mashima, D.; Jetcheva, J.G.; Mirzazad-Barijough, S. OpenADR 2.0 deployment architectures: Options and implications. In Proceedings of the 2014 IEEE International Conference on Smart Grid Communications (SmartGridComm), Venice, Italy, 3–6 November 2014; IEEE: New York City, NY, USA, 2014. 33. Jingxi, Z.; Min, C.; Ling, W.; Lixia, Y.; Shixiong, P.; Dong, L. Research on Architecture of Automatic Demand Response System Based on OpenADR. In Proceedings of the 2018 China International Conference on Electricity Distribution (CICED), Tianjin, China, 17–19 September 2018; IEEE: New York City, NY, USA, 2018. 34. Wilcox, J.; Kaleshi, D.; Sooriyabandara, M. DIRECTOR: A distributed communication transport manager for the Smart Grid. In Proceedings of the 2014 IEEE International Conference on Communications (ICC), Sydney, Australia, 10–14 June 2014; IEEE: New York City, NY, USA, 2014. 35. Koch, E.L. Automated demand response—mathsemicolonfrom peak shaving to ancillary services. In Proceedings of the 2012 IEEE PES Innovative Smart Grid Technologies (ISGT), Washington, DC, USA, 16–20 January 2012; IEEE: New York City, NY, USA, 2012. 36. Goli, S.; McKane, A.; Olsen, D. Demand Response Opportunities in Industrial Refrigerated Warehouses in California; Lawrence Berkeley National Lab. (LBNL): Berkeley, CA, USA, 2012. 37. Seo, J.; Jin, J.; Kim, J.Y.; Lee, J.-J. Automated Residential Demand Response Based on Advanced Metering Infrastructure Network. Int. J. Distrib. Sens. Netw. 2016,12, 4234806. [CrossRef] 38. Parejo, A.; Garcia, S.; Personal, E.; Garcia, A.; Guerrero, J.I.; Leon, C. Living-Lab for Smart Grid technologies teaching. In Proceedings of the 2020 XIV Technologies Applied to Electronics Teaching Conference (TAEE), Porto, Portugal, 8–10 July 2020 ; IEEE: New York City, NY, USA, 2020. 39. Fagiani, M.; Squartini, S.; Gabrielli, L.; Spinsante, S.; Piazza, F. A review of datasets and load forecasting techniques for smart natural gas and water grids: Analysis and experiments. Neurocomputing 2015,170, 448–465. [CrossRef] 40. Daut, M.A.M.; Hassan, M.Y.; Abdullah, H.; Rahman, H.A.; Abdullah, M.P.; Hussin, F. Building electrical energy consumption forecasting analysis using conventional and artificial intelligence methods: A review. Renew. Sustain. Energy Rev. 2017 ,70, 1108–1118. [CrossRef] 41. Deb, C.; Zhang, F.; Yang, J.; Lee, S.E.; Shah, K.W. A review on time series forecasting techniques for building energy consumption. Renew. Sustain. Energy Rev. 2017,74, 902–924. [CrossRef] 42. Piette, M.A.; Kiliccote, S.; Ghatikar, G.; McKane, A.T.; Matson, N.; Page, J.; MacDonald, J.S.; Aghajanzadeh, A.; Black, D.R.; Yin, R. Demand Response Research Center—Final Report; Berkeley Lab’s Energy Technologies Area (ETA): Berkeley, CA, USA, 2015. 43. Electric Power and Research Institute (EPRI). OpenADR 2.0b Open Source Virtual Top Node (OADR2.0b VTN) Version 0.9.7.0.; Electric Power and Research Institute (EPRI): Palo Alto, CA, USA, 2017. 44. Du, Y.F.; Jiang, L.; Duan, C.; Li, Y.Z.; Smith, J.S. Energy Consumption Scheduling of HVAC Considering Weather Forecast Error Through the Distributionally Robust Approach. IEEE Trans. Ind. Inform. 2018,14, 846–857. [CrossRef] 45. Bianco, V.; Piazza, G.; Scarpa, F.; Tagliafico, L.A. Energy, economic and environmental assessment of the utilization of heat pumps for buildings heating in the Italian residential sector. Int. J. Heat Technol. 2017,35, S117–S122. [CrossRef] 46. Hosseinzadeh, M.; Salmasi, F.R. Robust Optimal Power Management System for a Hybrid AC/DC Micro-Grid. IEEE Trans. Sustain. Energy 2015,6, 675–687. [CrossRef] 47. Lagrange, A.; de Simón-Martín, M.; González-Martínez, A.; Bracco, S.; Rosales-Asensio, E. Sustainable microgrids with energy storage as a means to increase power resilience in critical facilities: An application to a hospital. Int. J. Electr. Power Energy Syst. 2020,119, 105865. [CrossRef] 48. Huh, J.-H.; Otgonchimeg, S.; Seo, K. Advanced metering infrastructure design and test bed experiment using intelligent agents: Focusing on the PLC network base technology for Smart Grid system. J. Supercomput. 2016,72, 1862–1877. [CrossRef] 49. IPKeys Technologies. EISS Box 3.0 System/Device Monitoring and Management; IPKeys Technologies: Stafford, VA, USA, 2017. 35 sensors Article Instantaneous Disturbance Index for Power Distribution Networks María Dolores Borrás-Talavera *, Juan Carlos Bravo and César Álvarez-Arroyo   Citation: Borrás-Talavera, M.D.; Bravo, J.C.; Álvarez-Arroyo, C. Instantaneous Disturbance Index for Power Distribution Networks. Sensors 2021,21, 1348. https:// doi.org/10.3390/s21041348 Academic Editor: Hossam A. Gabbar Received: 14 January 2021 Accepted: 11 February 2021 Published: 14 February 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). Electrical Engineering Department, Escuela Politécnica Superior, Universidad de Sevilla, c/Virgen de África 9, 41011 Sevilla, Spain; [email protected] (J.C.B.); [email protected] (C.Á.-A.) *Correspondence: [email protected] Abstract: The stability of power systems is very sensitive to voltage or current variations caused by the discontinuous supply of renewable power feeders. Moreover, the impact of these anomalies varies depending on the sensitivity/resilience of customer and transmission system equipment to those deviations. From any of these points of view, an instantaneous characterization of power quality (PQ) aspects becomes an important task. For this purpose, a wavelet-based power quality indices (PQIs) are introduced in this paper. An instantaneous disturbance index (ITD(t)) and a Global Disturbance Ratio index (GDR) are defined to integrally reflect the PQ level in Power Distribution Networks (PDN) under steady-state and/or transient conditions. With only these two indices it is possible to quantify the effects of non-stationary disturbances with high resolution and precision. These PQIs offer an advantage over other similar because of the suitable choice of mother wavelet function that permits to minimize leakage errors between wavelet levels. The wavelet-based algorithms which give rise to these PQIs can be implemented in smart sensors and used for monitoring purposes in PDN. The applicability of the proposed indices is validated by using a real-time experimental platform. In this emulated power system, signals are generated and real-time data are analyzed by a specifically designed software. The effectiveness of this method of detection and identification of disturbances has been proven by comparing the proposed PQIs with classical indices. The results confirm that the proposed method efficiently extracts the characteristics of each component from the multi-event test signals and thus clearly indicates the combined effect of these events through an accurate estimation of the PQIs. Keywords: power quality indices; signal processing; multi-resolution analysis; renewable energy applications 1. Introduction The extensive use of power-switching devices for source conditioning, renewable energy supply and motion control in modern industrial applications has detrimental sideeffects on power quality, such as increase potential for unacceptable harmonic levels, poor power factor, or unbalanced currents and voltages in power distribution networks. In addition, transients caused by faults and switching events in power systems significantly affect the power-transfer quality of a supply [ 1 , 2 ]. All these undesirable effects cause huge economic losses [ 3 ] and require an effective power quality analysis in the grid and affected facilities. Smart sensors proposals are rapidly increasing in response to these new requirements [ 4 – 9 ]. Although the first applications were almost exclusively limited to billing, they now include new features related to power quality detection and have a common denominator: they require high processing speed to handle large computer data in a shorter time. In this context it is crucial to provide a small number of parameters and indicators that can effectively characterize all these events and derive from a very fast and efficient analysis tool. This task is the main objective of this work. The power quality index (PQI) is the summary of waveform distortions in voltage and current from the perfect sinusoids. The PQIs are used to characterize the degree of Sensors 2021,21, 1348. https://doi.org/10.3390/s21041348 https://www.mdpi.com/journal/sensors 37 Sensors 2021,21, 1348 quality degradation in a quantitative manner. They represent the impacts of non-ideal waveforms on electrical power systems in a compact but expressive manner. Existing indices such as Total Harmonic Distortion (THD), Power factor, Flicker factor, etc. reflect the degree of power disturbance in each of these categories individually, but fail to assess most of the phenomena mentioned together in an exhaustive and concise manner by a single value [10]. Several methods have been applied to research the real-time behavior of controllers, test and protection equipment [ 11 ] and fault diagnosis devices [ 12 , 13 ]. All of these applications, together with those mentioned above, have a positive and economic impact on the industry. A measuring device integrated into the system offers the additional characteristic of quantifying the various characteristics that affect power quality and thus identifies any specific aspect that needs attention. Power quality measurement and analysis has typically been divided between steadystate concerns, such as harmonic distortion, and transient concerns, such as those resulting from faults or switching transients. Fourier analysis has been applied to the first class of problems while Joint Time-Frequency Analysis (JTFA), including wavelets, has traditionally been used for the second class [14]. When using the discrete Fourier transform (DFT) in nonstationary situations, the estimation of a time-varying signal during a specific time interval results in a serious deterioration of measurement accuracy. The effect is equivalent to the frequency deviation of the power system from its nominal value when measuring electrical signals [ 15 , 16 ]. To avoid this problem, the Fourier Transform applied in time segments as the Short-Time Fourier Transform (STFT) determines the frequency contents of a signal in each time window. Nevertheless, the size of the window affects the capacity of multi-resolution capability and the cost of the calculations is high. These deficiencies can be solved by using the Discrete Wavelet Transform (DWT) that also avoids the problems of interference of time and frequency distributions [ 17 ]. Besides, the DWT method offers a better time and frequency resolutions for high and low-frequency components, respectively. This method is best for locating disturbances in non-stationary conditions. However, the limitation of the DWT uncertain can be a shortcoming that can be minimized by an appropriate choice of the analyzing wavelet basic function. In this article, the Daubechies40 wavelet offers the best results for the analyzed electrical signals present in power distribution networks. Traditionally, DWT [ 18 – 24 ], the generalized S-transform (GST) [ 25 – 30 ], the Time- Frequency distributions, and the Short-DFT (SDFT) [ 31 ] have been used for the analysis of the transient and time-varying nature of disturbance signals in electric power systems. In particular, power quality indices [ 32 – 38 ] have been defined using these transforms. Thus, the estimation of standard time-varying PQIs is done based on a proper JTFA [ 32 ], adaptive window-based fast generalized S-transform [ 33 ], empirical wavelet transform (EWT)-based time-frequency technique [ 34 ], cluster analysis of long-term power quality (PQ) data [ 35 ], wavelet packet transform [ 36 ], load composition rate and Euclidean norm of total harmonic distortions [ 37 ], and global harmonic parameters for phasor measurement units [ 38 ]. Most of these works use many PQIs in their proposals, for example, the global PQIs introduced in [ 35 ] contain up to seven PQI factors. As will be shown later, only two PQIs are used in the present work. Specific treatment of transient events is done in [ 39 ], whereas in [ 37 ] only stationary distortion is considered. Thus, it is necessary to conceive a measure of power quality in order to capture simultaneously both the “transient” characteristics of disturbance signals in electrical power systems [ 39 ] and the stationary ones. Moreover, it must be done with a minimum computational cost and using a single descriptive indicator. Wavelet-based techniques are good candidates for this purpose. However, a method based on empirical wavelet transform will work properly for off-line data processing but not for real-time analysis due to its computational cost [34,40]. Due to their powerful characteristics, in this work, DWT and multiresolution analysis (MRA) are chosen for the joint analysis of the stationary and transient parts of electrical signals. These parts are extracted from a monitoring window to ensure the correct use of the 38 Sensors 2021,21, 1348 DWT. In this way, the fundamental component of the electrical signal is extracted and the window containing the transient disturbance is processed. Both aspects are subsequently used to construct new DWT based power-quality indices that replace existing counterparts. On the other hand, the main drawback of this method is based on its limitation in the analysis of highly noisy signals. In these noisy conditions, the method has been proven to work properly until the minimum value of 34 dB for the signal-to-noise ratio (SNR) [ 19 ], which is more than enough for most of practical PQ events detections in low-voltage AC power distribution networks. Reference [ 41 ] presents an interesting overview of power quality in low-voltage DC distribution networks. This study highlights the most relevant disturbances in DC networks and how they can be accurately characterized by means of power quality indices, some of which have been defined based on DFT. Due to the close relationship between the indices defined in AC power system networks and DC networks, a characterization of PQ in DC networks using the current DWT-based method seems to be interesting and deserves future work. In this possible context, the implementation of the algorithms derived from the present work in smart sensors may provide new functionalities for PQ improvement. However, they could increase the power consumption requirements of such sensors. In this way, reference [ 42 ] presents a useful study of the power requirement of functional sensors in a traditional PV system. It is based on Neural Network maximum power point tracking with cloud method and estimates the reduction percentage of power consumption in functional sensors. In this paper we propose two wavelet-based power quality indices, the Instantaneous Disturbance Index (ITD) and the Global Disturbance Ratio (GDR), which comprehensively assess the power transfer quality of a given supply in steady-state and/or transient situations. The proposed ITD instantly shows the evolution of the PQ. It has the advantage of evaluating the PQ in real-time situations under any conditions and extracting the characteristics of the disturbance for any load in the power distribution networks. In addition, GDR has the advantage of assessing PQ by means of a single value and allows distinguishing between different events, so the GDR can be used as an input in a disturbance classifier. Therefore, the new PQ indices can identify properly practical waveform distortions in power networks. They can also be used to evaluate both the effectiveness and dynamic responses of PQ mitigation equipment in practical applications. In this work, a real-time platform is being developed to experimentally validate the feasibility of the proposed PQI measurement and analysis method. The rest of this document is organized as follows: In Section 2, a simplified outline of wavelet filter selection criteria is presented. Section 3briefly describes the DWT-based instantaneous indices used to evaluate PQ. Section 4shows a detailed summary of the proposed measurement process is shown. In Section 5, results are discussed. The last section draws conclusions from the results. 2. Fundamentals of the Proposed Indices 2.1. Components Signal Estimation The proposed estimation technique uses DWT and MRA for extracting the fundamental component of the input signal s(t), which can be described by wavelet coefficients [ 14 , 24 ] s(t)= 2J ∑ k=1 aJ,kφJ,k(t)+ J ∑ j=1 2J ∑ k=1 dj,kψj,k(t)(1) where jand kare the wavelet frequency scale and wavelet time scale, respectively, Jis the highest jscale, i.e., the lowest frequency band; a J,k and d J,k are the wavelet coefficients; and ψ (t) and ϕ (t) are the mother and scale wavelet functions, respectively. Equations (1)–(3) present the basic concepts of MRA. The aim of MRA is to develop representations of a complex signal f(t) in terms of scaling and wavelet functions. 39 Sensors 2021,21, 1348 The digitized version of input signal s(t) is a sequence of nsamples, s(n), which can be processed in the same way as the DFT. In addition, the number of DWT levels (decomposition) is limited by the number of the original signal samples, which in turn must be a power of two. Therefore, signal s(n) can be presented in terms of its frequency components, i.e., coefficient a J,k ,k=1, ... ,2 J , is the smoothed version of signal s(n), and coefficients d j,k ,k=1, ... ,2 J ,j=1, ... ,Jare detailed versions of s(n). They contain the lower and higher-frequency components, respectively [14]. s(n)=aJ(n)+ J ∑ j=1 dj(n)(2) With aj(k)=∑ ng(n−2k)aj−1(k) dj(k)=∑ nh(n−2k)aj−1(k)(3) where a j and d j are the approximation and detail coefficients at level j,g(k) and h(k)are the high pass and low pass filters corresponding to the scaling and wavelet filters. The coefficients have half of the original input data due to the downsampling process. For extracting the fundamental component of a signal by using MRA, the sampling rate and, so, the number of MRA steps must be specified. If it is assumed that His the frequency band order with central frequency, ω /(2 π ), equal to fundamental frequency and fs is the sampling frequency, the number of MRA steps, J, satisfies the following expression: fs 2J=2×H(4) In this work has been assumed that the most important transients occurring in actual situations of Power Distribution Networks (PDN) are captured into a frequency band of 6400 Hz, for fs =1/TS= 12.8 kHz. It assures accurate results with J=6. 2.2. Decomposition Structures of MRA Method MRA method uses a decomposition structure based on Quadrature Mirror Filter (QMF). QMF consists of two complementary filters, one low pass and the other high pass, which disjoin the frequency range into two equal parts. They decompose the input signal into two frequency intervals, the low pass filter output is downsampled and is used as new input of another identical filter pair corresponding to the next decomposition level. This operation is repeated recursively, decomposing the signal into approximation (a) and detail (d) coefficients for various scales. The coefficients of the scale gand wavelet hfilters, and the efficiency of wavelet analysis, are related to the selected mother wavelet. The right selection of the mother wavelet is the main task to obtain the desired results in signal analysis with WMRA. In this work, the selection of the analyzer function is based on the efficiency of the scale and wavelet filters used for this purpose. The QMF filters frequency response of the first level DWT decomposition is shown in Figure 1. It depicts a comparison of the scale and wavelet filters between the db4, db10, db20, db40, and dmey mother wavelets used on the MRA method. The filters have overlapping frequency bands and energy leakage occurs between the two adjacent bands affecting the obtained coefficients. It is less pronounced with db40 and dmey mother wavelet. 40 Sensors 2021,21, 1348 Figure 1. Comparative frequency response of mother wavelets. 2.3. Wavelet Filters Selection Several mother wavelets have been evaluated in order to select the most suitable for the specific application of event detection and classification methods. In this paper, the selection of an explicit wavelet is based on the following principles: - Minimum frequency leakage of QMF in the first levels of decomposition. - Number of filter coefficients. - Similarity between classical THD (Total Harmonic Distortion Ratio) with the TWD (Total Wavelet Disturbance Ratio) defined with wavelets. The main selection criterion is the frequency selectivity. It is based on the magnitude transition zone slope of the frequency response of the wavelet filters. The selected mother wavelet will have better characteristics if the slope of both filters has the highest value and the information dispersion related to the frequency content of the signal will be less. In this way, a greater concentration of energy is obtained in single frequency bands. A comparison between Fourier and Wavelet analysis is made to select the most appropriated. The STFT has good performance with signals disturbed by harmonics. According to the Parseval theorem, the energy of a signal can be decomposed in terms of the energy of aand dcoefficients, the selected mother wavelet will be that allocate the signal energy correctly. Figure 2a shows a signal that contains three harmonics, the 3rd (7% Urms), the 5th (10% Urms), and the 9th (10% Urms). This signal is chosen to support the selection of the best mother wavelet available. Figure 2b shows the FFT spectrum where the fundamental harmonic has 97.570% of the total signal energy, the 3rd has 0.478% and both the 5th and 9th have 0.976%. Figure 2c also shows as a bar graph the energy percentage of this disturbed signal obtained with various mother wavelets. Table 1shows the percentage of signal energy achieved with the mother wavelets depicted in the legend of Figure 2c. Table 2shows the THD and TWD results calculated, the results are similar excluding bior6.8 and db10 ones. According to Table 1, the mother wavelets with the best frequency response correctly distribute the energy of the signal according to the FFT spectrum (Figure 2). Therefore, the db40 and dmey wavelets are the best mother wavelets. Since the db40 wavelet has fewer coefficients than the dmey wavelet, the db40 is the mother wavelet with the best performance. 41 Sensors 2021,21, 1348 Figure 2. Voltage signal with harmonics: ( a ) Instantaneous representation, ( b ) FFT spectrum, ( c ) distribution of the energy percentage of the signal according to the coefficients of the mother wavelets. Table 1. Energies (%) of voltage signal decomposition shown in Figure 2. f(Hz) a6 0–100 d6 100–200 d5 200–400 d4 400–800 d3 800–1600 d2 100–200 d1 3200–6400 db10 97.6142 0.4378 1.1561 0.7912 0.0007 0.0000 0.0000 db20 97.5719 0.4610 1.0966 0.8706 0.0000 0.0000 0.0000 sym20 97.5705 0.5844 0.9745 0.8706 0.0000 0.0000 0.0000 db30 97.5704 0.4616 1.0550 0.9130 0.0000 0.0000 0.0000 bior6.8 97.2315 0.7903 1.1931 0.7817 0.0035 0.0000 0.0000 db40 97.5699 0.4673 1.0257 0.9371 0.0000 0.0000 0.0000 dmey 97.5705 0.4905 1.0116 0.9274 0.0000 0.0000 0.0000 Table 2. Total Harmonic Distortion (THD) and Total Wavelet Disturbance Ratio (TWD) of voltage signal decomposition shown in Figure 2. THD TWD db10 TWD db20 TWD sym20 TWD db30 TWD bior6.8 TWD db40 TWD dmey 15.7797 15.6335 15.7751 15.7798 15.7801 16.8741 15.7818 15.7796 3. Quantitative Formulations of Steady-State and Transient Power Quality Aspects This formulation is based on the IEC 61000-4-7 standard [ 43 ], EN-50160 standard [ 44 ] and the IEEE Std 1159 ™ -2009 [ 45 ] guidelines, in order to fulfil the essential requirements and to define the characteristics of the supplied voltage and the most common disturbances. 3.1. DWT-Based Disturbance Ratio The Total Harmonic Distortion ratio (THD) for single-phase networks (or polyphase balanced networks) has been defined traditionally as [43,46] THD =H ∑ h=2S2 k S1·100 H=fm 2·f0(5) THD =S2−S2 1 S1·100 (6) where Sdenotes RMS value of signal s(n), S 1 denotes fundamental component of s(n), and Skthe kcomponent of s(n). Using the MRA tool, Sis given by 42 Sensors 2021,21, 1348 Figure 8. Instantaneous Transient Disturbance ratio index ITD(t) of voltage disturbances corresponding to Figure 5. The ITD(t) (Figure 8) shows the instantaneous character of the disturbance noting the relevance of any suddenly introduced change. Instead, its averaged energy over the observation window is considered by the GDR. Furthermore, the GDR permits to distinguish between different disturbances and can be used for classification purposes. In particular, the GDR differentiates the proposed disturbances, with almost the same THD, by giving importance to both, the amplitude and the duration of the disturbance. This index offers the advantage to indicate clearly the most relevant of them, as can be seen in Table 4for any kind of the disturbances numbered as “1”. So transient events are perfectly characterized by this procedure. Nevertheless, the studied sag and swell signals have the same frequency contents and are undistinguished by all the indices, so the RMS value has to be considered as an assisting index. 5.2. Disturbances Combination in Voltage Waveforms In practical situations, a PQ event usually consists of a combination of singular disturbances, most of them treated in case A. Therefore, a set of complex disturbed signals with more severe and combined effect are studied. Figure 9a depicts the ITD(t) corresponding to a pronounced sag. It shows two peak value: 5.5% at 56.25 ms and 3.3% at 147.25 ms. Figure 9b shows this index for a combination of the mentioned sag and a steady-state distorted voltage with five harmonics (3rd, 5th, 7th, 9th, and 11th and Vrms equal to 10 V, 17 V, 7 V 13 V, and 3 V, respectively). The ITD(t) resulting from a steady-state distorted voltage only with the five harmonics is provided in Figure 9c. In addition, Figure 9d shows this index for a combination of the same steadystate voltage and an important transient oscillation. The ITD(t) of a voltage disturbed only with this single transient is shown in Figure 9e. It has a peak value of 30% at 127 ms. 49 Sensors 2021,21, 1348 In this case, the waveform offers more relevant changes than the previous set of examples because more substantial disturbances are present. When the signal is disturbed with two events, both are evinced in ITD(t) graph. The PQ indices obtained are presented in Table 5. By comparing the values obtained, the GDR index remarks the effect of the disturbances combination. As it can be observed from Table 5, the proposed index is the only one capable to indicate clearly the accumulative effect of the combined events in complex signals. Although the GDR index has not the additivity property, it performs the indication by significantly increasing its relative value when distinct disturbances are present. Figure 9. Instantaneous Transient Disturbance ratio index ITD(t) of voltage disturbances combination. 50 Sensors 2021,21, 1348 Table 5. Summary of Power Quality Indices for case B. Sag Sag + Harmonics Harmonics Harmonics + Transients Transients T0(ms) 100.62 100.62 0 62.81 62.81 Vrms 200.30 201.51 231.36 232.14 230.79 THD 16.8966 20.2710 10.8804 13.6716 8.2782 TWD 1.2597 11.0999 10.8804 13.6705 8.2766 <ITD> 0.7170 10.0536 10.1025 12.2901 4.1376 GDR 1.0778 15.1118 10.1025 16.1499 5.4370 6. Conclusions The motivation for this work stemmed from the growing need for a more effective analysis of power quality in electrical systems and equipment. In this way, the main contribution of this research is the introduction of a novel wavelet-based single indicator, designated global disturbance ratio (GDR) that has been tested on real signals to address an integral assessment of the electrical network PQ. The objective of these test is to guarantee the applicability of such index to smart network sensors in particular, and to PQ monitoring in general. To this end, a PQ System Analyzer based on wavelet techniques is developed, turning out to be an effective device to verify the behavior of the proposed indices. The GDR is based on an instantaneous index ITD(t), which is also introduced, and considers two quality aspects of the electrical signal: steady-state power quality relative to harmonic level, and the non-stationary index relative to oscillatory transients or sudden amplitude changes in the signal. In the initial theoretical stage of signal analysis it is remarkable the mother wavelet selection procedure used to guarantee the most accurate frequency decomposition of the voltage signal under steady-state and non-stationary events. The applicability of the GDR and ITD(t) is illustrated for typical source-load configurations. The proposed PQIs have been tested on single-phase stationary and non-stationary complex signals, as well as three-phase balanced and unbalanced signals. Finally, the ITD(t) and GDR indices are a powerful tool for detecting and monitoring non-stationary signal components and they can extract relevant characteristics. In particular, the GDR index may be applied as the input of a classifier. In this sense, the authors are working on the physical implementation of an intelligent sensor based on a DSP, which provides the proposed PQIs. This guideline deserves future research. Author Contributions: M.D.B.-T. Conceived and developed the idea of this research, generated the data, performed the simulations, designed the whole structure of the experimental device, and wrote most of the paper; J.C.B. Contributed to the methodology and wrote some of the paper; C.Á.-A. contributed to the development of the experimental device and the simulations and wrote some of the paper. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the Universidad de Sevilla (VI Plan Propio de Investigacióny Transferencia) under grant 2020/00000596. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Conflicts of Interest: The authors declare no conflict of interest. 51 Sensors 2021,21, 1348 References 1. Montaño, J.-C.; Bravo, J.-C.; Borrás, M.-D. Joint Time–Frequency Analysis of the Electrical Signal. In Power Quality: Mitigation Technologies in a Distributed Environment; Moreno-Muñoz, A., Ed.; Springer London: London, UK, 2007; pp. 41–72, ISBN 978-1- 84628-772-5. 2. Chawda, G.S.; Shaik, A.G.; Shaik, M.; Padmanaban, S.; Holm-Nielsen, J.B.; Mahela, O.P.; Kaliannan, P. Comprehensive Review on Detection and Classification of Power Quality Disturbances in Utility Grid with Renewable Energy Penetration. IEEE Access 2020 , 8, 146807–146830. [CrossRef] 3. Sharma, A.; Rajpurohit, B.S.; Singh, S.N. A review on economics of power quality: Impact, assessment and mitigation. Renew. Sustain. Energy Rev. 2018,88, 363–372. [CrossRef] 4. Viciana, E.; Alcayde, A.; Montoya, F.; Baños, R.; Arrabal-Campos, F.; Manzano-Agugliaro, F. An Open Hardware Design for Internet of Things Power Quality and Energy Saving Solutions. Sensors 2019,19, 627. [CrossRef] 5. Artale, G.; Caravello, G.; Cataliotti, A.; Cosentino, V.; Di Cara, D.; Dipaola, N.; Guaiana, S.; Panzavecchia, N.; Sambataro, M.G.; Tinè, G. PQ and Harmonic Assessment Issues on Low-Cost Smart Metering Platforms: A Case Study. Sensors 2020 ,20, 6361. [CrossRef][PubMed] 6. Thongkhao, Y.; Pora, W. A low-cost Wi-Fi smart plug with on-off and Energy Metering functions. In Proceedings of the 2016 13th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Chiang Mai, Thailand, 28 June–1 July 2016; IEEE: Piscataway, NJ, USA; pp. 1–5. 7. Yaemprayoon, S.; Boonplian, V.; Srinonchat, J. Developing an innovation smart meter based on CS5490. In Proceedings of the 2016 13th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Chiang Mai, Thailand, 28 June–1 July 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1–4. 8. Sun, Q.; Li, H.; Ma, Z.; Wang, C.; Campillo, J.; Zhang, Q.; Wallin, F.; Guo, J. A comprehensive review of smart energy meters in intelligent energy networks. IEEE Internet Things J. 2015,3, 464–479. [CrossRef] 9. Albu, M.M.; Sănduleac, M.; Stănescu, C. Syncretic use of smart meters for power quality monitoring in emerging networks. IEEE Trans. Smart Grid 2016,8, 485–492. [CrossRef] 10. Sharon, D.; Montano, J.; Lopez, A.; Castilla, M.; Borras, D.; Gutierrez, J. Power Quality Factor for Networks Supplying Unbalanced Nonlinear Loads. IEEE Trans. Instrum. Meas. 2008,57, 1268–1274. [CrossRef] 11. Montano, J.; Leon, C.; Garcia, A.; Lopez, A.; Monedero, I.; Personal, E. Random Generation of Arbitrary Waveforms for Emulating Three-Phase Systems. IEEE Trans. Ind. Electron. 2012,59, 4032–4040. [CrossRef] 12. Jin, H.; Titus, A.; Liu, Y.; Wang, Y.; Han, Z. Fault Diagnosis of Rotary Parts of a Heavy-Duty Horizontal Lathe Based on Wavelet Packet Transform and Support Vector Machine. Sensors 2019,19, 4069. [CrossRef] 13. Gao, Z.; Cecati, C.; Ding, S.X. A survey of fault diagnosis and fault-tolerant techniques-part I: Fault diagnosis with model-based and signal-based approaches. IEEE Trans. Ind. Electron. 2015,62, 3757–3767. [CrossRef] 14. Borras, D.; Castilla, M.; Moreno, N.; Montano, J.C. Wavelet and neural structure: A new tool for diagnostic of power system disturbances. IEEE Trans. Ind. Appl. 2001,37, 184–190. [CrossRef] 15. Wen, H.; Guo, S.; Teng, Z.; Li, F.; Yang, Y. Frequency Estimation of Distorted and Noisy Signals in Power Systems by FFT-Based Approach. IEEE Trans. Power Syst. 2014,29, 765–774. [CrossRef] 16. Lopez, A.; Montano, J.; Castilla, M.; Gutierrez, J.; Borras, M.D.; Bravo, J.C. Power System Frequency Measurement Under Nonstationary Situations. IEEE Trans. Power Deliv. 2008,23, 562–567. [CrossRef] 17. Jeong, J.; Williams, W.J. Kernel design for reduced interference distributions. IEEE Trans. Signal Process. 1992 ,40, 402–412. [CrossRef] 18. Borrás, M.D.; Montaño, J.C.; Castilla, M.; López, A.; Gutiérrez, J.; Bravo, J.C. Voltage index for stationary and transient states. In Proceedings of the Mediterranean Electrotechnical Conference-MELECON, Valletta, Malta, 26–28 April 2010; University of Seville: Sevilla, Spain, 2010; pp. 679–684. 19. Borrás, M.D.; Bravo, J.C.; Montaño, J.C. Disturbance Ratio for Optimal Multi-Event Classification in Power Distribution Networks. IEEE Trans. Ind. Electron. 2016,63, 3117–3124. [CrossRef] 20. Bravo, J.C.; Borras, M.D.; Torres, F.J. Development of a smart wavelet-based power quality monitoring system. In Proceedings of the 2018 International Conference on Smart Energy Systems and Technologies (SEST), Sevilla, Spain, 10–12 September 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–6. 21. Lu, S.-D.; Sian, H.-W.; Wang, M.-H.; Liao, R.-M. Application of Extension Neural Network with Discrete Wavelet Transform and Parseval’s Theorem for Power Quality Analysis. Appl. Sci. 2019,9, 2228. [CrossRef] 22. Khokhar, S.; Asuhaimi, A.; Zin, M.; Memon, A.P.; Safawi Mokhtar, A. A new optimal feature selection algorithm for classification of power quality disturbances using discrete wavelet transform and probabilistic neural network. Measurement 2017 ,95, 246–259. [CrossRef] 23. Wang, J.; Xu, Z.; Che, Y. Power Quality Disturbance Classification Based on DWT and Multilayer Perceptron Extreme Learning Machine. Appl. Sci. 2019,9, 2315. [CrossRef] 24. Costa, F.B. Boundary Wavelet Coefficients for Real-Time Detection of Transients Induced by Faults and Power-Quality Disturbances. IEEE Trans. Power Deliv. 2014,29, 2674–2687. [CrossRef] 25. Bravo-Rodríguez, J.C.; Torres, F.J.; Borrás, M.D. Hybrid Machine Learning Models for Classifying Power Quality Disturbances: A Comparative Study. Energies 2020,13, 2761. [CrossRef] 52 Sensors 2021,21, 1348 26. Kumar, R.; Singh, B.; Shahani, D.T.; Chandra, A.; Al-Haddad, K. Recognition of Power-Quality Disturbances Using S-Transform- Based ANN Classifier and Rule-Based Decision Tree. IEEE Trans. Ind. Appl. 2015,51, 1249–1258. [CrossRef] 27. Shafiullah, M.; Abido, M.A.; Abdel-Fattah, T. Distribution grids fault location employing ST based optimized machine learning approach. Energies 2018,11, 2328. [CrossRef] 28. Wang, H.; Wang, P.; Liu, T. Power quality disturbance classification using the S-transform and probabilistic neural network. Energies 2017,10, 107. [CrossRef] 29. Reddy, M.V.; Sodhi, R. A modified S-transform and random forests-based power quality assessment framework. IEEE Trans. Instrum. Meas. 2018,67, 78–89. [CrossRef] 30. Tang, Q.; Qiu, W.; Zhou, Y. Classification of Complex Power Quality Disturbances Using Optimized S-Transform and Kernel SVM. IEEE Trans. Ind. Electron. 2020,67, 9715–9723. [CrossRef] 31. Heydt, G.T.; Fjeld, P.S.; Liu, C.C.; Pierce, D.; Tu, L.; Hensley, G. Applications of the windowed FFT to electric power quality assessment. IEEE Trans. Power Deliv. 1999,14, 1411–1416. [CrossRef] 32. Lin, T.; Domijan, A. On power quality indices and real time measurement. IEEE Trans. Power Deliv. 2005 ,20, 2552–2562. [CrossRef] 33. Biswal, M.; Dash, P.K. Estimation of time-varying power quality indices with an adaptive window-based fast generalised S-transform. IET Sci. Meas. Technol. 2012,6, 189. [CrossRef] 34. Thirumala, K.; Umarikar, A.C.; Jain, T. Estimation of Single-Phase and Three-Phase Power-Quality Indices Using Empirical Wavelet Transform. IEEE Trans. Power Deliv. 2015,30, 445–454. [CrossRef] 35. Jasi´nski, M.; Sikorski, T.; Kostyła, P.; Leonowicz, Z.; Borkowski, K. Combined cluster analysis and global power quality indices for the qualitative assessment of the time-varying condition of power quality in an electrical power network with distributed generation. Energies 2020,13, 2050. [CrossRef] 36. Morsi, W.G.; El-Hawary, M.E. Wavelet Packet Transform-Based Power Quality Indices for Balanced and Unbalanced Three-Phase Systems Under Stationary or Nonstationary Operating Conditions. IEEE Trans. Power Deliv. 2009,24, 2300–2310. [CrossRef] 37. Jo, S.; Son, S.; Park, J. On Improving Distortion Power Quality Index in Distributed Power Grids. IEEE Trans. Smart Grid 2013 ,4, 586–595. [CrossRef] 38. Granados-Lieberman, D. Global harmonic parameters for estimation of power quality indices: An approach for PMUs. Energies 2020,13, 2337. [CrossRef] 39. Shin, Y.; Powers, E.J.; Grady, M.; Arapostathis, A. Power quality indices for transient disturbances. IEEE Trans. Power Deliv. 2006 , 21, 253–261. [CrossRef] 40. Sahani, M.; Dash, P.K.; Samal, D. A real-time power quality events recognition using variational mode decomposition and online-sequential extreme learning machine. Meas. J. Int. Meas. Confed. 2020,157, 107597. [CrossRef] 41. Barros, J.; de Apráiz, M.; Diego, R.I. Power quality in DC distribution networks. Energies 2019,12, 848. [CrossRef] 42. Chang, S.; Wang, Q.; Hu, H.; Ding, Z.; Guo, H. An NNwC MPPT-Based energy supply solution for sensor nodes in buildings and its feasibility study. Energies 2019,12, 101. [CrossRef] 43. IEC 61000-4-7. Electromagnetic Compatibility (EMC)—Part 4-7: Testing and Measurement Techniques—General Guide on Harmonics and Interharmonics Measurements and Instrumentation, for Power Supply Systems and Equipment Connected Thereto; International Electrotechnical Commission: Geneva, Switzerland, 2008. 44. EN 50160: Voltage Characteristics of Electricity Supplied by Public Distribution Network; British Standards: London, UK, 2010. 45. IEEE Recommended Practice for Monitoring Electric Power Quality; IEEE Std 1159-2019 (Revision of IEEE Std 1159-2009); Institute of Electrical and Electronics Engineers: Piscataway, NJ, USA, 2019; pp. 1–98. 46. IEEE Standard Definitions for the Measurement of Electric Power Quantities under Sinusoidal, Nonsinusoidal, Balanced, or Unbalanced Conditions; IEEE Std 1459-2010 (Revision of IEEE Std 1459-2000); Institute of Electrical and Electronics Engineers: Piscataway, NJ, USA, 2010; pp. 1–50. 47. Morsi, W.G.; El-Hawary, M.E. Reformulating Power Components Definitions Contained in the IEEE Standard 1459–2000 Using Discrete Wavelet Transform. IEEE Trans. Power Deliv. 2007,22, 1910–1916. [CrossRef] 48. Gritli, Y.; Zarri, L.; Rossi, C.; Filippetti, F.; Capolino, G.; Casadei, D. Advanced Diagnosis of Electrical Faults in Wound-Rotor Induction Machines. IEEE Trans. Ind. Electron. 2013,60, 4012–4024. [CrossRef] 49. IEC 61000-4-30:2015. Electromagnetic Compatibility (EMC)—Part 4-30: Testing and Measurement Techniques—Power Quality Measurement Methods; International Electrotechnical Commission: Geneva, Switzerland, 2015. 53 sensors Article AnkFall—Falls, Falling Risks and Daily-Life Activities Dataset with an Ankle-Placed Accelerometer and Training Using Recurrent Neural Networks Francisco Luna-Perejón 1,2,*, Luis Muñoz-Saavedra 1,2, Javier Civit-Masot 1,2, Anton Civit 1,2,3 and Manuel Domínguez-Morales 1,2,3   Citation: Luna-Perejón, F.; Muñoz-Saavedra, L.; Civit-Masot, J.; Civit, A.; Domínguez-Morales, M. AnkFall—Falls, Falling Risks and Daily-Life Activities Dataset with an Ankle-Placed Accelerometer and Training Using Recurrent Neural Networks. Sensors 2021,21, 1889. https://doi.org/10.3390/s21051889 Academic Editors: Ki H. Chon and Marco Iosa Received: 25 January 2021 Accepted: 5 March 2021 Published: 8 March 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1Architecture and Computer Technology Department, ETSII-EPS, University of Seville, 41004 Sevilla, Spain; [email protected] (L.M.-S.); [email protected] (J.C.-M.); [email protected] (A.C.); [email protected] (M.D.-M.) 2Robotics and Technology of Computers Laboratory, University of Seville, 41004 Sevilla, Spain 3Research Institute of Computer Engineering (I3US), University of Seville, 41004 Sevilla, Spain *Correspondence: fl[email protected] Abstract: Falls are one of the leading causes of permanent injury and/or disability among the elderly. When these people live alone, it is convenient that a caregiver or family member visits them periodically. However, these visits do not prevent falls when the elderly person is alone. Furthermore, in exceptional circumstances, such as a pandemic, we must avoid unnecessary mobility. This is why remote monitoring systems are currently on the rise, and several commercial solutions can be found. However, current solutions use devices attached to the waist or wrist, causing discomfort in the people who wear them. The users also tend to forget to wear the devices carried in these positions. Therefore, in order to prevent these problems, the main objective of this work is designing and recollecting a new dataset about falls, falling risks and activities of daily living using an ankle-placed device obtaining a good balance between the different activity types. This dataset will be a useful tool for researchers who want to integrate the fall detector in the footwear. Thus, in this work we design the fall-detection device, study the suitable activities to be collected, collect the dataset from 21 users performing the studied activities and evaluate the quality of the collected dataset. As an additional and secondary study, we implement a simple Deep Learning classifier based on this data to prove the system’s feasibility. Keywords: accelerometer; deep learning; embedded system; fall detection; wearable; recurrent neural networks 1. Introduction Among most events related to the gait study, fall detection clearly stands out. These events can lead to severe injuries and sometimes chronic problems or even death. Its importance grows as users age, becoming one of the most important causes of morbidity and mortality worldwide among the elderly according to the World Health Organization (WHO) [ 1 ]. The obtained results estimate that around 28% to 35% of people over 65 fall at least one time per year. Additionally, this rate increments in people who suffered falls in the past. Key factors regarding falls include not only physical variables but also psychological aspects, such as fear of falling again. These aspects condition the gait of these persons, leading to higher falling risks. All these factors are correlated with the person’s way of walking. After a fall event, it is very important to have a quick response to avoid increasing the consequences of the fall. However, in many occasions, elderly people live alone or spend too much time alone. In addition, in exceptional situations, such as the current global pandemic caused by COVID-19, we must significantly reduce mobility and contact with this population sector, since it is the most sensitive to these diseases. This is one of the Sensors 2021,21, 1889. https://doi.org/10.3390/s21051889 https://www.mdpi.com/journal/sensors 55 Sensors 2021,21, 1889 reasons why automatic fall detection systems have become very important in recent years and especially in the current pandemic situation. There are multiple fall detection system types that achieve good results in these situations—solutions integrated into the user’s home (within the “smart home” domain) using visual sensors, telecare solutions with user interaction, mobile applications or wearable systems, among others. As has already been explained, the responses of these systems occur only after the user falls, acting as a rapid intervention mechanism when these events occur. However, in order to continuously monitor and prevent these falls, it is very interesting to detect possible risk of fall situations, although these events are much more complex to detect. Thus, it is very important to analyze the efficiency of the system based on these three states—fall, falling risk and activity of daily-living (ADL). In order to design a new fall-detection system, it is important to collect a large amount of data to be able to use it during the system testing phase. For this purpose, researchers usually use public datasets to save time and resources in the design and implementation of the data gathering device as this eliminates or greatly reduces the workload for the collection, filtering and labeling of the required data. Also, the more data we have, the more reliable the system results will be. As an important point, it should be noted that there are several sensors and/or systems that can be used to detect falls. However, the most widely used sensor that is integrated in almost all fall detection systems is the accelerometer. Some works add other complementary sensors, like gyroscopes (Gyr) or magnetometers (Mag) to filter the information, but the accelerometer (Accel) is the most widely used sensor (sometimes integrated in a wearable independent device and others as part of a smartphone). In previous works, information about fall events was collected using several users, and the processed and labelled data were made publicly available for researchers as datasets. Most of these datasets are linked with published papers that describe the collecting process. Several of these datasets are summarized in Table 1. As can be observed, there are two main tendencies—using a smartphone or a wearable device. However, almost all published works use an accelerometer. Another important point shown in Table 1is the sensor (or device) location. Of these works, 31% place the device in the pocket (usually those that use a smartphone), another 31% place it on the waist, 6% on the chest and 6% on the wrist. Apart from this, two works place several devices in different locations and only one places it on the lower limb. With this summary, it can be observed that 87.5% of the analyzed datasets locate the device in the upper half of the body. Only one work locates the sensor at the bottom, and another uses multiple sensors (locating one of them on the ankle). The long-term objective of this work is to implement an integrated system to analyze events associated with gait, merging the fall detection system with other gait anomaly detection systems. To achieve this goal, the fall detection system must be near the foot, in order to be able to integrate this new system with the one developed in previous works [ 2 , 3 ], where the pressure sensors that measured the gait trend were located in the footwear insole. For this reason, in order to study the viability of this type of systems, we need a dataset that collects the information about falls and activities of daily living (ADL) from the foot or from the ankle. Moreover, as detailed previously, it is very important to detect not only falls, but also falling risks. As observed in Table 1, only two of the works include falling risk activities due to the difficulties regarding data collection. But, for our case, it would be very important to include these events. So, attending to the aforementioned problems and the detailed future integration work, we need a dataset obtained from the ankle or from the foot using an accelerometer and including three types of events: Falls, Falling risks and Activities of daily living (ADL). This dataset creation would set a precedence in the field of fall detectors and could be of great help to other researchers who work in this field. This dataset could be combined 56 Sensors 2021,21, 1889 in future works with classic Machine Learning (ML) or Deep Learning (DL) techniques to automatically classify the events related to the collected data. These techniques come from the field of Artificial Intelligence (AI), and they have had (and continue to have) great success when automatically extracting the meta-characteristics of large amounts of data in order to perform these classifications correctly. Not only in this area, but also in many others, both ML and DL are used assiduously to solve this type of problem; and as a result of this fact, several studies have emerged with remarkable results [4–8]. Table 1. Datasets summary. Dataset Year Participants Sensors Location Classes #Activities Frank et al. [9] 2010 16 Accel Waist Pasive, ADL, Risk, Fall 5 Kerdegari et al. [10] 2012 50 Accel Waist ADL, Fall 20 Anguita et al. [11] 2013 30 Smartphone (Accel) Waist Pasive, ADL 8 Medrano et al. [12] 2014 20 Smartphone (Accel) Pocket ADL, Fall 8 Ojetola et al. [13] 2015 42 Accel, Gyr Chest ADL, Risk, Fall 15 Fall-MobileGuard [14] 2015 20 Shimmer2R (Accel) Pocket ADL, Fall 29 Vilarinho et al. [15] 2015 3 Smartwatch (Accel) Hand ADL, Fall 19 Wertner et al. [16] 2015 5 Smartwatch (Accel, Gyr) Pocket ADL, Fall 14 MobiAct [17] 2016 57 Smartwatch (Accel, Gyr) Pocket ADL, Fall 13 UniMiB SHAR [18] 2017 30 Smartwatch (Accel) Pocket ADL, Fall 17 UMAFall [19] 2017 17 Smartwatch, Tags (Accel, Gyr, Mag) Ankle, Waist, Wrist, Chest ADL, Fall 11 SisFall [20] 2017 38 Accel, Gyr Waist ADL, Fall 33 Rescio et al. [21] 2018 15 EMG Lower limb muscles ADL, Fall 6 Quadros et al. [22] 2018 22 Accel, Gyr and Mag Wrist ADL, Fall 12 FallDroid [23] 2018 20 Smartphone (Accel) Waist ADL, Fall 19 UP-Fall [24] 2019 17 Wearables (Accel), EEG, Cameras, Context-aware Wrist, Neck, Pocket, Waist, Ankle ADL, Fall 11 Clearly, the application of these technologies in the field of e-Health, both in the analysis of physiological signals and as diagnostic aid systems with medical images, has an enormous impact and helps to significantly reduce the workload of healthcare professionals. Several works related to this area can be found [ 25 – 28 ], and currently we can even find some interesting applications related to the detection of COVID-19 [ 29 , 30 ]. Due to the rise of these systems, it is interesting to carry out a preliminary dataset study using DL techniques in order to evaluate its quality. Thus, summarizing the aforementioned, in this work one main fundamental objective is established: designing and developing a dataset for the detection of fall events, falling 57 Sensors 2021,21, 1889 risk events and activities of daily living (ADL) using a wearable device as data collector (based on low-power microcontroller, accelerometer and Bluetooth BLE transmission), located on the user’s ankle. As a complementary objective, but not one of the main objectives, we include a study regarding the other published datasets (16 in total) in order to extract their main characteristics and their most common activities. This information will be used to determine the activities used for our own dataset. It is important to mention that, although this is not one of the main objectives, this initial study will help to determine the most suitable activities to include in the dataset; thus, this study will help to accomplish the main objective. Finally, as an additional objective, a dataset classification study is carried out using DL techniques based on Recurrent Neural Networks (RNN). This final objective will determine the dataset quality and, for this reason, a deep study about the different parameters used for classification purposes is carried out. However, it is important to clarify that, as this is not one of our main objectives, we are not looking for the best classification result, but just for a good classifier that justifies the correct samples’ labelling. The results that will be shown regarding this classification can be (and will be) optimized using more complex RNNs (like in [31]) in future works after certifying the correctness of the dataset. About the device location, the ankle was chosen instead of the foot for the ease and comfort when integrating the designed prototype (larger than the final device). However, the final and future device will be miniaturized and located in the footwear in order to be combined with the gait pathology detection system implemented and presented in [3]. The rest of the paper is divided as follows—first, in the ’Materials and Methods’ Section, the characteristics of the collected dataset and the device used for this purpose are presented, as well as the recurrent neural network implemented to test the dataset quality. Next, the results obtained after training and testing the dataset collected are detailed and explained in the Results and Discussion Section. Finally, conclusions are presented. 2. Materials and Methods In this section, we will present in depth the collected dataset—we will discuss the users who have participated in the data collection as well as detail the activities selected to be carried out by all of them based mainly on the analysis of previous public datasets. In addition, we will emphasize the device designed for this task—we will detail the hardware design, its firmware and the user application that receives and stores the information. Last, but not least, we will detail the selected Recurrent Neural Network architectures used to test the quality of the dataset, as well as all the parameter configurations that have been tested. 2.1. Ankfall Dataset The dataset developed for this work has been called “AnkFALL” due to the location of the data recorder device (ankle), and it is publicly available to any researcher (https: //github.com/mjdominguez/AnkFall) taking into account the limitations indicated in the information web page (accessed on 8 March 2021) . This dataset is stored with version control. In this way, any modification of its content would provoke the creation of a new version; so that, when researchers download the dataset, they will be able to know which version they are using and thus be able to compare themselves with other works. At the time of publication of this work, the version of the dataset is “v1”, and it will be increased numerically as future changes are made. This section will detail the users who have participated in the data collection, the activities carried out by each one of them and the device designed for data collection. 2.1.1. Activity Set and Participants In order to consider the best activities for our dataset, it is important to analyze the sixteen previous studies related to the actual datasets that contain activities related to gait. All 58 Sensors 2021,21, 1889 In previous studies, we used these recurrent layers with accelerometer signals taken at the waist (using the information of other datasets), obtaining high effectiveness results and real-time classifications using low-power microcontrollers [ 3 ]. These layers implement a memory cell that retains relevant information from the analyzed sequence section, to use it in the analysis of the full sequence and, finally, to provide appropriate information for the classification problem. In summary, their operation is detailed below: • LSTM cell: this cell consists of an input gate, a forget gate and an output gate. The first gate adds new information to the cell from the sequence sample in the current instant of time t and the outputs of the same layer processing for the previous instant t− 1. With the same inputs, the forget gate determines the non-necessary data for the analysis of rest of the sequence. Finally, the output gate passes information from the memory that considers relevant as input to the layer processing the next instant t+ 1. • GRU cell: it has only two gates (no forget gate is used), so it can add (update gate) and remove (reset gate) information from the cell, allowing all stored information to be used by the neural network throughout the sequence analysis. In this study two main architectures were used (see Figure 4). Both of them contain a single recurrent layer and a dense layer with four nodes. A softmax activation function is used to normalize the results in a probability distribution; and, thanks to this, both the loss calculation and the 4-class classification results can be performed. Additionally, we use a batch normalization layer during the training step to normalize the input data, allowing a faster convergence of the system. The two architectures differ in the type of RNN layer used—the first one is based on LSTM cells while the other one uses GRU cells. Figure 4. Diagram of the Gated RNN architectures assessed. Data Segmentation and Labelling Criteria In order to adapt each registered activity to the RNN models, we established a fixed temporal window of 64 samples, which, considering an acquisition frequency of 50 Hz, is equivalent to 1.28 s. This window is appropriate to contemplate a fall event or even risk situations [ 34 ]. Each activity was split into blocks with the same window length of 64 and was labelled with a unique category. The criteria used for labelling each block consisted of assigning it to the most relevant class whose occurrence percentage exceeds an established threshold (see Figure 5). If none of the thresholds is reached, the segment is classified as background (BKG). In the first step of this study, we consider different threshold values for each relevant category and analyze the training results to obtain the best threshold values. In a second step, grid search is used for model optimization. The parameters considered were the number of nodes and dropout value for the recurrent layer, as well as learning rate and batch size used during training. All these tests and the obtained results will be detailed in the next section. 65 Sensors 2021,21, 1889 Figure 5. Labelling criteria based on occurrence percentage in each block. 3. Results and Discussion After analyzing all the datasets developed in recent years related to fall events and selecting the activities that must be collected for the new dataset, its final distribution is shown below. In addition, as already indicated above, we consider that it is necessary to test our dataset using a DL classification system. In this way, we will assess whether the collected dataset is useful to future fall detection systems studies. 3.1. Collected Dataset While recording the dataset, three repetitions of each activity were performed. However, due to the COVID-19 pandemic lockout, some activities could not be registered. It mainly affected activities 5 (going up and down stairs) and 8 (walking while dizzy) because they needed special considerations during the preparation. Thus, the data recollection for these activities was delayed until the other activities were recorded. While activity 8 (walking while dizzy) was performed by 10 users, activity 5 (going up and down stairs) could not be performed by any user. So, finally, eleven activities were recorded. Moreover, as activity 5 is considered an ADL (activity of daily living), the final dataset is more balanced than our initial expectations as it contains 4 ADL activities, 4 fall activities and 3 falling risk activities. Additionally, some participants did not carry out some activities for security reasons or at their own request and, in some cases, some errors were detected after some recordings (in particular with two activities for participant seventeen); all these reasons reduced the final amount of records. Figure 6illustrates the process to obtain the final dataset. Finally, the total set contains 615 records. So, after recording the AnkFALL dataset, it is important to compare it with the datasets presented in the Introduction Section. However, as there are too many differences among them, only datasets that use a wearable device with an accelerometer (not a smartphone or smartwatch) and that include fall events are taken into account. The comparison is presented on Table 4. 66 Sensors 2021,21, 1889 Figure 6. Resulting dataset. Table 4. Comparison between AnkFALL and the most similar datasets (from Table 1). The last column indicates the number of activities from each type (in order: activities of daily living (ADL), Risk, Fall). Dataset Year #Users Sensors Location Classes #Activities Balance Frank et al. [9] 2010 16 Accel Waist Pasive, ADL, Risk, Fall 5 3-1-1 Kerdegari et al. [10] 2012 50 Accel Waist ADL, Fall 20 17-0-3 Ojetola et al. [13] 2015 42 Accel, Gyr Chest ADL, Risk, Fall 15 8-1-6 SisFall [20] 2017 38 Accel, Gyr Waist ADL, Fall 33 29-0-4 Quadros et al. [22] 2018 22 Accel, Gyr and Mag Wrist ADL, Falls 12 11-0-1 AnkFALL 2020 21 Accel Ankle BKG, ADL, Risk, Fall 11 4-3-4 As can be observed in Table 4, AnkFALL registers a total number of participants and a number of activities slightly lower than average (around 30 and 19, respectively). As detailed before, the dataset is continuously growing and, in the near future, the total number of participants will increase; however, due to the pandemic lockout, no participant record has been registered since March 2020. Regarding the number of activities, AnkFALL seems to have very few, but the dataset it was designed to be as balanced as possible—as it is shown in the last column of Table 4, we can observe some goodness about AnkFALL: • AnkFALL is the most balanced dataset among the existing ones according to the number of activities: around 36%, 28% and 36% of ADL, Risks and Falls, respectively. If we do not take into consideration AnkFall, the more balanced dataset is the one presented by Ojetola et al. [ 13 ] with 53%, 7% and 40% of ADL, Risks and Falls, respectively. • Only two datasets from Table 1register more Fall events than AnkFALL—the one from Ojetola et al. [ 13 ] with six different Fall activities and FallDroid [ 23 ], which has no Risk activities and uses a smartwatch. 67 Sensors 2021,21, 1889 • No other dataset has more Risk activities than AnkFALL—the datasets that have Risk activities only register one or two types. Thus, the two most important things that make AnkFALL useful for researchers is that it has several Risk activities records, and it is clearly the most balanced available dataset. These two characteristics are very useful when implementing ML classification systems due to the importance of balancing the training data for obtaining good classification results without overtraining one of the classes. The internal data structure of AnkFall dataset can be observed in Figure 7. Figure 7. AnkFall internal structure: ( a ) Folder structure; ( b ) Data collected from user 21 during activity 10, repetition 1; ( c ) Labels for each sample of data collected during activity “(b)”. 3.2. Testing the Dataset Although the main goal of this work is already achieved, it is very interesting to perform a first classification study in order to check that the information recorded and the applied pre-processing techniques are correct. For this purpose, the Hold-Out technique has been applied, dividing the dataset in two subsets—one subset was used for training and the remaining was used for evaluation. The distribution was carried out in such a way that each subset contained data from different users, to avoid bias. Data from 5 users were randomly selected, avoiding choosing the users who had carried out the least number of activities. In the dataset segmentation process, we used a 64-sample temp window, as mentioned above, and a 25% displacement of the sample size, which corresponds to 16 temporal samples and applied this approach as a data augmentation technique: this means that there is an overlapping of 75% of the information between two consecutive temporal windows. Data distribution is shown in Table 5. 68 Sensors 2021,21, 1889 Table 5. Participant distribution for each subset using the hold-Out method. Subsets Users # Activities Train 1, 2, 3, 4, 6, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 463 Validation 5, 8, 9, 10, 21 152 We compared the effectiveness of the classification system using different and wellknown metrics: sensitivity (also known as recall), specificity, precision and F1-score [ 35 ]. This last metric is the harmonic mean of precision and sensitivity. Those metrics are presented in the next equations: Specificity =∑ c TNc TNc+FPc,c∈classes (1) Precision =∑ c TPc TPc+FPc,c∈classes (2) Sensitivity =∑ c TPc TPc+FNc,c∈classes (3) F1-score =2∗precision ∗sensitivity precision +sensitivity. (4) About those metrics: • Specificity: proportion of “true negative” values in all cases that don’t belong to this class (see Equation (1)). • Precision: proportion of “true positive” values in all cases that have been classified as it (see Equation (2)). • Sensitivity (or Recall): proportion of “true positive” values in all the cases that belong to this class (see Equation (3)). • F1: It considers both the precision and the sensitivity (recall) of the test to compute the score. It is the harmonic mean of both parameters (see Equation (4)). In the first stage of the study, the best threshold values for block labelling were analysed. We considered the values shown in Table 6. Table 6. Occurrence thresholds values analyzed for each class. Threshold Set of Values ADL 0.3, 0.4, 0.5, 0.6, 0.7 RISK 0.2, 0.3, 0.4, 0.5 FALL 0.1, 0.2, 0.3, 0.4 For training our classification system, we used the best architecture considered in a previous work [ 31 ]—this architecture consists, as detailed in the previous section, of a recurrent layer followed by a dense layer and a softmax activation function. The resulting best values for thresholds were 30% for Fall events, 20% for Falling risk events and 30% for ADL, with a mean F1-score up to 0.75 and a standard deviation 0.08 after three training repetitions. The dataset distribution using the considered thresholds can be consulted in Table 7. It can be seen how, despite trying to homogenize the number of samples for each type of event, due to the short duration of risk situations and fall events, an imbalance in the number of dataset samples can be observed. 69 Sensors 2021,21, 1889 Table 7. Dataset distribution for each subset with best labelling thresholds. Samples Per Class Subsets Total ADL BKG Risk Fall Train 9099 2064 4587 1165 1283 Validation 3172 837 1500 331 504 The model optimization was carried out with a grid search considering the parameter values shown in Table 8. The parameters consisted of the number of nodes of the recurrent layer, dropout rate, learning rate and batch size. This range of values was selected based on the parameters considered in previous studies [31,34]. Table 8. Grid search values for exhaustive parameters optimization. Parameters Set of Values for Grid Search Learning rate 0.0005, 0.001, 0.0015, 0.002 Batch size 32, 48, 64 Number of nodes 24, 32, 40 Dropout 0, 0.2, 0.35 After carrying out all the training studies with all the possibilities for each parameter, the best trained models were obtained and are shown in Table 9. In order to check the quality of these models, this table shows the values of the metrics described in Equations (1)–(4) . As can be observed, two possibilities are shown in this table based on the two network types considered in this work—the first one contains LSTM units, and the second one contains GRU units. Table 9. Best results obtained after grid search optimization. RNN RNN Learn. Batch RNN Architecture Nodes Rate Size Drop. Precision Specificity Sensitivity F1-Score One LSTM layer 32 0.002 32 0.2 0.780 0.928 0.768 0.774 One GRU layer 40 0.002 32 0.2 0.762 0.924 0.771 0.766 If we analyze the results shown in Table 9, we can obtain the next conclusions for each metric: • Specificity: we can see values over 92% that denote a high rate of true negatives among the total amount of true negatives and false positives. So, in general, this system classifies correctly the values that do not belong to each class. • Precision and Sensitivity: these values are lower than the specificity, however both have similar values around 77–78%. These results are not bad at all, but the important difference between the specificity and these two metrics indicate that the system does not behave in the same way with all classes. Maybe one of the four classes has quite worse results than the others. In fact, if we study the system deeply, we can check that the class RISK has much worse results than the others due to two main reasons: the difficulty of distinguishing it and the alternation between RISK, ADL and BKG during the Falling risk activities. This aspect will be studied later. • F1-score: As this metric is obtained from precision and sensitivity, it is normal that its value is similar to them. The main conclusion obtained with this value is the same that was already obtained for the previous metrics. 70 Sensors 2021,21, 1889 After presenting all the metrics, we can observe that the obtained “specificity” values are high (92–93%). This fact indicates that the system classifies the values that not belong to each class correctly. This means that the different classes can be easily differentiated and, thus, the dataset collecting and labeling processes are proven to be correct. So, this is a good metric to evaluate the quality of the collected dataset. Regarding the “precision“ metric, it depends on each class independently. But, thanks to the good specificity results, we can assume that, using more complex RNNs the classification results can be improved. So, in order to corroborate the results predicted previously by evaluating the metrics, it is important to study each class independently for the two neural networks (LSTM and GRU from Table 9) selected after our initial study. For this purpose, the confusion matrices are shown in Figure 8. This figure reveals the predicted main obstacle to be faced in the detection problem— the correct distinction of falling risk activities from activities of daily living (ADL). However, if we focus our attention on the other classes, their results are high, obtaining an 87% success rate when classifying falls. These results demonstrate that the information recorded for the AnkFALL dataset is very useful for being used in fall detection studies. The problem concerning the RISK class is not new as, in a previous work [ 31 ], a RISK class was included in the SisFALL dataset, and the conclusions were similar—the falling risk events are much more difficult to distinguish than any other event type. LSTM GRU Figure 8. Confusion matrices for best LSTM and GRU models. Each specific box in the confusion matrix represents the percentage of samples of the class indicated by the row that have been classified as the class indicated in the column. Concerning the system ROC curves (see Figure 9), they reveal the same problem again: RISK class obtains worse results than the others. However, the area under the curve is high, revealing that, in binary classifications of the One-vs-All type, the system has the ability to distinguish each event class individually. The curves corresponding to ADL and RISK are those with the lowest area under the curve, 90% and 80% respectively, which is correctly correlated with the results shown in the confusion matrices. Regarding the RISK class, a stagnation can be seen from a sensitivity value of 0.75, which reveals that there are samples of this class which are very complex to identify. These samples are mainly those corresponding to activity 8 (walking dizzy); so, it would be interesting to eliminate this activity from a future training study in order to corroborate these preliminary conclusions. 71 Sensors 2021,21, 1889 Figure 9. ROC curves for best LSTM and GRU hmodels. 4. Conclusions In this work, a new dataset called AnkFALL is presented. It contains labelled information from 21 participants performing 11 activities, including activities of daily-living (ADL), Falls and Falling risk activities. These activities have been carefully selected by performing a study of the different available datasets and taking into account that one of the main objectives is to obtain a dataset that contains all types of activities and that is as balanced as possible. According to the explanations given in the introduction, the results of this work can be used in healthcare and fitness areas. Our main focus area is healthcare, and more precisely telecare services for older people who use devices to alert about emergency situations (mainly falls). Using a device placed in the footwear may prevent forgetfulness and improve wearing comfort and, thanks to this dataset, researchers can start to develop footwear-placed fall detectors. Although this is our main interest area, these principles can be easily applied to the fitness field too. To record the dataset, a personalized wearable device, based on a low-power microcontroller, an accelerometer and a Bluetooth low energy interface, has been designed and implemented to transmit the information from the user’s ankle; on the other hand, a computer application has been implemented for the reception, visualization and storage of the received data. The labelling task has been thoroughly carried out by checking the received information sample by sample with the synchronized videoclips recorded while performing the activities. The final result is the first dataset of its kind that collects information from the ankle, and it is also the most balanced fall-detection dataset among of all the datasets currently described in the literature. Also, this dataset includes a type of activity (falling risk) that is only available in a small subset of the available datasets. To verify the quality of the dataset, a deep learning classifier system based on recurrent neural networks has been designed and implemented for the classification of the four classes in this dataset—BKG, ADL, Fall and Risk. A detailed study is carried out with multiple variations of thresholds, learning rates, batch sizes, number of nodes and dropout; obtaining, in combination, more than 600 different neural networks that have been trained for this purpose. The best result for each parameter is presented and the final classification results are shown. The obtained specificity values (between 92% and 93% depending on the implementation) indicate that the system classifies correctly the values that do not belong to each class. This means that the different classes can be differentiated easily, so the quality of the dataset is proven by this result. Regarding the precision, it depends on each class but, thanks to the specificity results, we can assume that, using more complex RNNs the classification results can be improved. 72 Sensors 2021,21, 1889 Moreover, the classification results show that the main difficulty is detecting falling risk events—this is mainly due to the speed at which they occur and the combination with other ADL activities; not surprisingly, most of the datasets do not include this type of activities and most of the research studies do not include them either. For future versions this dataset needs to be expanded with a larger number of participants. Moreover, after testing the dataset, we have observed that the information collected for one of the activities has several errors and this affects the classification results; thus, in the next versions of the dataset we need to evaluate whether the instructions given for the participants were clear enough and if this activity is essential or not. Finally, in order to obtain a better adequacy of the activities collected for the dataset, we have started to apply “Explainable Deep Learning” techniques to the collected data; we hope that these techniques will help us improve the next versions of the dataset. Author Contributions: Conceptualization, F.L.-P. and M.D.-M.; methodology, F.L.-P. and M.D.-M.; software, F.L.-P. and L.M.-S.; validation, F.L.-P., L.M.-S. and J.C.-M.; formal analysis, F.L.-P. and J.C.-M.; investigation, F.L.-P., L.M.-S. and M.D.-M.; resources, F.L.-P. and M.D.-M.; data curation, L.M.-S. and J.C.-M..; writing—original draft preparation, F.L.-P. and M.D.-M.; writing—review and editing, L.M.-S., J.C.-M. and A.C.; visualization, F.L.-P., L.M.-S. and J.C.-M.; supervision, M.D.-M. and A.C.; project administration, M.D.-M. and A.C.; funding acquisition, M.D.-M. and A.C. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the Spanish Agencia Estatal de Investigación (AEI) project MINDROB: “Percepción y Cognición Neuromórfica para Actuación Robótica de Alta Velocidad” (PID2019-105556GB-C33/AEI/10.13039/501100011033). Institutional Review Board Statement: Ethical review and approval were exempted for this study, because all the participants are part of the research group or relatives of some of its members, being aware of the risk and participating completely voluntarily. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: Public available AnkFall Dataset on https://github.com/mjdominguez/ AnkFall. Acknowledgments: We want to thank the Telefonica Chair “Intelligence in Network”, but also the Research Group “TEP-108: Robotics and Computer Technology” from Universidad de Sevilla (Spain) for using its resources. Conflicts of Interest: The authors declare no conflict of interest. References 1. World Health Organization: Ageing and Life Course Unit. WHO Global Report on Falls Prevention in Older Age; World Health Organization: Geneva, Switzerland, 2008. 2. Domínguez-Morales, M.J.; Luna-Perejón, F.; Miró-Amarante, L.; Hernández-Velázquez, M.; Sevillano-Ramos, J.L. Smart Footwear Insole for Recognition of Foot Pronation and Supination Using Neural Networks. Appl. Sci. 2019,9, 3970. [CrossRef] 3. Luna-Perejón, F.; Domínguez-Morales, M.; Gutiérrez-Galán, D.; Civit-Balcells, A. Low-Power Embedded System for Gait Classification Using Neural Networks. J. Low Power Electron. Appl. 2020,10, 14. [CrossRef] 4. Muñoz-Saavedra, L.; Luna-Perejón, F.; Civit-Masot, J.; Miró-Amarante, L.; Civit, A.; Domínguez-Morales, M. Affective State Assistant for Helping Users with Cognition Disabilities Using Neural Networks. Electronics 2020,9, 1843. [CrossRef] 5. Gutierrez-Galan, D.; Dominguez-Morales, J.P.; Cerezuela-Escudero, E.; Rios-Navarro, A.; Tapiador-Morales, R.; Rivas-Perez, M.; Dominguez-Morales, M.; Jimenez-Fernandez, A.; Linares-Barranco, A. Embedded neural network for real-time animal behavior classification. Neurocomputing 2018,272, 17–26. [CrossRef] 6. Saleem, M.H.; Potgieter, J.; Arif, K.M. Plant Disease Classification: A Comparative Evaluation of Convolutional Neural Networks and Deep Learning Optimizers. Plants 2020,9, 1319. [CrossRef] 7. Chen, C.H.; Kung, H.Y.; Hwang, F.J. Deep learning techniques for agronomy applications. Agronomy 2019,9, 142. [CrossRef] 8. Wang, J.; Ma, Y.; Zhang, L.; Gao, R.X.; Wu, D. Deep learning for smart manufacturing: Methods and applications. J. Manuf. Syst. 2018,48, 144–156. [CrossRef] 9. Frank, K.; Vera Nadales, M.J.; Robertson, P.; Pfeifer, T. Bayesian recognition of motion related activities with inertial sensors. In Proceedings of the 12th ACM International Conference Adjunct Papers on Ubiquitous Computing-Adjunct, Copenhagen, Denmark, 26–29 September 2010; pp. 445–446. 73 Sensors 2021,21, 1889 10. Kerdegari, H.; Samsudin, K.; Ramli, A.R.; Mokaram, S. Evaluation of fall detection classification approaches. In Proceedings of the 2012 4th International Conference on Intelligent and Advanced Systems (ICIAS2012), Kuala Lumpur, Malaysia, 12–14 June 2012; Volume 1, pp. 131–136. 11. Anguita, D.; Ghio, A.; Oneto, L.; Parra, X.; Reyes-Ortiz, J.L. A Public Domain Dataset for Human Activity Recognition Using Smartphones; Esann: Midlothian, VA, USA, 2013. 12. Medrano, C.; Igual, R.; Plaza, I.; Castro, M. Detecting falls as novelties in acceleration patterns acquired with smartphones. PLoS ONE 2014,9, e94811. [CrossRef] 13. Ojetola, O.; Gaura, E.; Brusey, J. Data set for fall events and daily activities from inertial sensors. In Proceedings of the 6th ACM Multimedia Systems Conference, Portland, OR, USA, 18–20 March 2015; pp. 243–248. 14. Fortino, G.; Gravina, R. Fall-MobileGuard: A smart real-time fall detection system. In Proceedings of the 10th EAI International Conference on Body Area Networks, Sydney, Australia, 28–30 September 2015; ICST (Institute for Computer Sciences, Social- Informatics and Telecommunications Engineering: Brussels, Belgium, 2015; pp. 44–50. 15. Vilarinho, T.; Farshchian, B.; Bajer, D.G.; Dahl, O.H.; Egge, I.; Hegdal, S.S.; Lønes, A.; Slettevold, J.N.; Weggersen, S.M. A combined smartphone and smartwatch fall detection system. In Proceedings of the 2015 IEEE International Conference on Computer and Information Technology, Ubiquitous Computing and Communications, Dependable, Autonomic and Secure Computing, Pervasive Intelligence and Computing, Liverpool, UK, 26–28 October 2015; pp. 1443–1448. 16. Wertner, A.; Czech, P.; Pammer-Schindler, V. An open labelled dataset for mobile phone sensing based fall detection. In Proceedings of the 12th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, Coimbra, Portugal, 22–24 July 2015; pp. 277–278. 17. Vavoulas, G.; Chatzaki, C.; Malliotakis, T.; Pediaditis, M.; Tsiknakis, M. The MobiAct Dataset: Recognition of Activities of Daily Living using Smartphones. In Proceedings of the ICT4AgeingWell, Rome, Italy, 21–22 April 2016; pp. 143–151. 18. Micucci, D.; Mobilio, M.; Napoletano, P. Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Appl. Sci. 2017,7, 1101. [CrossRef] 19. Casilari, E.; Santoyo-Ramón, J.A.; Cano-García, J.M. Umafall: A multisensor dataset for the research on automatic fall detection. Procedia Comput. Sci. 2017,110, 32–39. [CrossRef] 20. Sucerquia, A.; López, J.; Vargas-Bonilla, J. SisFall: A fall and movement dataset. Sensors 2017,17, 198. [CrossRef] 21. Rescio, G.; Leone, A.; Siciliano, P. Supervised machine learning scheme for electromyography-based pre-fall detection system. Expert Syst. Appl. 2018,100, 95–105. [CrossRef] 22. de Quadros, T.; Lazzaretti, A.E.; Schneider, F.K. A movement decomposition and machine learning-based fall detection system using wrist wearable device. IEEE Sens. J. 2018,18, 5082–5089. [CrossRef] 23. Shahzad, A.; Kim, K. FallDroid: An automated smart-phone-based fall detection system using multiple kernel learning. IEEE Trans. Ind. Inform. 2018,15, 35–44. [CrossRef] 24. Martínez-Villaseñor, L.; Ponce, H.; Brieva, J.; Moya-Albor, E.; Núñez-Martínez, J.; Peñafort-Asturiano, C. UP-fall detection dataset: A multimodal approach. Sensors 2019,19, 1988. [CrossRef] 25. Civit-Masot, J.; Domínguez-Morales, M.J.; Vicente-Díaz, S.; Civit, A. Dual Machine-Learning System to Aid Glaucoma Diagnosis Using Disc and Cup Feature Extraction. IEEE Access 2020,8, 127519–127529. [CrossRef] 26. Dominguez-Morales, J.P.; Jimenez-Fernandez, A.F.; Dominguez-Morales, M.J.; Jimenez-Moreno, G. Deep neural networks for the recognition and classification of heart murmurs using neuromorphic auditory sensors. IEEE Trans. Biomed. Circuits Syst. 2017 , 12, 24–34. [CrossRef] 27. Sahoo, A.K.; Pradhan, C.; Das, H. Performance evaluation of different machine learning methods and deep-learning based convolutional neural network for health decision making. In Nature Inspired Computing for Data Science; Springer: Berlin/Heidelberg, Germany, 2020; pp. 201–212. 28. Sahoo, A.K.; Pradhan, C.; Barik, R.K.; Dubey, H. DeepReco: Deep learning based health recommender system using collaborative filtering. Computation 2019,7, 25. [CrossRef] 29. Civit-Masot, J.; Luna-Perejón, F.; Domínguez Morales, M.; Civit, A. Deep learning system for COVID-19 diagnosis aid using X-ray pulmonary images. Appl. Sci. 2020,10, 4640. [CrossRef] 30. Duran-Lopez, L.; Dominguez-Morales, J.P.; Corral-Jaime, J.; Vicente-Diaz, S.; Linares-Barranco, A. COVID-XNet: A custom deep learning system to diagnose and locate COVID-19 in chest X-ray images. Appl. Sci. 2020,10, 5683. [CrossRef] 31. Luna-Perejón, F.; Domínguez-Morales, M.J.; Civit-Balcells, A. Wearable Fall Detector Using Recurrent Neural Networks. Sensors 2019,19, 4885. [CrossRef][PubMed] 32. Gao, C.; Neil, D.; Ceolini, E.; Liu, S.C.; Delbruck, T. DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA, 25–27 February 2018; ACM: New York, NY, USA, 2018; pp. 21–30. [CrossRef] 33. Yu, S. Residual Learning and LSTM Networks for Wearable Human Activity Recognition Problem. In Proceedings of the 2018 37th Chinese Control Conference (CCC), Wuhan, China, 25–27 July 2018; pp. 9440–9447. 34. Musci, M.; De Martini, D.; Blago, N.; Facchinetti, T.; Piastra, M. Online fall detection using recurrent neural networks. arXiv 2018 , arXiv:1804.04976. 35. Sokolova, M.; Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 2009 , 45, 427–437. [CrossRef] 74 Sensors 2021,21, 1945 Step 1: Two equidistant points yn,1 i and yn,2 i are calculated in the normal direction at Xisn−th position point xn iaccording to Equations (1) and (2), respectively. yn,1 i·x=xn i−dn i×sinθn yn,1 i·y=xn i+dn i×cosθn (1) yn,2 i·x=xn i+dn i×sinθn yn,2 i·y=xn i−dn i×cosθn (2) Here, θn is the angle between the tangent line at point xn i and the horizontal positive half-axis, that is, θn=arctan(kn) . kn is the tangent slope at the n-th point xn i of particle Xi . The isometric points of all position points are connected head to tail to form the corresponding polygon Siof particle Xi. Step 2: The valuating indicator fxiof particle Xiis calculated as fXi=0.5 ×ASi Art +0.5 ×Nc Si Nrt (3) The fitting value fXi is expressed as the sum of two parts: one corresponds to the ratio of the text aggregation degree in Si and rt , and the other corresponds to the ratio of the number of characters in Si and rt . Each part has a weight of 0.5. Here, ASi and Art are the text aggregation degree of Si and rt , respectively. NC Si and Nrt is the number of characters in Si and rt . Here the variable Nrt takes the value M . Given the text confidence map Im , which is obtained in Section 3.1, the text aggregation degree ASi is defined as the ratio of the number of white pixels in the polygon Si within Im to the area of Si , and Art is the ratio of the number of white pixels in the text proposal rtto the area of rt. The reason for considering the parameter ratio rather than the parameter itself of each part is that doing so can ensure the completeness of the text instance while ensuring the compactness. Step 3: After k−th single loop ends, the individual particle optima Pbesti and the particle swarm global optimum Gbestk are found from the history, and each particle’s velocity, position and distance are updated according to Equations (4)–(6), respectively, in which c1 , c2 represent the acceleration constants, μ1 , μ2 are uniformly distributed random numbers and Δδis white Gaussian noise. Xk i.v=Xk−1 i.v+c1×μ1×(Pbesti−Xk i.p)+c2×μ2×(Gbestk−Xk i.p)(4) Xk i.p=Xk−1 i.p+Xk i.v(5) Xk i.d=Xk−1 i.d+Δδ(6) When the optimization is completed, the optimal polygon will be obtained, which is composed of 14 equidistant points connected in series after the PSO-TSA optimal approximation. 3.2.2. ICMN Even if the PSO-TSA algorithm is used to revise the EAST detection result from quadrilateral to polygon, the approximate polygon may not be a complete text instance but only a part of the text instance. To avoid incomplete scene text detection when processing long words or text lines, it is necessary to merge adjacent or intersecting text components that belong to the same text instance. Therefore, ICMN is proposed to solve the instance incomplete detection problem. The intuition behind the ICMN is that humans can easily judge whether two separate text components belong to a complete text instance and naturally perceive its shape without examining the entire instance; similarly, a specially designed neural network might also be able to infer the boundary points of a text instance. The specific architecture of the ICMN is shown in Figure 3, which consists of an input layer, a pooling layer, two fully connected layers (FC) and an output layer. The input layer 81 Sensors 2021,21, 1945 first accepts the boundary points of two adjacent polygons as individual features, both of which have a size of 1 × 28. In the pooling layer, these two features from the input are simply concatenated together to build a 1 × 56-sized snippet feature. There are two fully connected layers in the ICMN, the sizes of which are 56 × 48 and 48 × 30. Using fully connected layers in the ICMN reduces the dimensions of the pooled feature to the desired size of 1 × 30. The final output consists of two sibling parts: the first one (with 1 × 2 −sized ) outputs two confidence scores indicating whether the two input components belong to the same text instance or not, and the second one (with 1 × 28 −sized ) outputs 14 regression boundary points of the instance, half of which are upper boundary points and half of which are lower boundary points. Because each point has x and y coordinates, the ICMN outputs a total of 28 corresponding point coordinates. The salient aspects of the ICMN are that it can predict whether two components merge and can refine the text instance boundary points by spatial coordinate regression. Figure 3. ICMN architecture. For training the ICMN, we assign a binary class label (whether it belongs to an instance) to each text snippet. A positive label is assigned to a snippet if (i) two subsections in it belong to an instance or (ii) the Intersection over Union (IoU) is greater than or equal to 0.3. We design a multitask loss L to jointly train classification and coordinate regression. L=Lcls +λLreg (7) Here, Lcls is the loss for merging/non-merging classification, which is a standard softmax loss; Lreg is the loss for spatial coordinate regression; and λ is a hyperparameter used to control over fitting. The regression loss is Lreg =1 Npos Nb ∑ i=1l∗ i×1 14 14 ∑ j=1  Pi j.x−Pi∗ j.x+Pi j.y−Pi∗ j.y   2 2(8) where L2 distance is adopted; l∗ i is the label, which is 1 for positive samples and 0 for negative samples; Npos is the number of positive samples in a mini-batch; and Nb is the number of all samples in a mini-batch. Here, Pi j . x and Pi j . y denote the x and y coordinates of the predicted j−th point Pi j of the i−th sample, respectively, and Pi∗ j . x and Pi∗ j . y denote the x and y coordinates of the j−th ground truth point of the i−th sample, respectively. Before training, we obtain the real ground truth by the method proposed in [ 8 ]. The regression loss is calculated only for positive samples. To assist LOMT in dealing with long curved text, a small and light ICMN is used in conjunction with PSO-TSA. Two adjacent or intersecting polygons may be subparts of the same text instance, so they are input to the ICMN to decide whether to merge them into a more complete text instance. The input of the ICMN is 28 points formed by concatenating each pair of 14 points of adjacent polygons, and the outputs of the network are the merging flag of the two parts and the regressed 82 Sensors 2021,21, 1945 14 points of the more complete text polygon. After a series of merging steps, each complete text instance appearing in the scene can be detected. 4. Experiment and Discussion 4.1. Datasets and Implementation Details 4.1.1. Datasets ICDAR2015 (IC15) [ 36 ]: IC15 was built for Challenge 4 of the ICDAR-2015 Robust Reading Competition and consists of 1000 training images and 500 test images. The images are acquired using Google Glass, and the text accidentally appears in the scene. The ground truth is annotated by a word-level quadrangle. ICDAR2017 (IC17) [ 41 ]: IC17 contains 7200 training images, 1800 validation images, and 9000 test images with text in 9 languages for multilingual scene text detection. Similar to IC15, the text regions in IC17 are also annotated by the 4 vertices of quadrilaterals. MSRA-TD500 (TD500) [ 37 ]: TD500 contains 500 natural images, which are split into 300 training images and 200 test images, collected both indoors and outdoors using a pocket camera. The images contain English and Chinese scripts. This dataset is mainly used for multidirectional text detection, in which text regions are annotated by rotated rectangles. SCUT-CTW1500 (CTW1500) [ 8 ]: CTW1500 consists of 1000 training and 500 test images. Every image has curved text instances, which are annotated by polygons with 14 vertices. The annotation is given at the text-line level such that a complete sentence is annotated as a single polygon. Total-Text (Total-Text) [ 10 ]: Total-Text contains 1555 scene images, which are divided into 1255 training images and 300 test images. This dataset mainly provides curved texts, which are annotated at the word level by polygon-shaped bounding boxes. 4.1.2. Implementation Details In this section, we carry out experiments on curved and non-curved datasets to test our method. In actual experiments, while maintaining the aspect ratio, the longer sides of the images in IC15, IC17, and TD500 are resized to 768, 1024, and 768, respectively, the longer sides of the images within Total-Text and CTW1500 are resized to 1280 and 1024, respectively. All experiments are performed with a single image resolution. The results are evaluated using the standard PASCAL VOC protocol [ 42 ], which uses a 0.5 intersection-over union (IoU) threshold to determine true and false positives. For training the text proposal detector, a union of the IC15 and IC17 dataset is used. The network is trained using an improved adaptive moments (ADAM) [ 43 ] optimizer. To speed up learning, we uniformly sample 512 × 512 crops from images to form a mini-batch of size 24. The learning rate of ADAM starts from 1 × 10 −3 , decays to one-tenth every 25,000 mini-batches, and stops at 1 × 10 −5 . The network is trained until the performance stops improving. When executing the PSO-TSA model, the parameters are set as follows. The particle swarm population consists of I particles moving in the N -dimensional search space, in which I and N are set as 20 and 7, respectively. The radius of the sampling neighborhood, r 1, is a certain proportion of the maximum of the average width and height of characters, which is set to 0.25 in the experiment. The distance range of the particle’s position point and its two equidistant points is [d1,d2] , in which d1= 1.5 ×r 1 and d2= 2.5 ×r 2. The maximum number of iteration K and the minimum error εmin are set 20 and 1 × 10 −3 respectively. The particle’s acceleration constants C1 and C2 are the same, both of which are set to 1.2, and both u1and u2are random numbers in (0, 1). In the objective function of ICMN, the hyper-parameter λ was set as 1.5. During training of the ICMN network model, the negative-to-positive sample ratio was set to 10 in a minibatch, the learning rate was set to 0.005, the batch size was 128, and the number of iterations was set to 30,000. In text detection process, when the IoU of two polygons is greater than or equal to 0.3, the two polygons are input into ICMN network for further merging. 83 Sensors 2021,21, 1945 4.2. Comparison with State-of-the-Arts 4.2.1. Experiments on Multi-Oriented Text (IC15, IC17 and TD500) We first conduct experiments on three multi-oriented text datasets, IC15, IC17, and TD500, to demonstrate that the proposed method performs well for multi-oriented scene text detection. For IC15, annotations are provided at the word level, so the test experiments are performed at the word level. For the IC17 and TD500 datasets, annotations are provided at the line level, without a post-processing step applied to generate word boxes, similar to the method in [ 34 ]. We evaluate the performance of the proposed method directly at the line level. To make a fair comparison with end-to-end methods [ 11 , 18 ], we include their detection-only results by referring to the original papers. We also compare the proposed method with other state-of-the-art methods, including five recent multi-oriented text detectors, EAST [ 9 ], TextBoxes++ [ 5 ], FTPN [ 7 ] and the methods in [ 25 , 32 ] and five arbitrarily shaped text detectors: TextSnake [ 33 ], PixelLink [ 23 ], PSENET-1s [ 17 ], CRAFT [ 34 ] and ICG (SegLink++) [ 13 ]. The comparison is given in Table 1. As depicted in Table 1, LOMT achieves 85.7%, 74.6%, and 82.7% in terms of the harmonic mean (H-mean) on IC15, IC17, and TD500, respectively, outperforming most competitors. Table 1. Results on quadrilateral-type datasets, such as IC15, IC17 and TD500. P, R and H refer to the precision, recall rate and H-mean, respectively. The best H score is highlighted in bold, and * denotes that the results based on multiscale tests. Method Backbone Year IC15 IC17 MSRA-TD500 P (%) R (%) H (%) P (%) R (%) H (%) P (%) R (%) H (%) Zhang et al. [25] VGG-16 + FCN 2016 71 43 54 - - - 83 67 74 EAST * [9] VGG-16 2017 80.5 72.8 76.4 - - - 81.7 61.6 70.2 TextSnake [33] VGG-16 + UNet 2018 84.9 80.4 82.6 - - - 83.2 73.9 78.3 TextBoxes++ * [5] VGG-16 2018 87.8 78.5 82.9 - - - - - - PixelLink [23] VGG-16 2018 85.5 82 83.7 - - - 83 73.2 77.8 Lyu et al. [32] VGG-16 + FCN 2018 89.5 79.7 84.3 74.3 70.6 72.4 87.6 76.2 81.5 E2E-MLT [11] ResNet-34 + FPN 2018 - - - 64.6 53.8 58.7 - - - FOTS [18] ResNet-50 + FPN 2018 88.8 82.0 85.3 79.5 57.5 66.7 - - - PSENET-1s [17] ResNet + FPN 2019 86.9 84.5 85.7 75.3 69.2 72.2 - - - CRAFT [34] VGG-16 + UNet 2019 89.8 84.3 86.9 80.6 68.2 73.9 88.2 78.2 82.9 FTPN [7] ResNet + FPN 2019 68.2 78.0 72.8 - - - - - - ICG (SegLink++) [13] VGG-16 2019 83.7 80.3 82.0 - - - - - - LOMT ResNet-50 + FPN 2020 86.9 84.6 85.7 79.1 70.6 74.6 88.9 77.3 82.7 IC15 is a dataset of complicated backgrounds, and the text size is small. As shown in the fourth to sixth columns of Table 1, the H-mean performance of LOMT is better than all other competitors, except that it is slightly lower than the CRAFT method. In addition, we demonstrate some test examples in Figure 4a, and LOMT can accurately locate text instances in different directions and different sizes. IC17 is a large-scale multilingual text dataset. We next verify the ability of our method to detect multilingual text for IC17. As shown in columns 7 to 9 of Table 1, our method outperforms all other competing methods in terms of H-mean. Some qualitative detection results on the IC17 dataset are given in Figure 4b, and the results show that our LOMT method can accurately detect multilingual text such as English, Chinese and Korean in different directions and sizes. On MSRA-TD500, we evaluate the performance of LOMT for detecting long and multilingual text lines. As shown in the results of columns 10 to 12 of Table 1, our method has certain advantages in the detection of long texts, and the H-mean is only 0.2% lower than CRAFT [ 34 ]. In addition, we demonstrate some test examples in Figure 4c, and our method can accurately locate long text instances with various orientations. 84 Sensors 2021,21, 1945 Figure 4. Some qualitative results of the proposed method on IC15 (a), IC17 (b), and TD500 (c). 4.2.2. Experiments on Two Curve Text Datasets (Total-Text and CTW1500) To test the ability of LOMT for the detection of curved text, we evaluate the detection results on both the Total-Text and CTW1500 datasets, which mainly contain curved text. We first conduct text detection experiments on the Total-Text dataset. Total-Text was constructed in late 2017, and curved text was collected from various scenes, including text-like and low-contrast background scenes. Most images in this dataset contain at least one curved text, which is annotated in word-level granularity using a strictly 14-point polygon. To compare LOMT with other algorithms under the same conditions, the number of uniformly sampled points on the character centerline is set to 7 so that the polygon with the same 14 vertices as the ground truth. Columns 3 to 5 of Table 2show the results of different methods on Total-Text. As shown in Table 2, the proposed method achieves 88.2%, 79.6%, and 83.7% in precision, recall and H-mean on Total-Text, respectively. The results show that our method outperformed recent state-of-the-art methods on Total-Text. Especially, compared with the text detector EAST, the H-mean is improved by 41.7%, and our method is slightly better than the excellent CRAFT algorithm. Examples of the detection results of the proposed method are illustrated in Figure 5. From the results, we can find that the proposed method can work well for text of any shape and length. In addition, we show some test examples of different methods on Total-Text in Figure 6, where the results in columns (a), (b) and (c) correspond to CTD + TLOC [ 8 ], CRAFT [ 34 ] and the proposed LOMT method, respectively. CTD + TLOC [ 8 ] detects a false text in the first row 85 Sensors 2021,21, 1945 in Figure 6a, and all detection results are inaccurate. The detection results of CRAFT [ 34 ] shown in Figure 6b are relatively ideal, but the parts of a single character, such as the letter “C”, are not completely contained in the detected text box. Compared to other methods, the proposed method has better completeness of the text instances and smoother boundaries of the polygons. Table 2. Results on curved text datasets Total-Text and CTW1500. The best H score is highlighted in bold and * denotes the results based on multiscale tests. Method Year Total-Text CTW1500 P (%) R (%) H (%) P (%) R (%) H (%) EAST * [9] 2017 50.0 36.2 42.0 78.7 49.1 60.4 DeconvNet * [10] 2017 33.0 40.0 36.0 - - - TextSnake [33] 2018 82.7 74.5 78.4 67.9 85.3 75.6 PSENet-1s [17] 2019 81.8 75.1 78.3 80.6 75.6 78.0 LOMO-1s [12] 2019 88.6 75.7 81.6 - - - CTD + TLOC [8] 2019 77.4 69.8 73.4 77.4 69.8 73.4 ICG (SegLink++) [13] 2019 82.1 80.9 81.5 82.8 79.8 81.3 CRAFT [34] 2019 87.6 79.9 83.6 86.0 81.1 83.5 Wang et al. [19] 2019 80.9 76.2 78.5 80.1 80.2 80.1 Mask-TTD [29] 2020 74.5 79.1 76.7 79.7 79.0 79.4 LOMT 2020 88.2 79.6 83.7 85.1 79.9 82.5 Figure 5. Some qualitative results of the proposed method on Total-Text. Figure 6. Comparison results of several methods, (a) CTD + TOLC [8], (b) CRAFT [34], and (c) LOMT. 86 Sensors 2021,21, 1945 We continue to evaluate the results of LOMT on CTW1500, whose annotation is given at the text-line level such that a complete text instance is annotated as a single 14-point polygon. A comparison with the quantitative evaluations of other methods is depicted in Table 3, indicating that H-mean performance of the proposed LOMT method is 22.1% higher than the EAST [ 9 ] algorithm. Compared with CTD + TLOC [ 8 ], TextSnake [ 33 ], PSENet-1s [ 17 ] and Mask-TTD [ 29 ], H-mean improvement is 9.1%, 6.9%, 4.5% and 3.1%, respectively. The proposed LOMT is slightly better in H-mean performance than ICG (SegLink++) [ 13 ] and the method in [ 19 ] and is only 1.0% lower than CRAFT [ 34 ]. The results demonstrate that the proposed method can handle arbitrarily shaped line-level text detection. Some qualitative results are shown in Figure 7. The results show that LOMT can handle long text detection well. Table 3. The ablation experiment results. Dataset EAST MSER Extraction PSO-TSA ICMN H (%) CTW1500 √X X X 60.4 √√ √ X 71.2 √√ √√ 81.5 Total-Text √X X X 42.0 √√ √ X 81.7 √√ √√ 83.7 Figure 7. Some qualitative results of the proposed method on CTW1500. 4.2.3. Speed and Ablation Analysis All the experiments are conducted on a PC using a single NIVIDA GeForce GTX 1080 graphics card with the Pascal architecture and an Intel(R) Core(TM) i7-6700K @ 4.00 GHZ CPU. The speed of PSO-TSA and ICMN are 20.0 FPS and 30.9 FPS, respectively, which can basically achieve real-time running. Although the H-mean performance of the CRAFT algorithm is currently the most competitive compared with other state of the arts, its average frame rate is only 3.3. The proposed method consists of four modules: text proposal detection, candidate character extraction using MSER, text shape approximations by PSO-TSA, and text subsection merging via ICMN. The first two modules are the prerequisites for PSOTSA. Without the extracted characters, the PSO-TSA module cannot be executed independently. That is, the contribution of the character extraction module is included in the performance improvement of PSO-TSA. Therefore, we need to verify only the effectiveness of the three modules, namely, text proposal detection, PSO-TSA and ICMN. Table 3shows the ablation experiment results on two curved text benchmarks. From Table 3, for the CTW1500 and Total-Text datasets, compared to the text proposal detection module, the improvements in the H-mean performance of the PSO-TSA module are 10.8% and 39.7%, respectively. Furthermore, compared with the PSO-TSA module without ICMN, the gain of the H-mean 87 Sensors 2021,21, 1945 performance of LOMT is 10.3% and 2.0%. In particular, it should be emphasized that compared to the general regression-based detection module, EAST, the overall performance of our algorithm can be improved by at least 21.1% and up to 41.7%. The experimental results demonstrate that each module in Table 3has a certain contribution to the overall performance, thus verifying the necessity of each part. 4.2.4. Disadvantages and Advantages The main disadvantage of the proposed method is not an end-to-end framework. The traditional MSER method is used in our character extraction module, and the accuracy rate is 68.3% in CTW1500 data set, while the accuracy rate of CRAFT algorithm is 75.7%. Due to unsatisfactory text detection performance of MSER, the accuracy of the proposed method is not good enough compared to the existing methods, especially CRAFT. Although the performance of our method is slightly inferior to the CRAFT method, one of the biggest advantages of our method is that it does not depend too much on a specific data set. In order to test the generalization of the algorithm, Table 4lists the comparison results of the generalization experiment of the two methods, in which CRAFT-IC15-20k.pth and CRAFT-MLT-25k.pth denote the CRAFT models trained on IC15 and IC13 + IC17 data sets, respectively. It can be seen from Table 4that the model CRAFT-IC15-20k.pth performs well on the IC15 dataset, but its performances on the other four datasets have dropped a lot, and the performances of the model CRAFT-MLT-25k.pth on the IC15 and IC17-MLT data sets are also inferior to our method. In other words, the CRAFT model is strongly based on training samples, and its generalization ability is not particularly ideal. Fortunately, neither our character extraction module nor our PSO-TSA module depends on a specific dataset; consequently, they are highly applicable to any dataset. ICMN module can be readily plugged into to any regression-based text positioning module to complete the text instance detection. Table 4. Comparison of generalization experimental results. Data Sets CRAFT-IC15-20k.pth CRAFT-MLT-25k.pth LOMT P (%) R (%) H (%) P (%) R (%) H (%) P (%) R (%) H (%) IC15 89.8 84.3 86.9 81.7 82.5 82.0 86.9 84.6 85.7 IC17-MLT 51.2 47.7 49.4 80.6 68.2 73.9 79.1 70.6 74.6 TD500 16.1 29.6 20.9 88.0 78.1 82.9 88.9 77.3 82.7 CTW1500 69.8 70.6 70.2 86.0 81.1 83.5 85.1 79.9 82.5 Total-Text 72.5 76.3 74.3 87.6 79.9 83.6 88.2 79.6 83.7 5. Conclusions In this paper, we have proposed a straightforward and efficient instance-aware curved text detector, which is composed of quadrilateral text proposal detection, character extraction, particle swarm optimization-based text shape approximation (PSO-TSA), and an instance-aware component merging network (ICMN). Without training or learning in advance, PSO-TSA can adjust the quadrilateral box obtained by a regression-based text detector to a polygon, thus approaches the shape of the text. ICMN can merge two adjacent or intersecting text components into one single text instance that is more complete. A large number of experiments have been carried out on five scene text detection benchmark datasets, H-mean performance achieve to 85.7%, 74.6%, 82.7%, 82.5% and 83.7% for IC15, IC17, MSRA-TD500, CTW1500 and Total-Text dataset, respectively. As the two core components of the text detector, the average running speeds of PSO-TSA and ICMN modules are 20.0 FPS and 30.9 FPS, respectively. Experimental results demonstrate that the proposed algorithm has competitive advantages in H-mean performance and execution speed. 88 Sensors 2021,21, 1945 In the future, we would like to replace the MSER extraction with more elegant character detection method to further improve the efficiency of the algorithm. Author Contributions: Conceptualization, F.Z.; methodology, F.Z., S.S. and L.Z.; software, S.S. and L.Z.; validation, Z.W.; formal analysis, S.S. and L.Z.; writing—original draft preparation, F.Z.; writing—review and editing, Z.W. All authors have read and agreed to the published version of the manuscript. Funding: This work was funded by the National Natural Science Foundation of China (NSFC), grant number 61671376 and 61771386. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: The datasets IC15 and IC17 in this paper can be obtained from the following link: https://rrc.cvc.uab.es/?ch=4&com=downloads, accessed on 26 August 2015. The dataset TD500 in this paper can be obtained at http://www.iapr-tc11.org/mediawiki/index.php/ MSRA_Text_Detection_500_Database_%28MSRA-TD500%29, accessed on 20 June 2012. The dataset CTW1500 in this paper can be obtained at https://www.github.com/Yuliang-Liu/Curve-Text- Detector, accessed on 6 December 2017. The dataset TotalText in this paper can be obtained at https://github.com/cs-chan/Total-Text-Dataset, accessed on 15 November 2017. Conflicts of Interest: The authors declare no conflict of interest. References 1. Epshtein, B.; Ofek, E.; Wexler, Y. Detecting Text in Natural Scenes with Stroke Width Transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010; pp. 2963–2970. 2. Neumann, L.; Matas, J. A method for text localization and recognition in real-world images. In Proceedings of the Asian Conference on Computer Vision, Queenstown, New Zealand, 8–12 November 2010; pp. 770–783. 3. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada, 7–10 December 2015; pp. 91–99. 4. Ma, J.; Shao, W.; Ye, H.; Wang, L.; Wang, H.; Zheng, Y.; Xue, X. Arbitrary-oriented scene text detection via rotation proposals. IEEE Trans. Multimed. 2018,20, 3111–3122. [CrossRef] 5. Liao, M.; Shi, B.; Bai, X. Textboxes++: A single-shot oriented scene text detector. IEEE Trans. Image Process. 2018 ,27, 303–338. [CrossRef] 6. Wang, F.; Zhao, L.; Li, X.; Wang, X. Geometry-aware scene text detection with instance transformation network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 1381–1389. 7. Liu, F.; Chen, C.; Gu, D.; Zheng, J. Ftpn: Scene text detection with feature pyramid based text proposal network. IEEE Access 2019 , 7, 44219–44228. [CrossRef] 8. Liu, Y.; Jin, L.; Zhang, S.; Luo, C.; Zhang, S. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognit. 2019,90, 337–345. [CrossRef] 9. Zhou, X.; Yao, C.; Wen, H.; Wang, Y.; Zhou, S.; He, W.; Liang, J. East: An efficient and accurate scene text detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2642–2651. 10. Ch’ng, C.K.; Chan, C.S. Total-text: A comprehensive dataset for scene text detection and recognition. In Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan, 9–15 November 2017; Volume 1, pp. 935–942. 11. Bušta, M.; Patel, Y.; Matas, J. E2e-mlt-an unconstrained end-to-end method for multi-language scene text. In Proceedings of the Asian Conference on Computer Vision, Perth, WA, Australia, 2–6 December 2018; Springer: Cham, Switzerland; pp. 127–143. 12. Zhang, C.; Liang, B.; Huang, Z.; En, M.; Han, J.; Ding, E.; Ding, X. Look more than once: An accurate detector for text of arbitrary shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 10552–10561. 13. Tang, J.; Yang, Z.; Wang, Y.; Zheng, Q.; Xu, Y.; Bai, X. Seglink++: Detecting dense and arbitrary-shaped scene text by instance-aware component grouping. Pattern Recognit. 2019,96, 106954. [CrossRef] 14. NguyenVan, D.; Lu, S.; Tian, S.; Ouarti, N.; Mokhtari, M. A pooling based scene text proposal technique for scene text reading in the wild. Pattern Recognit. 2019,87, 118–129. [CrossRef] 15. He, W.; Zhang, X.-Y.; Yin, F.; Luo, Z.; Ogier, J.-M.; Liu, C.-L. Realtime multi-scale scene text detection with scale-based region proposal network. Pattern Recognit. 2020,98, 107026. [CrossRef] 16. Zhu, Y.; Ma, C.; Du, J. Rotated cascade R-CNN: A shape robust detector with coordinate regression. Pattern Recognit. 2019 ,96, 106964. [CrossRef] 89 Sensors 2021,21, 1945 17. Wang, W.; Xie, E.; Li, X.; Hou, W.; Lu, T.; Yu, G.; Shao, S. Shape robust text detection with progressive scale expansion network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 9336–9345. 18. Liu, X.; Liang, D.; Yan, S.; Chen, D.; Qiao, Y.; Yan, J. Fots: Fast oriented text spotting with a unified network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 5676–5685. 19. Wang, X.; Jiang, Y.; Luo, Z.; Liu, C.L.; Choi, H.; Kim, S. Arbitrary shape scene text detection with adaptive text region representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 6449–6458. 20. He, W.; Zhang, X.-Y.; Yin, F.; Liu, C.-L. Deep direct regression for multi-oriented scene text detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 745–753. 21. Wu, Y.; Natarajan, P. Self-organized text detection with minimal post-processing via border learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 5000–5009. 22. He, D.; Yang, X.; Liang, C.; Zhou, Z.; Ororbi, A.G.; Kifer, D.; Giles, C.L. Multi-scale fcn with cascaded instance aware segmentation for arbitrary oriented word spotting in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 474–483. 23. Deng, D.; Liu, H.; Li, X.; Cai, D. Pixellink: Detecting scene text via instance segmentation. In Proceedings of the AAAI-18 AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2017; pp. 6773–6780. 24. Xue, C.; Lu, S.; Zhan, F. Accurate scene text detection through border semantics awareness and bootstrapping. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 35–372. 25. Zhang, Z.; Zhang, C.; Shen, W.; Yao, C.; Liu, W.; Bai, X. Multi-oriented text detection with fully convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 4159–4167. 26. Bazazian, D.; G’omez, R.; Nicolaou, A.; Gomez, L.; Karatzas, D.; Bagdanov, A.D. Fast: Facilitated and accurate scene text proposals through fcn guided pruning. Pattern Recognit. Lett. 2019,119, 112–120. [CrossRef] 27. Xu, Y.; Wang, Y.; Zhou, W.; Wang, Y.; Yang, Z.; Bai, X. Textfield: Learning a deep direction field for irregular scene text detection. IEEE Trans. Image Process. 2019,28, 5566–5579. [CrossRef][PubMed] 28. Li, J.; Zhang, C.; Sun, Y.; Han, J.; Ding, E. Detecting text in the wild with deep character embedding network. In Proceedings of the Asian Conference on Computer Vision, Perth, WA, Australia, 2–6 December 2018; Springer: Cham, Switzerland, 2018. 29. Liu, Y.; Jin, L.; Fang, C. Arbitrarily Shaped Scene Text Detection with a Mask Tightness Text Detector. IEEE Trans. Image Process. 2020,29, 2918–2930. [CrossRef][PubMed] 30. Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. 31. Xie, E.; Zang, Y.; Shao, S.; Yu, G.; Yao, C.; Li, G. Scene text detection with supervised pyramid context network. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; pp. 9038–9045. 32. Lyu, P.; Yao, C.; Wu, W.; Yan, S.; Bai, X. Multi-oriented scene text detection via corner localization and region segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 7553–7563. 33. Long, S.; Ruan, J.; Zhang, W.; He, X.; Wu, W.; Yao, C. Textsnake: A flexible representation for detecting text of arbitrary shapes. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 20–36. 34. Baek, Y.; Lee, B.; Han, D.; Yun, S.; Lee, H. Character region awareness for text detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 9365–9374. 35. Karatzas, D.; Shafait, F.; Uchida, S.; Iwamura, M.; Bigorda, L.G.i.; Mestre, S.R.; Mas, J.; Mota, D.F.; Almazan, J.A.; de las Heras, L.P . ICDAR 2013 robust reading competition. Document Analysis and Recognition (ICDAR). In Proceedings of the 2013 12th International Conference on IEEE Computer Society, Niigata, Japan, 16–20 June 2013; pp. 1484–1493. 36. Karatzas, D.; Gomez-Bigorda, L.; Nicolaou, A.; Ghosh, S.; Bagdanov, A.; Iwamura, M.; J Matas, L.N.; Chandrasekhar, V.R.; Lu, S. ICDAR 2015 competition on robust reading. In Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), Tunis, Tunisia, 23–26 August 2015; pp. 1156–1160. 37. Yao, C.; Bai, X.; Liu, W.; Ma, Y.; Tu, Z. Detecting texts of arbitrary orientations in natural images. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 18–20 June 2012; pp. 1083–1090. 38. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. 39. Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2117–2125. 40. Kennedy, J.; Eberhar, R. Particle swarm optimization. In Proceedings of the ICNN’95-International Conference on Neural Networks, Perth, WA, Australia, 27 November–1 December 1995; pp. 1942–1948. 90 Sensors 2021,21, 3144 so that non-linear dependencies between different training data can be considered. The mapping in a higher dimensionality enabled the detection of similarities between data characteristics. The Gaussian radial basis function was used as the kernel, and accordingly, the following Equation (2) was obtained: Kxi,xj=exp−γxi−xj2,γ=1 2σ2(2) σ is the variance. γ is a hyperparameter that smoothed the kernel function, and this means a stronger or weaker relationship between the samples can be found depending on the γ values. To estimate the optimal hyperparameters γ and C for 1C-SVM, Nelder-Mead’s heuristic method was used [ 18 , 19 ]. The selected criterion to find the optimal classifier was the area under the Receiver Operating Characteristic curve (AUC-ROC), which is increasingly used in machine learning and evaluating more significant amounts of data. The inputs for the ensemble algorithm were: 1. Training data Sk={(xi,yi)|i=1,.........,Nk. The data set consisted of Nk training data points with xi∈Rd , where drepresents the number of dimensions and yi∈ {−1,1} , the associated class. The Nk data points were randomly selected from the kth database (DBk). 2. A primary classifier to generate hypothesis h. The classifier required that at least 50 percent of the training data record was classified correctly. 3. An integer Tkthat specified the number of iteration steps t=1,2, . . . . . .., Tkfor each data set, with t∈N. The prediction error could be reduced sufficiently with Tk. The ensemble algorithm started with the initialization of a series of weights αt for the training data set Sk and a distribution Dt obtained from αt [ 1 ]. According to Dt , Sk was divided into two subsets, TRt for training and TEt for validating during the tth iteration of the algorithm. Dt was initially defined uniformly without deductive information. At each iteration, the weights adjusted at iteration t− 1 were divided by the sum of all Wt− 1to ensure a legitimate distribution, and a new Dt was computed. Training and test subsets were drawn randomly according to Dt . A hypothesis ht was obtained as the tth classifier, whose error εt(3) was computed on the entire data set Skwith: εt=∑i:ht(xi)Dt·|yi−hk(xi)| ∑i:ht(xi)Dt(3) εt was required to be less than 0.5 to ensure a reasonable performance of ht . If the condition was satisfied, ht was accepted, and the error was normalized to calculate the normalization error βt(4): βt=εt (1−εt),0<βt<1 (4) The current ht was discarded if the condition was not satisfied, and a new training subset was selected. All hypotheses generated so far were then combined using the weighted majority voting to obtain a composite one Ht (5), which allowed efficient incremental learning capability when new classes were introduced. The hypothesis with good performance was awarded a higher voting weight [20]. Ht=argmaxy∈Y∑ t:ht(x)=y log1 βt(5) The error of Ht was computed with (6) and must have also been less than 0.5 to ensure a reasonable performance of Ht ; otherwise, the algorithm discarded that one and returned to select a new TRt. Et=∑i:Ht(xi)Dt·|yi−Ht(xi)| ∑i:Ht(xi)Dt(6) 97 Sensors 2021,21, 3144 Then Btwas computed with (7). Bt=Et (1−Et),0<Bt<1 (7) The rule of Equation (8) was used to reduce the weights of those data points that were correctly classified by the composite hypothesis Ht . Furthermore, this lowered the probability of being selected in the following training subset. αt+1(i)=αt(i)·Bt,if H t(xi)=yi 1, otherwise (8) The hypothesis HF for the training subset and the subset of features could be obtained by combining all hypotheses that had been generated so far using the weighted majority voting rule (see Figure 2). HF(x)=argmaxy∈Y K ∑ k=1 ∑ t:Ht(x)=y log1 Bt(9) Figure 2. Schematic representation of the incremental learning ensemble classifier (altered from [20]). 2.4. Evaluation The water spectrum was subtracted from each sample’s spectrum. Then, 454 main features from intervals of the spectrum were used to evaluate the data. The intervals contained the spectrum’s descriptive peaks, which were between the Raman shifts 850–910, 1010–1130, 1410–1510, and 2840–3010 cm −1 . These characteristics corresponded to the value of the derivative and the integral in each interval. The following methodology was implemented to obtain an objective evaluation. The data set was divided into three subsets: training, validation, and test. The validation subset aimed to make a fair estimation of performance, independency of the test data, and to pick 98 Sensors 2021,21, 3144 optimal parameters, which in turn provided a more generalized solution. We implemented a nested cross-validation (CV) method for unbiased estimation of prediction error in an independent test data. Leave-one-out (LOO) was used to evaluate the classifier, i.e., nine experiments were used for training and validation and one experiment for independent testing. To estimate the classifier’s unknown tuning parameters, tenfold CV was used with data points from the nine experiments selected for training and validation. The data points were randomly permuted and divided in ten parts, one part for the validation and nine for training. Statistical measures such as the sensitivity true positive results (TPR) and precision were also computed (see Equation (10)). A thresholding procedure was used to design a binary classification. The threshold Th was selected as the concentration level such that a particular concentration was higher than the threshold concentration if the probability pyi xi≥Th else it was classified as lower concentration than required, and a ground truth table based on this idea was generated for every concentration. The use of TPR and Positive Predictive Value (PPV) provided additional information about the classifier’s performance and could be used to compare results obtained with other methods. Additional parameters like the PPV and TPR were also calculated as follows: TPR =TP TP +FN (10) PPV =TP TP +FP (11) TP and TN denote the number of true positive and true negative data points, respectively. FP and FN denote the number of false-positive and false-negative data points, respectively. The SVM model in The Unscrambler X version 10.4 (Camo Software, Oslo, Norway) was used to compare the algorithm with a standard program. The ground truth table generated using the MATLAB algorithm was used as input data for binary classification. For each concentration, cross-validation was performed in the same way the MATLAB algorithm does. This resulted in the creation of nine models and predictions per concentration. 3. Results Figure 3shows the resulting Raman spectra of ethanol and water in the complete recorded region. The descriptive peaks for ethanol in between 850–910, 1010–1130, 1410– 1510, and 2840–3010 cm −1 were used in the algorithm. Additionally, two Raman peaks at 435 and 1275 cm −1 were also visible in the ethanol spectrum. Water showed typical bands at 500 cm −1 (hydrogen bond), 1640 cm −1 (OH bending), and 3100–3600 cm −1 (OH stretching). Remaining peaks were derived from the glass beaker. In Figure 4, the spectral data of the dilution series is displayed. For reasons of clarity, the range in between 850–1130 cm −1 was enlarged to show the linear concentration dependency more clearly. The outputs from the predictions obtained through Unscrambler X must be categorised manually into true positive, true negative, false positive, and false negative to calculate the mean value for accuracy, precision, and recall (sensitivity) for each concentration. The results obtained through Unscrambler X are shown in Table 2. 99 Sensors 2021,21, 3144 Figure 3. Raman spectra of water and ethanol with evaluated intervals. Figure 4. Raman spectra of ethanol dilution series (sample number 1–9). 100 Sensors 2021,21, 3144 Table 2. Calculated mean value for accuracy, precision, recall/sensitivity, and computation time for each concentration with the c-value and gamma obtained from the algorithm using Unscrambler X. Calculated EtOH Concentration (Vol%) Accuracy Unscrambler (Unscrambler Parameters) (%) Precision (-) Recall/Sensitivity (-) 0.9620 100.0 1.00 1.00 0.4810 100.0 1.00 1.00 0.2405 100.0 1.00 1.00 0.1203 98.7 0.99 0.98 0.0601 95.8 0.95 0.97 0.0301 90.9 0.95 0.90 0.0150 83.4 0.90 0.87 0.0075 84.7 0.90 0.92 0.0038 90.7 0.93 0.98 Table 3displays the results obtained from the MATLAB algorithm. The different parameters for the performance of the classifier were computed using the algorithm. The training time, as well as the classification time, was calculated for each concentration. The time taken for validation and training was also calculated. Table 3. Calculated mean value for accuracy, precision, recall/sensitivity, and time required for training and validation for each concentration obtained from the MATLAB algorithm. Calculated EtOH Concentration (Vol%) Accuracy MATLAB Algorithm (MATLAB Algorithm Parameters) (%) Precision (-) Recall/ Sensitivity (-) Training and Validation Time (s) 0.9620 100.0 1.00 1.00 14.5748 0.4810 100.0 1.00 1.00 17.8897 0.2405 99.9 0.99 1.00 43.3492 0.1203 99.9 0.98 0.98 40.5498 0.0601 99.0 0.97 0.97 148.2467 0.0301 96.2 0.96 0.95 225.5823 0.0150 91.9 0.97 0.93 140.3069 0.0075 89.1 0.98 0.93 92.7786 0.0038 83.7 0.99 0.94 48.4829 As can be observed from the results obtained, when the concentration is higher, both the methods’ performances were more or less identical. However, as the concentration decreases, the MATLAB algorithm showed better accuracy, precision, and sensitivity. For concentrations from 0.962 to 0.12025, the performance with respect to precision and sensitivity of both methods was almost the same, but the MATLAB algorithm’s accuracy was still higher. As we go to lower concentrations from 0.060125 to 0.007515625, there was a significant difference between the accuracy, precision, and sensitivity from the MATLAB algorithm and Unscrambler X. This shows that the MATLAB algorithm gave better results in terms of accuracy, precision, and sensitivity. In Figure 5, the accuracy from The Unscrambler X and the accuracy from the MATLAB algorithm are plotted over concentration. For concentrations from sample number 1–3, the Unscrambler X models with the MATLAB parameters showed a significantly lower accuracy, while the MATLAB algorithm and the Unscrambler X model with the new parameters showed an accuracy of 100%. This implied that the Unscrambler X and MATLAB work with different parameters that cannot be taken from each other to create the same results. For calculated ethanol concentrations lower than 0.481 Vol% (sample number 3), the MATLAB algorithm generated higher accuracy and displayed slighter decrease in accuracy over the samples. At a calculated ethanol concentration of 0.0300625 Vol% (sample number 7), the 101 Sensors 2021,21, 3144 MATLAB algorithm had the highest difference in accuracy compared to the Unscrambler model with approximately 8%. Figure 5. Comparison of the accuracy from The Unscrambler X and the MATLAB code. 4. Discussion To solve the problem of unlearning in models used in PAT processes, pattern recognition systems that can be better adapted, scalable, and given the ability to learn different data distributions dynamically should be used, but are not available. The method for incremental learning algorithm that uses an ensemble algorithm for classification in a rectification process of ethanol is a possible solution for unlearning. It can learn additional information as soon as new data are available without unlearning the previously acquired knowledge and prevents the unlearning of old Raman spectroscopic data. The system is continuously updated with the current data and significantly increases robustness and accuracy without the loss of any data. The main advantage of the proposed method over the standard methods available, like Unscrambler X, was that the computation time was less, accuracy was higher, and manual intervention was not involved. The disadvantage in the SVM model created using Unscrambler X was that the data had to be prepared manually, and for each concentration, a model had to be created. The proposed method works without manual intervention saving the users a lot of time and effort. Additionally, a Grid Search had to be performed for each concentration in Unscrambler X. This resulted in a total of 81 models that had to be evaluated manually. Hence, the total time to calculate the accuracy for all concentrations depended on the operator’s efficiency and speed. The algorithm of the developed new model’s self-learning follow-up enabled the computing time (batch processing, parallel execution, and distributed data) to be reduced. Additionally, the accuracy and the detection sensitivity of the measuring devices that are used can be increased. In this paper, the spectrum range peaks were selected and used, but the algorithm’s application can be extended to any number of peaks and any range in the spectrum that suits the application. The algorithm can be adapted for different data and 102 Sensors 2021,21, 3144 applications where continuous monitoring is required with the addition of new data. The algorithm performs better than standard methods until the concentration is above 0.0075. Further investigations need to be performed using different ranges of spectroscopic data and with different chemical processes. The method presented further increases the robustness against outliers or poorly representative data, which do occur in the measured values obtained during operation determined by the system’s operating personnel. The fast availability of measurement data increases the system flexibility and thus the constant optimization to changing demands. To the best of our knowledge, the present work is the first to apply the incremental learning SVM model in Raman spectroscopic data used in various chemical processes where continuous monitoring is required. Author Contributions: Conceptualization: D.H. and S.C.; methodology: D.H. and S.C.; software: J.F.G.M., S.C., and E.S.; validation: D.H. and S.C.; formal analysis: D.H., S.C. and E.S.; investigation: D.H. and M.N.; writing—original draft preparation: D.H. and S.C.; writing—review and editing: D.H. and S.C.; visualization: D.H. and S.C.; supervision: M.R.; funding acquisition: M.R. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the Bundesministerium für Wirtschaft und Energie (BMWi) and the Arbeitsgemeinschaft industrieller Forschungsvereinigungen (AiF, ZF4013934RE7). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Restrictions apply to the availability of these data. Data was obtained from JoséFernando García Molina’s thesis and is available from the authors with the permission of García Molina. Acknowledgments: The authors like to thank Frank Braun for his technical support. Conflicts of Interest: The authors declare no conflict of interest. References 1. Breitkopf, A. Energieverbrauch in Deutschland Nach Wirtschaftszweig 2018: Energieverbrauch Des Verarbeitenden Gewerbes in Deutschland Nach Ausgewählten Sektoren im Jahr 2018. Available online: https://de.statista.com/statistik/daten/studie/4325 96/umfrage/energieverbrauch-im-verarbeitenden-gewerbe-in-deutschland-nach-sektor/ (accessed on 31 March 2020). 2. Hohmann, M. Entwicklung von Produktion, Energieverbrauch und Treibhausgasemissionen der Chemisch-Pharmazeutischen Industrie in Deutschland im Zeitraum 1990 bis 2018. Available online: https://de.statista.com/statistik/daten/studie/186496 /umfrage/entwicklung-der-chemischen-industrie-in-deutschland/ (accessed on 31 March 2020). 3. Schalk, R.; Braun, F.; Frank, R.; Rädle, M.; Gretz, N.; Methner, F.-J.; Beuermann, T. Non-contact Raman spectroscopy for in-line monitoring of glucose and ethanol during yeast fermentations. Bioprocess Biosyst. Eng. 2017 ,40, 1519–1527. [CrossRef][PubMed] 4. Braun, F.; Schalk, R.; Brunner, J.; Eckhardt, H.S.; Theuer, M.; Veith, U.; Hennig, S.; Ferstl, W.; Methner, F.-J.; Beuermann, T.; et al. Nicht-invasive Prozesssonde zur Inline-Ramananalyse durch optische Schaugläser. tm—Tech. Messen 2016,83.[CrossRef] 5. Ai, H.; Wu, X.; Zhang, L.; Qi, M.; Zhao, Y.; Zhao, Q.; Zhao, J.; Liu, H. QSAR modelling study of the bioconcentration factor and toxicity of organic compounds to aquatic organisms using machine learning and ensemble methods. Ecotoxicol. Environ. Saf. 2019 , 179, 71–78. [CrossRef][PubMed] 6. Zaidi, S. Novel application of support vector machines to model the two phase boiling heat transfer coefficient in a vertical tube thermosiphon reboiler. Chem. Eng. Res. Des. 2015,98, 44–58. [CrossRef] 7. Li, M.; Wei, D.; Liu, T.; Liu, Y.; Yan, L.; Wei, Q.; Du, B.; Xu, W. EDTA functionalized magnetic biochar for Pb(II) removal: Adsorption performance, mechanism and SVM model prediction. Sep. Purif. Technol. 2019,227, 115696. [CrossRef] 8. French, R. Catastrophic forgetting in connectionist networks. Trends Cognit. Sci. 1999,3, 128–135. [CrossRef] 9. Polikar, R.; Upda, L.; Upda, S.S.; Honavar, V. Learn++: An incremental learning algorithm for supervised neural networks. IEEE Trans. Syst. Man Cybern. C 2001,31, 497–508. [CrossRef] 10. Erdem, Z.; Polikar, R.; Gurgen, F.; Yumusak, N. Ensemble of SVMs for Incremental Learning. In Multiple Classifier Systems, Proceedings of the 6th International Workshop, MCS 2005, Seaside, CA, USA, 13–15 June 2005; Oza, N.C., Ed.; Springer: Berlin, Germany, 2005; pp. 246–256. ISBN 978-3-540-26306-7. 11. Kho, J.B.; Lee, W.; Choi, H.; Kim, J. An incremental learning method for spoof fingerprint detection. Expert Syst. Appl. 2019 ,116, 52–64. [CrossRef] 12. Li, Y.; Su, B.; Liu, G. An Incremental Learning Algorithm for SVM Based on Combined Reserved Set. J. Shanghai Jiaotong Univ. 2016.[CrossRef] 103 Sensors 2021,21, 3144 13. Liu, J.; Pan, T.-J.; Zhang, Z.-Y. Incremental Support Vector Machine Combined with Ultraviolet-Visible Spectroscopy for Rapid Discriminant Analysis of Red Wine. J. Spectrosc. 2018,2018, 4230681. [CrossRef] 14. Ardakani, M.; Escudero, G.; Graells, M.; Espuña, A. Incremental Learning Fault Detection Algorithm Based on Hyperplane- Distance. In 26th European Symposium on Computer Aided Process Engineering: Part A; Kravanja, Z., Ed.; Elsevier Science: Amsterdam, The Netherlands, 2016; pp. 1105–1110. ISBN 9780444634283. 15. García Molina, J.F.; Zheng, L.; Sertdemir, M.; Dinter, D.J.; Schönberg, S.; Rädle, M. Incremental learning with SVM for multimodal classification of prostatic adenocarcinoma. PLoS ONE 2014,9, e93600. [CrossRef][PubMed] 16. Xu, G.; Zhou, H.; Chen, J. CNC internal data based incremental cost-sensitive support vector machine method for tool breakage monitoring in end milling. Eng. Appl. Artif. Intell. 2018,74, 90–103. [CrossRef] 17. Hastie, T.; Tibshirani, R.; Friedman, J.H. The Elements of Statistical Learning: Data Mining, Inference, and Prediction; Springer: New York, NY, USA, 2009. 18. Cosme, R.D.C.; Krohling, R.A. Support Vector Machines Applied to Noisy Data Classification Using Differential Evolution with Local Search; Technical Report; Universidade Federal do Espirito Santo: Vitória, Espírito Santo, Brazil, 2011. 19. Nelder, J.A.; Mead, R. A Simplex Method for Function Minimization. Comput. J. 1965,7, 308–313. [CrossRef] 20. García Molina, J.F. Modeling and Analysis of Prostate Cancer Structures Alongside an Incremental and Supervised Learning Algorithm. Ph.D. Thesis, Ruprecht-Karls University of Heidelberg, Heidelberg, Germany, 2014. 104 sensors Article WS-RCNN: Learning to Score Proposals for Weakly Supervised Instance Segmentation Jia-Rong Ou, Shu-Le Deng and Jin-Gang Yu *   Citation: Ou, J.-R.; Deng, S.-L.; Yu, J.-G. WS-RCNN: Learning to Score Proposals for Weakly Supervised Instance Segmentation. Sensors 2021, 21, 3475. https://doi.org/ 10.3390/s21103475 Academic Editor: Loris Nanni Received: 26 March 2021 Accepted: 7 May 2021 Published: 17 May 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). School of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, China; [email protected] (J.-R.O.); [email protected] (S.-L.D.) *Correspondence: [email protected] Abstract: Weakly supervised instance segmentation (WSIS) provides a promising way to address instance segmentation in the absence of sufficient labeled data for training. Previous attempts on WSIS usually follow a proposal-based paradigm, critical to which is the proposal scoring strategy. These works mostly rely on certain heuristic strategies for proposal scoring, which largely hampers the sustainable advances concerning WSIS. Towards this end, this paper introduces a novel framework for weakly supervised instance segmentation, called Weakly Supervised R-CNN (WS-RCNN). The basic idea is to deploy a deep network to learn to score proposals, under the special setting of weak supervision. To tackle the key issue of acquiring proposal-level pseudo labels for model training, we propose a so-called Attention-Guided Pseudo Labeling (AGPL) strategy, which leverages the local maximal (peaks) in image-level attention maps and the spatial relationship among peaks and proposals to infer pseudo labels. We also suggest a novel training loss, called Entropic OpenSet Loss, to handle background proposals more effectively so as to further improve the robustness. Comprehensive experiments on two standard benchmarking datasets demonstrate that the proposed WS-RCNN can outperform the state-of-the-art by a large margin, with an improvement of 11.6% on PASCAL VOC 2012 and 10.7% on MS COCO 2014 in terms of mAP 50 , which indicates that learning-based proposal scoring and the proposed WS-RCNN framework might be a promising way towards WSIS. Keywords: weakly supervised learning; instance segmentation; proposal scoring network 1. Introduction Instance segmentation [ 1 – 4 ] refers to the task of jointly localizing, categorizing and segmenting the spatial extents of individual visual objects from a given image. Like many other computer vision tasks, remarkable progress has recently been made on instance segmentation driven by the prosperity of convolutional neural networks (CNN) [ 2 , 4 – 7 ]. Nevertheless, CNN-based solutions to vision tasks commonly suffer from the data-hungry nature, i.e., the necessity of a large amount of annotated data for training. This may be particularly infeasible for instance segmentation since pixel-wise annotations at the instance level are extremely labor-intensive. Weakly supervised instance segmentation (WSIS) [ 8 , 9 ] is one possible way to alleviate the dependency on such strong annotations, which aims to achieve instance segmentation by the use of weaker and thus less laborintensive annotations [ 10 – 12 ], ideally image-level labels only [ 8 ] as we are concerned with in the present work. There recently emerge a couple of attempts on WSIS with image-level labels in the literature [ 8 , 9 , 13 – 15 ], which can be typically outlined as three major steps: (1) proposal generation, i.e., a number of class-agonistic segment proposals are generated from the given image; (2) proposal scoring, i.e., classification scores are assigned to the proposals; (3) postprocessing, i.e., final results are retrieved from the scored proposals by using nonmaximal suppression or certain postprocessing procedures (e.g., applying Mask R-CNN for refinement [ 14 ]). It has been well-established that CNN classifiers trained globally at Sensors 2021,21, 3475. https://doi.org/10.3390/s21103475 https://www.mdpi.com/journal/sensors 105 Sensors 2021,21, 3475 the image level have remarkable ability of spatial localization [ 16 , 17 ], and the so-called attention maps of a certain form (e.g., Class Activation Maps [ 18 ], Occlusion Maps [ 19 ], Saliency Maps [ 20 ], Excitation Backprop [ 17 ]) are utilized to represent such localization cues, where the intensity stands for the possibility of spatial occurrence of visual objects. Taking advantage of this fact, previous works on WSIS mainly put their efforts on the step of proposal scoring, i.e., how to infer reasonable classification scores for the proposals from global attention maps. As a pioneering work on WSIS, Zhou et al. [ 8 ] proposed Peak Response Map (PRM) to boost the instance localization ability of CNNs, which then enables better proposal scoring. Zhu et al. [ 13 ] presented the Instance Activation Map which aims to enhance the spatial extents of instances in PRM and consequently improves proposal scoring. Ahn et al. [ 14 ] deployed the Inter-pixel Relation Network to learn a high-quality proposal generator, as well as pairwise semantic affinities which guide the proposal scoring. While these wisdoms of making improvements on proposal scoring (or other particular components) can indeed benefit to an extent, previous works on WSIS share one major limitation that they commonly follow a heuristic way to exploit attention maps for proposal scoring, lacking of a unified framework. Concretely, they utilize certain attention maps and hand-crafted scoring rules to assign classification scores to proposals [ 8 , 13 ]. In this way, the attention maps are expected to be aware of the spatial extents of visual objects, such that they can match the shapes of true objects and consequently assign them with high scores. On the other hand, attention maps can only sparsely highlight some sites discriminative for classification, usually at very coarse spatial solutions and unaware of object extents. Such discrepancy explains why previous works all devote their efforts to enhancing the perception of object extents of attention maps. Nevertheless, this is by nature a rather difficult perceptual grouping task (see Figure 1), which may limit the substantial advances on this topic. These observations motivate us to consider if these exist a simpler but more effective way towards WSIS. Figure 1. Illustration of the motivation of our work. Existing proposal-based approaches to WSIS commonly follow a heuristic way to exploit attention maps for proposal scoring. In order to assign high scores to the proposals of true objects, the attention maps are expected to be aware of the spatial extents of objects, which is also the main focus of previous efforts on WSIS. But unfortunately, this is by nature a very difficult perceptual grouping task, since attention maps can only have very coarse resolution sparsely highlighting some discriminative sites (unaware of object extents). For intuition, we show some exemplary results obtained by the poineering approach of PRM [ 8 ], where ( a )is the original image, ( b ) is the PRM of the cow class, and ( c , d ) are two proposals and the obtained classification scores. As can be seen, the more favorable proposal in ( c ) is undesirably assigned with a lower score. In this paper, we introduce a general framework for weakly supervised instance segmentation, called Weakly Supervised R-CNN (WS-RCNN). The underlying idea is 106 Sensors 2021,21, 3475 Algorithm 1 Attention-Guided Pseudo Labeling (AGPL) Input: The label vector y=[y1 , ... , yC]∈{ 0,1 }C×1 , the segment proposals {Rn}N n=1 and the class-specific attention maps {Mc}C c=1 (associated with the image I ); the parameter β. Output: The one-hot pseudo class labels {zn}N n=1for the proposals. 1: Initialize zn=0,Q←∅. 2: for all cwith yc=1 (target class) do 3: Find the local maxima {pc i}mc i=1in Mc; 4: for i=1, . . ., mcdo 5: Calculate the support mask Sc iusing Equations (3) and (4); 6: Q←(pc i,Sc i); 7: end for 8: end for 9: Sort Qin the descending order of the values {Mc(pc i)}, denoted by ˜ Qthe sorted set. 10: L←{1, . . . , N}; 11: for all (pc i,Sc i)∈˜ Qdo 12: Find the proposals indexed by I⊆Lsatisfying Equation (5); 13: Set (zn)c=1, ∀n∈I; 14: L←L\I; 15: end for To address this issue, we observe that the task of background handling here is by nature an open-set recognition (OSR) problem, which has been well studied in robust pattern recognition. Hence, we propose to introduce the Entropic Open-Set Loss (OSEL), which is a representative method for OSR [ 25 ], to address our background handling problem. The basic idea of OSEL is to treat the samples from target and background classes separately in establishing the training loss. For target classes, the standard cross-entropy loss is used, while for the background class, an entropic loss is used to encourage predicting uniformly-distributed classification scores. Since the C scores sum up to 1 (output by the softmax layer), encouraging uniform distribution on background class will make these scores small and therefore suppressed during the Non-maximal Suppression (NMS) procedure. Formally, suppose the predicted score vector of a proposal is x=[x1 , ... , xC]∈ RC×1 and the corresponding one-hot pseudo label vector is z=[z1 , ... , zC]∈{ 0,1 }C×1 , the EOSL is defined by EOSL(x,z)=⎧ ⎪ ⎨ ⎪ ⎩ −zclog xc,ifzc=0, − C ∑ i=1 log xi,ifz=0.(6) To our knowledge, this is the first work which addresses the background handling problem in object detection/instance segmentation from the perspective of open-set recognition. 3.5. Remarks Despite the proposed WS-RCNN is very natural and simple, we will later experimentally demonstrate that it has overwhelming superiority in performance over the state-of-the-art methods, which is very likely due to two key insights of our approach. First, we adopt a learning-based approach to proposal scoring, which is advantageous in being able to learn to directly map feature representations of proposals to classification scores, rather than depending on hand-crafted scoring rules. Second, AGPL only utilizes very loose information to achieve pseudo labeling, i.e., some sparse points (peaks in attention maps) and their spatial relationship with proposal regions, which makes our approach much easier and thereby more robust to complex scenes in comparison with existing works that involve the consideration of spatial extents of objects. The proposed WSRCNN frame- 113 Sensors 2021,21, 3475 work can be simply interpreted as “learning to score proposal under weak supervision”, which may provide a promising new perspective for addressing WSIS. 4. Experiments In this section, we perform extensive experiments to evaluate the effectiveness of the proposed WS-RCNN, mainly including: (1) comparison with the state-of-the-art WSIS methods, both quantitatively and qualitatively; (2) validation of the effectiveness of some key components by comparing with variant baselines; (3) parameter study; (4) analysis on failure patterns. Our method was implemented in PyTorch on a workstation with 2 Nvidia Titan XP GPUs, Intel Core(TM) i7-8700 3.70 GHz CPU, 32 GB RAM and Ubuntu 18.04 OS. 4.1. Experimental Setup 4.1.1. Datasets Two well-known benchmarking datasets for instance segmentation, namely PASCAL VOC 2012 (termed as VOC )[ 53 ] and Microsoft COCO 2014 (termed as COCO )[ 54 ]are adopted for our experiments throughout this paper. VOC may be the most representative one for evaluating WSIS methods [ 8 , 13 , 33 ], which includes 10,582 images for training (trainset) and 1449 images for validation (valset) from 20 object classes. COCO is a much larger and more challenging dataset, including 82,783 images for training (trainset) and 40,775 images for validation (valset) from 80 object classes. This dataset has rarely been utilized for the task of WSIS before, and we consider it to enable more comprehensive evaluation. For both datasets, we take trainset for training and valset for testing, and no other annotations except for image-level class labels are used for training according to our problem definition of WSIS. 4.1.2. Performance Metrics We adopt mAP r , the most commonly-used performance metric for instance segmentation [ 8 , 13 , 33 , 53 ], for quantitative evaluation and comparison, where r is the IoU threshold utilized to calculate the metric [ 53 , 55 ]. In our experiments on WSIS, we use mAP 50 ( r= 0.5) as the major metric for comparison and analysis, but we also report mAP 25 (r=0.25) and mAP 75 (r=0.75) for more in-depth evaluation. 4.1.3. Implementation Details We use VGG-16 [ 56 ] pre-trained on ImageNet as the backbone for the proposed WSRCNN. The number of segment proposals per image is set to be N= 200 following [ 8 , 13 ]. For training WS-RCNN, the SGD optimizer is used with a initial learning rate of 5 × 10 −4 for the first 3.5 × 10 4 iterations. In the following 10 4 iterations, the learning rate decreases to 5 × 10 −5 . For data augmentation, we use five image scales { 480,576,688,864, 1200 } (for the shorter side) with horizontal flips for both training and testing. 4.2. Comparison with State-of-the-Art The study on WSIS is still at its early stage and there have not been many works so far. We consider four state-of-the-art WSIS methods in the literature for our comparative study, termed as PRM [ 8 ], IAM [ 13 ], Label-PEnet [ 9 ] and IRnet [ 14 ] respectively. For PRM [ 8 ] and IRnet [ 14 ], we use the source codes as well as configurations provided by the authors themselves. For IAM [ 13 ] and Label-PEnet [ 9 ], since no source codes are publicly released, we directly cite the results in the original literature [ 9 , 13 ] wherever available (marked with “-” if the results are unavailable). Besides, we also construct a variant of WSDDN [ 44 ], which is a very impactful method for weakly supervised object detection (rather than instance segmentation), termed as WSDDN-seg . For this sake, WSDDN-seg can be easily adapted to our task utilizes segments as the proposals to replace the original bounding-boxes in WSDDN (while keeping everything else unchanged). We consider WSDDN-seg because this method is spiritually similar to ours in that it also deploys a deep network to learn to score proposals. For fair comparison, we use the same method [ 51 , 52 ] 114 Sensors 2021,21, 3475 to generate segment proposals for all the compared methods except IRnet [ 14 ], because the method focuses on improving the component of proposal generation. In addition, some methods perform post-processing (typically taking the obtained instance masks as pseudo annotations to run Mask R-CNN like [ 14 ]) to further refine the results while others do not. For fair and comprehensive comparison, we consider the two settings separately, i.e., without and with (marked by “+p”) using Mask R-CNN for refinement. 4.2.1. Results on VOC The quantitative results obtained by various methods on VOC are reported in Table 1. As can be observed, our WS-RCNN outperforms the state-of-the-art (IRnet [ 14 ]) by 11.6% in terms of the major metric mAP 50 , which also achieves the best performance in terms of other metrics, either without or with refinement. Relative to the pioneering work of PRM [ 8 ], all the other methods make an improvement to an extent by enhancing certain aspects, including localization maps (IAM [ 13 ] and IRnet [ 14 ]), proposal generation (IAM [ 13 ]) and combination with object detection (Label-PEnet [ 9 ]). Comparatively, our WS-RCNN is able to improve by a far larger margin, demonstrating its effectiveness. Interestingly, one can observe that WSDDN-seg [ 44 ] can manage to achieve comparable or even better results than PRM [ 8 ]. Notice that WSDDN-seg exploits a CNN-based model to simply learn to score proposals (like ours), even without using any localization map at all. This may suggest that proposal scoring is indeed critical to WSIS with a large room for improvement, and “learning to score” is a promising strategy worthy of further exploration. We also show some comparisons of instance segmentation performance under different supervision on VOC in Table 2, including SDI [ 10 ], Mask R-CNN [ 2 ] and our WS-RCNN. SDI [ 10 ] uses bounding box supervision and Mask R-CNN [2] uses full supervision. Table 1. Quantitative results obtained by various WSIS methods on VOC. Methods mAP50 mAP25 mAP75 mAP50(+p)mAP25(+p)mAP75(+p) PRM [8] 26.8 44.3 9.0 38.0 52.8 14.1 IAM [13] 28.8 45.9 11.9 - - - Label-PEnet [9] 30.2 49.1 12.9 - - - IRnet [14] 31.1 49.2 10.7 46.7 60.5 15.6 WSDDN-seg [44] 27.5 47.5 9.8 43.7 55.9 16.9 WS-RCNN (ours) 42.7 57.2 19.4 47.3 62.2 19.8 Table 2. Instance segmentation performance under different supervision on VOC. Method Supervision mAP50 mAP25 mAP75 SDI [10] Box-Level 44.8 - 16.3 Mask R-CNN [2] Fully-Supervised 69.0 76.7 52.5 WS-RCNN Image-Level 47.3 62.2 19.8 To intuitively justify the motivation and merits of WS-RCNN, we comparatively visualize in Figure 5two sets of representative intermediate results, namely, the scores of a same set of proposals acquired by PRM [ 8 ], WSDDN-seg [ 44 ] and our WS-RCNN (shown in the descending order of the proposal scores). One can observe that WSDDN- seg [44] tends to highlight one dominating instance while ignoring the others of the same class, and PRM [ 8 ] tends to highly score object parts or adjoining objects undesirably. Comparatively, our method can get more favorable proposal scores, which we argue is likely due to our advantageous learning mechanism. More representative results obtained by the various methods are further visualized in Figure 6. 115 Sensors 2021,21, 3475 Figure 5. ( a , b ) are visualization of two sets of representative proposal scores obtained by various methods. In each set, from top to bottom are the results of PRM [ 8 ], WSDDN-seg [ 44 ] and our WS-RCNN respectively. In each row, the first column is the final WSIS result, and the second to sixth columns are the scores of the five proposals obtained by the method. Figure 6. Representative results on VOC obtained by various methods. In each row, from left to right are ( a ) the input image, ( b ) the ground truth, and the results of ( c ) PRM [ 8 ], ( d ) IRnet [ 14 ], ( e ) WSDDN-seg [ 44 ] and ( f ) our WS-RCNN, respectively. Notice that we always output the same number of instances as that in the ground truth for all the methods to enable in-depth comparison. 4.2.2. Results on COCO Table 3shows the quantitative results on the COCO dataset. Notice that COCO has not been considered in previous works compared, and the source codes of IAM [ 13 ] and Label-PEnet [ 9 ] are not publicly released, so the results for these two methods are unavailable. This dataset is far more difficult than VOC, as can be seen by the much worse overall performance of all the methods. However, our WS-RCNN can still outperform the compared methods, with a remarkable margin of 10.7% in terms of mAP 50 over the state- of-the-art (IRnet [ 14 ]), and 4.6% over WSDDN-seg constructed by ourselves. The relative performance of the compared methods in terms of other metric remains consistent with those on VOC, which further verifies the effectiveness of our approach. Some typical results are presented in Figure 7, which further demonstrate that our WS-RCNN can obtain more favorable instance masks, especially in case of multiple or adjoining instances. 116 Sensors 2021,21, 3475 Table 3. Quantitative results obtained by various WSIS methods on COCO. Methods mAP50 mAP25 mAP75 mAP50(+p)mAP25(+p)mAP75(+p) PRM [8] 5.8 12.1 1.8 14.7 23.7 6.6 IRnet [14] 7.3 13.9 2.4 13.2 22.2 4.8 WSDDN-seg [44] 14.4 24.0 6.0 23.0 30.9 11.5 WS-RCNN (ours) 18.0 27.4 7.2 24.2 32.1 11.6 Figure 7. Representative results on COCO obtained by various methods. In each row, from left to right are ( a ) the input image, ( b ) the ground truth, and the results of ( c ) PRM [ 8 ], ( d ) IRnet [ 14 ], ( e ) WSDDN-seg [ 44 ] and ( f ) our WS-RCNN, respectively. Notice that we always output the same number of instances as that in the ground truth for all the methods to enable in-depth comparison. 4.3. Validation of Key Components The superiority of the proposed WS-RCNN, to a large extent, should be attributed to the considerate design of its key components. We further carry out experiments on the VOC dataset to validate this point. For each component of concern, we construct some variants to replace the original one in WS-RCNN while keeping everything else unchanged, and compare the overall performance in terms of mAP 50. 4.3.1. Pseudo Labeling Strategy We consider the following three variants of AGPL: • PLS-1: For each pc i we assign those proposals which spatially include this point to the class c , i.e., assigning zn=c if pc i∈Rn , without considering the support mask Sc i (removing Line 5-8 in Algorithm 1and meanwhile changing the condition in Equation (5)tobe“pc i∈Rn”). • PLS-2: For each pc i we thresold Mc (using the threshold value of 0.5) and take the connected component surrounding this peak as the support mask Sc i (changing Line 6 in Algorithm 1). • PLS-3: We simply adopt the method of PRM [ 8 ] for pseudo labeling, i.e., taking the proposal classification results obtained by the whole pipeline in [ 8 ] as the pseudo labels (changing Line 6 in Algorithm 1). The results are reported in Table 4, which shows all these variants significantly underperform the proposed AGPL. The results validate the merit of AGPL, i.e., relying on peaks in attention maps and the spatial relationship between peak points and proposal regions to achieve pseudo labeling. 117 Sensors 2021,21, 3475 Table 4. Comparison with Various Pseudo Labeling Strategies on VOC. PLS-1 PLS-2 PLS-3 Ours 24.0 25.2 36.2 42.7 4.3.2. Training Strategy In our WS-RCNN, we adopt a particular two-phase strategy for network training (see Section 3.2. To verify the effectiveness of such a design, we construct two different training strategies for comparative study as follows: • TS-coupled: When training the proposal scoring branch in the second phase, the backbone CNN is not reinitialized by the parameters pre-trained on ImageNet, but by those obtained during the training of the image-level classifier in the first phase. • TS-joint: The image-level classifier is trained first, and the proposal scoring branch is trained jointly with the image-level classifier, i.e., combining the two training losses when training this branch. The results in Table 5demonstrate that the training strategy adopted by our WS-RCNN are more advantageous than the variants compared. Table 5. Comparison with Various Network Structures and Training Strategies on VOC. TS-Coupled TS-Joint Ours 36.5 37.6 42.7 4.3.3. Training Loss To validate the effectiveness of our proposed EOSL loss, we compare it with a widelyused strategy for background handling as below • LOSS-CE: We add a dummy class to accommodate background proposals and adopt the conventional binary cross-entropy to replace the EOSL loss in Equation (6). As can be observed from Table 6that our EOSL loss performs much better. Table 6. Comparison with Different Training Loss for Background Handling on VOC. LOSS-CE Ours 34.5 42.7 4.4. Parameter Study Our approach involves specifying two parameters, i.e., the number of proposals N and the threshold value β in AGPL. Here we further conduct experiments to analyze the impact of these parameters on performance. 4.4.1. The Number of Proposals N We vary the number of proposals to be N={ 50,100,150, 200,300,400,500 } , and report the results in Figure 8a. As can be seen, the performance will not increase significantly after N> 200, and we adopt N= 200 throughout our experiments for the best tradeoff between performance and efficiency. 118 Sensors 2021,21, 3475 Figure 8. Impacts of varying the key parameters ( a ) the number of proposals N and ( b ) the threshold value βin AGPL. 4.4.2. The Threshold βin AGPL We vary the threshold value to be β={ 0.4,0.5,0.6,0.7,0.8,0.9 } , and the results are depicted in Figure 8b. One can observe that the performance is relatively stable when β∈[0.5,0.8]and we choose a fixed value of β=0.7 throughout our experiments. 4.5. Failure Cases Despite the effectiveness of WS-RCNN, WSIS is essentially a challenging task and the overall performance still has much room for improvement. Here we show several typical failure cases of WS-RCNN in Figure 9. Since our method is proposal-based, it relies much on the quality of proposals. If the proposal generator fails to cover the spatial extents of the true instance, our method will fail consequently (see Figure 9a). Another typical failure case is that if there are a number of instances of a class with overlap among or close to each other, our method may pick up those large proposals which cover multiple instances (see Figure 9b) or the small proposals which cover only a part of a instance (Figure 9c) (Notice that the desired proposals are present in (b) and (c). Figure 9. Typical failure cases of WS-RCNN. ( a ) Poor proposals. ( b ) Picking up large proposals covering multiple instances. (c) Picking up small proposals covering only a part of a instance. 5. Conclusions In this paper, we have presented a simple, natural but surprisingly effective framework, termed as Weakly Supervised R-CNN (WS-RCNN), for weakly supervised instance segmentation (WSIS). The basic idea is to deploy a deep network to learn to score proposals under the particular setting of weak supervision. For this sake, a strategy called Attention- Guided Pseudo Labeling is proposed to address the key issue of proposal-level pseudo labeling. And a so-called Entropic Open-Set Loss is introduced for model training to further improve the robustness. Comprehensive experiments on two well-known datasets, i.e., PASCAL VOC 2012 and Microsoft COCO 2014, have demonstrated that the proposed WS-RCNN can significantly outperform the state-of-the-art. Experiments have also been carried out to validate effectiveness of the key components and to study the impacts of some key parameters. In our future work, we will consider integrating a learning-based proposal network to replace the current heuristic proposal generator in WS-RCNN in order to alleviate its dependency on the quality of proposals. We will also explore how to 119 Sensors 2021,21, 3475 further improve our approach in dealing with complex scenes (e.g., complex background, multiple or adjoining instances of the same class) since there is generally much room for improvement. Author Contributions: Methodology, J.-R.O.; writing—original draft preparation, J.-R.O., S.-L.D.; writing—review and editing, J.-R.O., S.-L.D., J.-G.Y.; project administration, J.-G.Y.; funding acquisition, J.-G.Y. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by the Key R&D Program of Guangdong Province under Grant 2018B030339001 and National Natural Science Foundation of China under Grants 62076099, and also in part by the Guangzhou Science and Technology Program under Grant 201904010299. Institutional Review Board Statement: The study did not involve humans or animals. Informed Consent Statement: The study did not involve humans. Data Availability Statement: The study did not report any data. Conflicts of Interest: The authors declare no conflict of interest. References 1. Hariharan, B.; Arbeláez, P.; Girshick, R.; Malik, J. Simultaneous detection and segmentation. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 297–312. 2. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 Ocotber 2017; pp. 2961–2969. 3. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. 4. Liu, S.; Qi, L.; Qin, H.; Shi, J.; Jia, J. Path aggregation network for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 8759–8768. 5. Chen, L.C.; Hermans, A.; Papandreou, G.; Schroff, F.; Wang, P.; Adam, H. Masklab: Instance segmentation by refining object detection with semantic and direction features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4013–4022. 6. Chen, K.; Pang, J.; Wang, J.; Xiong, Y.; Li, X.; Sun, S.; Feng, W.; Liu, Z.; Shi, J.; Ouyang, W.; et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 4974–4983. 7. Jeong, D.; Kim, B.G.; Dong, S.Y. Deep joint spatiotemporal network (DJSTN) for efficient facial expression recognition. Sensors 2020,20, 1936. [CrossRef] 8. Zhou, Y.; Zhu, Y.; Ye, Q.; Qiu, Q.; Jiao, J. Weakly supervised instance segmentation using class peak response. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3791–3800. 9. Ge, W.; Guo, S.; Huang, W.; Scott, M.R. Label-PEnet: Sequential label propagation and enhancement Networks for Weakly Supervised Instance Segmentation. In Proceedings of the IEEE International Conference on Computer Vision, Long Beach, CA, USA, 16–20 June 2019; pp. 3345–3354. 10. Khoreva, A.; Benenson, R.; Hosang, J.; Hein, M.; Schiele, B. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Venice, Italy, 22–29 Ocotber 2017; pp. 876–885. 11. Remez, T.; Huang, J.; Brown, M. Learning to segment via cut-and-paste. In Proceedings of the European Conference on Computer Vision, Salt Lake City, UT, USA, 18–23 June 2018; pp. 37–52. 12. Kuo, W.; Angelova, A.; Malik, J.; Lin, T.Y. Shapemask: Learning to segment novel objects by refining shape priors. In Proceedings of the IEEE International Conference on Computer Vision, Long Beach, CA, USA, 16–20 June 2019; pp. 9207–9216. 13. Zhu, Y.; Zhou, Y.; Xu, H.; Ye, Q.; Doermann, D.; Jiao, J. Learning instance activation maps for weakly supervised instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 3116–3125. 14. Ahn, J.; Cho, S.; Kwak, S. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 2209–2218. 15. Arun, A.; Jawahar, C.; Kumar, M.P. Weakly supervised instance segmentation by learning annotation consistent instances. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 254–270. 16. Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Object detectors emerge in deep scene cnns. arXiv 2014 , arXiv:1412.6856. 17. Zhang, J.; Bargal, S.A.; Lin, Z.; Brandt, J.; Shen, X.; Sclaroff, S. Top-Down Neural Attention by Excitation Backprop. Int. J. Comput. Vis. 2018,126, 1084–1102. [CrossRef] 18. Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2921–2929. 120 Sensors 2021,21, 3475 19. Zeiler, M.D.; Fergus, R. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 818–833. 20. Simonyan, K.; Vedaldi, A.; Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv 2013, arXiv:1312.6034. 21. Girshick, R. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 13–16 December 2015; pp. 1440–1448. 22. Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Zurich, Switzerland, 6–12 September 2014; pp. 580–587. 23. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015; pp. 91–99. 24. Geng, C.; Huang, S.j.; Chen, S. Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2020 . [CrossRef][PubMed] 25. Dhamija, A.R.; Günther, M.; Boult, T. Reducing network agnostophobia. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 3–8 December 2018; pp. 9157–9168. 26. Laradji, I.H.; Vazquez, D.; Schmidt, M. Where are the Masks: Instance Segmentation with Image-level Supervision. arXiv 2019 , arXiv:1907.01430. 27. Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Santiago, Chile, 13–16 December 2015; pp. 3431–3440. 28. Kolesnikov, A.; Lampert, C.H. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 8–16 October 2016; pp. 695–711. 29. Roy, A.; Todorovic, S. Combining bottom-up, top-down, and smoothness cues for weakly supervised image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 3529–3538. 30. Pinheiro, P.O.; Collobert, R. From image-level to pixel-level labeling with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1713–1721. 31. Papandreou, G.; Chen, L.C.; Murphy, K.; Yuille, A. Weakly-and semi-supervised learning of a DCNN for semantic image segmentation. arXiv 2015, arXiv:1502.02734. 32. Huang, Z.; Wang, X.; Wang, J.; Liu, W.; Wang, J. Weakly-supervised semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7014–7023. 33. Ahn, J.; Kwak, S. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4981–4990. 34. Wei, Y.; Xiao, H.; Shi, H.; Jie, Z.; Feng, J.; Huang, T.S. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7268–7277. 35. Zhang, X.; Wei, Y.; Feng, J.; Yang, Y.; Huang, T.S. Adversarial complementary learning for weakly supervised object localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1325–1334. 36. Li, K.; Wu, Z.; Peng, K.C.; Ernst, J.; Fu, Y. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 9215–9223. 37. Wang, X.; You, S.; Li, X.; Ma, H. Weakly-supervised semantic segmentation by iteratively mining common object features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1354–1362. 38. Hou, Q.; Jiang, P.; Wei, Y.; Cheng, M.M. Self-erasing network for integral object attention. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 3–8 December 2018; pp. 549–559. 39. Fan, R.; Hou, Q.; Cheng, M.M.; Yu, G.; Martin, R.R.; Hu, S.M. Associating inter-image salient instances for weakly supervised semantic segmentation. In Proceedings of the European Conference on Computer Vision, Salt Lake City, UT, USA, 18–23 June 2018; pp. 367–383. 40. Fan, R.; Cheng, M.M.; Hou, Q.; Mu, T.J.; Wang, J.; Hu, S.M. S4Net: Single stage salient-instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 6103–6112. 41. Zhang, C.; Platt, J.C.; Viola, P.A. Multiple instance boosting for object detection. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 4–7 December 2006; pp. 1417–1424. 42. Ren, W.; Huang, K.; Tao, D.; Tan, T. Weakly supervised large scale object localization with multiple instance learning and bag splitting. IEEE Trans. Pattern Anal. Mach. Intell. 2015,38, 405–416. [CrossRef][PubMed] 43. Wang, X.; Zhu, Z.; Yao, C.; Bai, X. Relaxed multiple-instance SVM with application to object discovery. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 13–26 December 2015; pp. 1224–1232. 121 Sensors 2021,21, 3475 44. Bilen, H.; Vedaldi, A. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2846–2854. 45. Diba, A.; Sharma, V.; Pazandeh, A.; Pirsiavash, H.; Van Gool, L. Weakly supervised cascaded convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 914–922. 46. Tang, P.; Wang, X.; Wang, A.; Yan, Y.; Liu, W.; Huang, J.; Yuille, A. Weakly supervised region proposal network and object detection. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 352–368. 47. Tang, P.; Wang, X.; Bai, S.; Shen, W.; Bai, X.; Liu, W.; Yuille, A. Pcl: Proposal cluster learning for weakly supervised object detection. IEEE Trans. Pattern Anal. Mach. Intell. 2018,42, 176–191. [CrossRef][PubMed] 48. Wan, F.; Wei, P.; Jiao, J.; Han, Z.; Ye, Q. Min-Entropy Latent Model for Weakly Supervised Object Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2019,41, 2395. [CrossRef][PubMed] 49. Shen, Y.; Ji, R.; Yang, K.; Deng, C.; Wang, C. Category-Aware spatial constraint for weakly supervised detection. IEEE Trans. Image Process. 2019,29, 843–858. [CrossRef][PubMed] 50. Bendale, A.; Boult, T.E. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 July 2016; pp. 1563–1572. 51. Pont-Tuset, J.; Arbelaez, P.; Barron, J.T.; Marques, F.; Malik, J. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell. 2016,39, 128–140. [CrossRef][PubMed] 52. Maninis, K.K.; Pont-Tuset, J.; Arbeláez, P.; Van Gool, L. Convolutional oriented boundaries: From image segmentation to high-level tasks. IEEE Trans. Pattern Anal. Mach. Intell. 2017,40, 819–833. [CrossRef][PubMed] 53. Everingham, M.; Eslami, S.A.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vis. 2015,111, 98–136. [CrossRef] 54. Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 740–755. 55. Hariharan, B.; Arbeláez, P.; Bourdev, L.; Maji, S.; Malik, J. Semantic contours from inverse detectors. In Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 991–998. 56. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. 122 Metals 2021,11, 383 2.3.5. Cytotoxicity and Cell Viability. We cultured 5 × 10 5 Jurkat cells cells in a 48-well plate, in the absence or presence of NPs at the indicated concentrations. Cell viability was analyzed 24 h after addition of NPs by a As previously commented, one key property (MTT)-based assay as previously described. Briefly, MTT reactant (thiazolyl blue tetrazolium bromide, TOX1-1KT, “Millipore-Sigma”, Burlington, MA, USA) was added to the cells in a 1:10 ratio (MTT solution/culture medium) and incubated during 3 h at 37 ◦ C. Then, formazan crystals, formed inside viable cells, were dissolved by adding MTT Solubilization Solution M-8910, (“Millipore-Sigma”, Burlington, MA, USA) at 1:2 with vigorous pipetting. Optical density at 570 nm was evaluated to quantify the amount of formazan crystals, which is proportional to the number of viable cells (background absorbance was measured at 690 nm and subtracted from the 570 nm measurement). Viability of cell in the presence of NPs was compared to untreated controls (considered 100% viability). Media in the absence of cells was used as blank; while cells, cultured in the presence of 10% dimethyl sulfoxide (DMSO), was used as control for decreased cell viability, being always below 10%. Statistical analyses were performed using the Statgraphics software (“Statpoint Technologies”, Warrenton, VA, USA ). Significances were determined by using analyses of variance (ANOVA) and multiple range tests. 3. Results As previously commented, in order to determine the potential of the presented NPs for biomedicine applications, it is required to previously study their structural properties, their optical behavior, as well as additional features such as colloidal stability at physiological values of salinity and pH, colloidal stability at temperatures ranging from 35 to 40 ◦ C and cytotoxicity. 3.1. Structural Features and Physical Properties Figure 2shows results on the size and optical properties of Au-GSH samples, as well as an overview on the state of aggregation of the NPs when deposited on the holed carbon grid. A general view of these particles can be observed in Figure 2a, which shows a HAADF-STEM micrograph. This technique is ideal for imaging gold-based particles, due to the high atomic number of gold, which produces a high contrast with the carbon support in HAADF images. This micrograph reveals the formation of relatively small and well-dispersed NPs with nearly round shape. Figure 2b shows the NP size distribution obtained from more than 300 NPs in this and similar images. From this histogram, it can be affirmed that particle size is homogeneous, with an average diameter of 3.8 nm (see Table 1 , in which particle diameter for all materials are collected), obtained from the fitting of this distribution to a Gaussian function. In any case, although this type of images shows, with a good contrast, the size and shape of the particles, hinting on the high atomic number of its constituent material, the exact chemical composition of the NPs is not determined with this technique. The presence of gold in this nanostructure can be confirmed by the UV-Visible absorption spectrum of the colloidal solution (Figure 2c). The inset in the figure is a photograph of the obtained red-wine colored colloidal solution. The spectrum shows the characteristic 520 nm LSPR absorption band, indicating the presence of small gold NPs. Position and width of LSPR bands give information on NP size and size homogeneity. The Au-GSH LSPR band appears as a shoulder, showing a wide profile which can be a consequence of the small average size of these NPs. The high number of surface atoms in small NPs increases the damping of oscillating electrons at conduction bands. This effect can be even higher for small thiol-capped NPs, due to the loss of itinerancy of electrons involved in Au-S bonding [ 46 ]. This ligand-metal charge transfer transition, produced due to the capping, can contribute to the relatively strong emission bands, in the blue range of the spectra, for the Au-GSH NPs, as shown in Figure 2d, where the PL emission spectra, recorded for the Au NPs colloidal solution (exciting at 317 nm), is shown. This emission could be an interesting feature for the use of this NPs for biomarker applications [42]. 129 Metals 2021,11, 383 Figure 2. High-angle annular dark field (HAADF) image ( a ), as well as the histogram displaying particle size distribution ( b ) of Au-GSH NPs. ( c ) UV-Visible absorption spectrum and PL emission spectrum ( d ) of Au-GSH NPs. The inset in Figure 2c shows a digital photo image of an Au-GSH colloidal solution. Table 1. Average diameter obtained via transmission electron microscopy (TEM), dynamic light scattering (DLS) hydrodynamic diameter and polydispersity index (PDI). Sample TEM Average Size (nm) DLS Average Size (nm) PDI (%) Au-GSH 3.8 ±0.2 6.2 ±1.2 0.2 Mag-GSH 6.1 ±0.4 12.7 ±1.2 0.2 QD-GSH 2.9 ±0.3 32.6 ±1.3 0.1 Au-Mag-GSH 6.9 ±0.4 42.3 ±1.3 0.2 Figure 3shows results obtained from the optical and structural characterization of Mag-GSH samples. A general view of the NPs is showed in Figure 3a. The BF-TEM micrograph reveals the obtained particle distribution for this preparation, less dispersed as in the case of gold NPs, most probably due to the small clustering promoted by magnetic interactions. Figure 3b displays and histogram showing the particle size distribution for this sample. Fitting of these results to a Gaussian function reveals a NP average size of 6.1 nm (Table 1). Figure 3c shows the magnetization curve, i.e., applied field versus magnetization, at room temperature. This curve shows the magnetic behavior of these NPs [ 47 ]. The inset in Figure 3c shows the response of Mag-GSH NPs (flask on the left) to a permanent magnetic field, applied using a permanent magnet adjacent to the flask that contains the NP colloidal dispersion. 130 Metals 2021,11, 383 Figure 3. Bright field TEM (BF-TEM) micrograph ( a ), as well as the histogram displaying the particle size distribution ( b ) of Mag-GSH NPs. Magnetization curve of Mag-GSH NPs ( c ). The inset in this figure shows a digital photo image of Mag-GSH NPs placed in the absence and presence of a magnet located close to the vials. Figure 4shows a general view and optical features of QD-GSH NPs. Figure 4a corresponds to a HAADF-STEM image of an accumulation of these QD-GSH NPs. In this case, we can observe the presence of extremely small NPs immersed in the GSH capping. The histogram corresponding to these NPs is shown in Figure 4b. By fitting these results to a Gaussian function, an average NP diameter of 2.9 nm is obtained (Table 1). As we have previously commented, NP size is especially important for their optical properties, determining the wavelength of their emission, which is especially important in this case, since QD functionality in biomedical applications rely on their fluorescence. In this sense, the proposed synthesis allows control of NP final size and thus, its PL emission. Specifically, Figure 4c shows PL emission spectra obtained from the sample in Figure 4a (with average size reflected in Figure 4b), labelled as QD-GSH1 and prepared at a synthesis temperature of 90 ◦ C for 30 min. Two additional QD-GSH samples of bigger average sizes (QD-GSH2, prepared at 90 ◦ C for 3 h, and QD-GSH3, heating during 3 h at 90 ◦ C and one more hour at 110 ◦ C) are also included. As NP size increases (sizes for QD-GSH1, QD-GSH2 and QD-GSH3 are 2.9 nm, 3.6 nm and 3.9 nm, respectively), a red-shift in the emission peak maximum is observed, illustrating wavelength size-dependency of the emission. The photograph of colloidal solutions of these three samples, under UV radiation, is added to this figure, as an inset. As it can be observed from both, PL spectra and photograph of the colloidal solutions; by changing just a few nanometers in size, we can tune the emission color from blue to red. Regarding hybrid Au-Mag-GSH NPs, one of the first and important results to be confirmed after their preparation was that each nanostructure is indeed formed by both iron oxide and gold. Taking into account that the synthesis has been initiated by heterogeneous nucleation of iron oxide seeds, the gold phase necessarily has to be located around the 131 Metals 2021,11, 383 iron oxide phase. In this sense, a relevant measurement is shown in Figure 5a, namely the comparison of UV-Vis absorption spectra for the colloidal solutions of Au-GSH and Au- Mag-GSH samples. Both colloids, display the LSPR absorption band (associated to gold), however, the hybrid Au-Mag NP curve has a pronounced red-shift, from 560 nm to 520 nm. According to previous reported results, this red-shift can be attributed to the reduced electron deficiency of Au NPs caused by the interfacial communication between Au and magnetic-iron-oxide [ 48 ]. Therefore, these results reveal the formation of a hybrid NP, containing gold and magnetic-iron-oxide. The Au-Mag-GSH spectrum, shows a second and less intense near-infrared (NIR) absorption band that could be attributed to the presence of anisotropic NPs or even to iron oxide-gold interactions in the hybrid structure. This characteristic can be of interest in phototherapy applications [ 49 , 50 ]. A photograph of the Au-Mag-GSH colloidal solution is also shown in this figure (see inset image in Figure 5a ). The spectra have not been normalized because the determinant parameters are the LSPR band position, the wavelength at which the LSPR is observed and the shape and width of the band. A BF-TEM micrograph of these particles is shown in Figure 5b, revealing the formation of homogeneously shaped structures, without the presence of clusters as dense as those observed for Mag-GSH. NP size distribution, and its fitting to a Gaussian function is shown in Figure 5c, showing an average particle size of 6.9 nm (Table 1). Figure 4. HAADF image ( a ), as well as the histogram displaying particle size distribution ( b ) of quantum dot (QD)-GSH1 NPs. ( c ) PL-emission spectra for QD-GSH1 (blue) as well as for QD-GSH2 (green) and QD-GSH3 (red), with higher average size. The inset in this figure shows a digital photo image of the three QD-GSH colloidal solutions used in this figure, under UV irradiation. 132 Metals 2021,11, 383 Figure 5. Ultraviolet (UV)–visible spectra for Au-GSH and Au-Mag-GSH colloidal solutions ( a ); TEM micrograph of Au-Mag-GSH (b) and its corresponding histogram displaying NP size distribution (c). Figure 6summarizes the main (S)TEM results on the structure and composition of the different GSH-capped NP samples. At this point, it is worth mentioning that the study via electron beam-based techniques of these materials produced a large amount of data and information. Therefore, more details on the (S)TEM characterization can be found elsewhere [ 45 ]. Figure 6a,b show respectively HRTEM and HR-HAADF micrographs, for Au-GSH NPs taken from the same region. The comparison of these modes, for the same NP, not only allows visualization of the crystalline structure, but also indicates the chemical homogeneity of such structure. Also, it is possible to visualize (Figure 6a) a thin amorphous layer (GSH) covering the NP core. Images such as the ones in Figure 6a,b allow measuring inter-planar distances of 2.17 and 2.38 Å, which respectively agree well with the spacing between {002} and {111} families of atomic planes for cubic Au. These values of inter-planar atomic distances have been obtained using the fast Fourier transform, FFT, (Figure 6a, inset) of a large enough region of the NP that show a crystalline structure with the same orientation. Using a generic diffraction software [ 51 ] to compare the experimental FFT with a simulated diffraction diagram, the zone axis can be identified and the diffraction peaks indexed. Note that FFT data are equivalent to electron diffraction diagrams when in Fourier condition, as is the case for the microscope optics. Similar analyses (not shown here) also revealed that crystalline structures were formed, in all samples and for each type of NP, corresponding to the expected elements and compounds. 133 Metals 2021,11, 383 Figure 6. High-resolution TEM (HRTEM) ( a ) and HR-HAADF ( b ) images of NPs in sample Au-GSH. Punctual energy-dispersive X-ray (EDX) spectrum from QD-GSH ( c ) and from Au-Mag-GSH ( d ) NPs. 134 Metals 2021,11, 383 Compositional sensitive STEM techniques, such as EDX, can complement these results, allowing analyze the uniformity of the GSH layer. In this sense, we have utilized the sulfur (S) peak associated to the K-level transition in order to detect GSH. Figure 6c presents an EDX-punctual spectrum for a NP of the same sample. Such a spectrum is taken from the area defined by the scanning STEM probe, which is estimated to correspond to 5nm×5nm . Thus, the spectrum corresponds, approximately, to a region spanning, the size of one NP. Note, that other expected peaks, such as the ones associated to carbon and oxygen, also presented in the spectrum, are mostly due to both the carbon support and organic residual particles in the electron microscopy grid. In this sense, the copper signal, visible in all spectra is produced by the copper in such grid. In a previous work [ 45 ], EDX-maps for the different elements were presented, so a more comprehensive view of the STEM study for these samples can be obtained from this communication. Note that EDX results, such as those shown in Figure 5f from the study presented in reference 45, only indicate the iron-containing composition of the NP. The affirmation that the NP contains iron oxide would need further evidence. On the other hand, Figure 6d presents the EDX punctual spectrum of isolated NPs from a sample of Au-Mag-GSH. All the main studied elements were found, revealing a hybrid structure in a large amount of NPs. Base on all these results, it can be affirmed that the synthetic routes utilized to prepare the different NPs lead to the formation of small NPs, with homogeneous size distribution, a crystalline structure and a homogeneous composition, both for the core and for the polymer GSH covering layer. The in situ functionalization with GSH probably contributes to this morphological feature. These NPs show suitable characteristics for different biomedical applications, depending on the core composition. However, the use of NPs as potential tools in biomedicine, requires potential capability to be bioconjugated with biological species. The effectiveness of the functionalization process is, in any case, fundamental for subsequent bioconjugations. 3.2. Colloidal Stability and Superficial Properties 3.2.1. Hydrodynamic Size To evaluate the colloidal stability of the different NPs, DLS experiments have been carried out. Figure 7shows the DLS diameter distribution (represented in number) obtained for the different GSH NPs. Each distribution has been fitted to a log-normal function, also shown in the graphics. Colloidal average diameters obtained from these fittings, as well as the PDI (polydispersity index) are shown in Table 1. All NPs show a relatively narrow size distribution with colloidal average sizes smaller than 50 nm, and with a relatively low polydispersity index (PDI in Table 1). As expected, all NPs show DLS hydrodynamic sizes bigger than those obtained from TEM, since this last technique measures mostly the crystalline volume of the particles (as explained before, in most particles, the polymer layer is veiled by the carbon support in the images). This effect is not only due to the GSH layer, but also to the presence of extra hydrate layers in aqueous medium. However, the similar sizes found from both TEM and DLS, in the case of Au-GSH, reveals the high colloidal stability of these NPs. Also, in the case of QD-GSH, Mag-GSH and Mag-Au-GSH, the formation of clusters and particle associations in solution could influence the DSL measurements. In any case, the four types of synthetized GSH-NPs show DLS diameters in the appropriate range for biomedical applications. DLS measurements over time show no important size differences after 48 h post-redispersion for any type of NP, being the size increase lower than 1.2% in all cases. 3.2.2. Z-Potential and Surface Characteristics To find out the electric charge surrounding NP surface we calculated the Z-potential. Figure 8a displays Z-potential versus pH for different NP colloidal solutions. This figure indicates that all NP systems in this work show suitable features for biomedical applications, since the values in the Z-potential graphic reveals a relatively high colloidal stability in the studied pH range, and particularly at physiological pH. 135 Metals 2021,11, 383 These measurements shed information about the sign of the charge on the NP surface (as can be seen, all GSH samples present a negative Z-potential, which indicate an excess electric negative charge at the surface), that is also relevant for potential crosslinking applications. In general, the GSH molecule has thiol, amine and two carboxylic terminal groups available for linking to the NP surface. In the case of CdTe, due to the affinity between cadmium and sulfur, the GSH probably links the NP by its thiol group, keeping free the carboxylic terminal groups, responsible for the negative net charge. In order to study the linking mechanism for gold and iron oxide surfaces, we have prepared iron oxide and gold NPs, using a synthetic method similar to that used for Au-GSH and Mag-GSH NPs, but replacing the GSH with cysteamine (CYS), a molecule containing only two terminal groups: thiol and a primary amine. Table 2shows Z-potential values at pH 7.2 for Au-GSH and Mag-GSH NPs, as well as for Mag-CYS and Au-CYS NPs. In a previous work [ 42 ], we demonstrated that the CYS molecule, links to the gold NP surface by the thiol group, due to the high affinity of thiols for the gold core, thus these CYS-capped NPs show positive Z-potential values due to the free amine terminal groups. However, in this work, we have obtained negative Z-potential for a Mag-CYS colloidal solution. This result points out that the affinity of amine terminal groups for the iron oxide core is higher than for the thiol group. Consequently, the GSH amine terminal group could have higher affinity for the iron oxide core than the GSH thiol group. We deduce that there are two different mechanisms for the linking between CYS and gold versus iron oxide NPs (Figure 8b,c). Interestingly, Mag-GSH NPs show the highest negative Z-potential values in all pH range (Figure 8a). This maximum negative charge could be better explained if we propose a different linking mechanism for this NP, in which GSH binds the NP surface by the amine group, with the thiol and the two carboxylic groups remaining free and, thus, contributing to the negative superficial charge. These results, which indicate that the iron oxide surface shows a higher affinity for amine than for thiol groups, can explain the higher negative Z-potential for the Mag-GSH NPs, due to the GSH linking to the iron oxide surface by its amine group instead of by the thiol group, contrary to what happens for gold NPs (Figure 8d,e). Figure 7. DLS size distributions of Au-GSH ( a ), Mag-GSH ( b ), CdTe-GSH1 ( c ) and Au-Mag-GSH (d) in PBS (phosphate-buffered saline) colloidal solutions. 136 Metals 2021,11, 383 Figure 8. Z-potential versus pH for the different GSH capped NPs ( a ). Diagram of the cysteamine (CYS)-capped ( b ) and GSH-capped (c) magnetic-iron-oxide NPs. Diagram of the CYS-capped (d) and GSH-capped (e) gold NPs. Table 2. Z-potential values obtained for gold and magnetic-iron-oxide NPs capped with CYS and with GSH at 25 ◦C and in PBS solution. Sample Z-Potential (mV) Au-GSH −32.8 Mag-GSH −33.5 Au-CYS +88.0 Mag-CYS −27.5 In order to deepen knowledge of the differences in linking mechanisms between gold and iron oxide surfaces, we performed FTIR spectroscopy. Figure 9shows the FTIR spectra for the free GSH ligand, as well as for Au-GSH, Mag-GSH and Au-Mag-GSH. The GSH spectrum present bands characteristics of amino (3350 cm −1 ), carboxylic ( 1600 cm−1 ) and thiol (2530 cm −1 ) groups. The peak at 2530 cm −1 that corresponds to the S–H stretching vibration of GSH clearly disappeared in the Au-GSH and Au-Mag-GSH spectra, indicating that GSH anchors on the surface of these NPs through Au-S bonding. Conversely, a band at this position can be appreciated for Mag-GSH. The presence of the S–H stretching vibration band in this case, indicates a different GSH anchorage for this type of NP [52]. These FTIR spectra not only show a different GSH linking mechanism for gold and iron oxide NPs, but also suggests that gold is covering the iron oxide core in the case of hybrid iron oxide/gold NPs, as GSH linking occurs as it does for gold, with disappearance of the S–H stretching vibration of GSH. These differences in the linking mechanism could also introduce differences in the shell thickness, as we have previously commented. A schematic representation of this linking mechanism is shown in Figure 8b through Figure 8e. 3.2.3. Temperature Effect NPs used for biological applications require appropriate colloidal stability at human body temperature, ranging from 37 to 40 ◦ C, as well as at different saline concentrations. We studied the stability of the NP dispersions under different environmental conditions (temperature and salinity). DLS experiments were also performed at 37 and 40 ◦ C; and also subjecting NPs to an environment of a higher ionic strength, after dispersion of NPs in a 0.18 M solution of NaCl. Table 3shows NP diameters obtained by DLS for temperatures of 25, 37 and 40 ◦ C for all NPs in PBS dispersions, as well as at 25 ◦ C in a 0.18 M NaCl dispersion. Results reveal that the only NPs that present a colloidal stability dependence with temperature is Au-GSH, which has an extremely low hydrodynamic size at 25 ◦ C. 137 Metals 2021,11, 383 The rest of the NPs do not show a strong variation in the dynamic size when temperature increases. However, when the ionic strength changes, a significant increase in the dynamic diameter is observed for all GSH-NPs. In all cases NPs remain in the appropriate range for biomedical applications. Figure 9. Fourier transform infrared (FTIR) spectra for free GSH ligand as well as for Au-Mag-GSH, Au-GSH and Mag-GSH NPs. Table 3. NPs and their corresponding dynamic light scattering (DLS) hydrodynamic sizes at different temperatures. Sample Temperature (◦C) Dispersion Medium DLSAverage Size (nm) Au-GSH 25 PBS 6.2 ±1.2 37 PBS 24.9 ±2.2 40 PBS 13.5 ±1.5 25 0.18 NaCl 28.2 ±2.5 QD-GSH 25 PBS 32.6 ±1.3 37 PBS 39.3 ±2.7 40 PBS 36.3 ±2.7 25 0.18 NaCl 58.1 ±2.5 Mag-GSH 25 PBS 12.7 ±1.2 37 PBS 11.3 ±1.6 40 PBS 12.9 ±1.7 25 0.18 NaCl 29.2 ±2.6 All these colloidal characterization results confirm that the NP core is capped by GSH molecules that properly functionalize the NP, allowing for subsequent biofunctionalization and thus corroborating previously presented (S)TEM results. Functionalized NPs show 138 Polymers 2021,13, 1597 Table 2. CIP materials with and without surface treatment. Material Property HS CC EW-I Density (g/cm3)7.73 7.89 7.58 d50 (μm) 1.9 4.7 3.4 Surface treatment No SiO2SiO2+ phosphate The magnetorheological thermoplastic elastomers were mixed in a torque rheometer (Rheomix 600, Thermofisher, Karlsruhe, Germany). For high shear mixing, the TPE material was heated to 170 ◦ C, and the CIP filler was slowly added. Finally, the feedstocks were compounded for 30 min at 30 RPM. The naming convention for the samples consisted of the last three letters of the polymer grade (either STT or STL), followed by the volume percent of the filler and the type of used CIP (either CC, HS, or EW-I). Therefore, a sample made with the TF1 STL copolymer, loaded with 50 vol.-% of HS CIP, will be named STL50HS. After mixing, the density of the feedstock was analyzed by He-pycnometer (AccuPyc 1340, Micrometrics, Norcross, GA, USA). Cylindrical samples with a thickness of 5 mm were warm-pressed above the melting temperature of the used TPE materials at 220 ◦ C.A5kNload was applied for 8 min using a cylindrical die with a diameter of 20 mm. To introduce anisotropic orientation of the CIP particles, warm-pressed samples were placed in a magnetic field of 0.8 T generated by two permanent magnets (Q-51-51-25-N, Webcraft GmbH, Uster, Switzerland) before cooling from 220 ◦ C to room temperature actively under flowing water. Figure 1shows the sample processing schematically. (a) (b) (c) Figure 1. ( a ) Warm-pressing of the MRE samples, ( b ) pre-structuring the MRE sample under magnetic field H, and ( c ) CPI alignment in the TPE-based MR elastomer. The magnetic properties of the MRE with various filler content and different surface treatments were measured using a vibrating sample magnetometer (VSM) from Quantum Design, USA. A 10 −2 g sample was used for the VSM measurements. A magnetic field from 0 T to 2.2 T at ~0.1 T/min was applied. Then the magnetic field was ramped down to −2.2 T at the same rate and finally back up to 2.2 T. An R 2000 rheometer (TA Instruments, New Castle, UK) was used to measure the complex viscosity above the melting temperature (160 ◦ C) with a constant oscillating frequency of 10 Hz and 1% strain amplitude. MR elastomers with a 2 mm thickness were fixed between two parallel plates of 25 mm in diameter. For comparison, the complex viscosity at 170 ◦ C will be reported in the result part for the two different SEBS elastomers filled with 30 vol.-% of carbonyl iron particles, grade HS. For static and dynamic testing, MRE samples were assembled into a double shear testing structure made of V155 grade steel for the dynamic testing. To achieve good adhesion, the surface of the V155 grade steel was sanded before applying an adhesive paste (Araldite 2011 two-component epoxy). 241 Polymers 2021,13, 1597 An Eplexor 500 machine (NETZSCH-Gerätebau GmbH, Selb, Germany) was used for the DMA testing in shear mode. The prepared sample was fixed in the DMA as shown in Figure 2a,b, respectively. (a) (b) Figure 2. (a) Sample preparation in the dynamic mechanical analyzer. (b) A sketch of the sample with applied magnets at the edges of the sample. To induce a magnetic field, disc-shaped permanent magnets with a diameter of 20 mm and thickness of 2 mm (N45, Webcraft GmbH, Uster, Switzerland) were fixed at the faces of the assembled double shear structure. Pairs of 0, 2, 4, and 6 magnets in total were used to induce a magnetic field inside the magnetorheological elastomer composite samples. The intensity and homogeneity of the induced magnetic fields were calculated using finite element analysis (FEMM). In Figure 3, the calculation for one pair of magnets and the results for a higher number of paired permanent magnets is shown. The magnetic field strength calculated for 2 and 3 pairs of magnets is shown in Appendix A. (a) (b) Figure 3. ( a ) Calculated magnetic field intensity using 1 pair of permanent magnets and ( b ) the variability of the field across the MRE. As already mentioned, in this study, we investigated the static and dynamic magnetorheological behavior of the SEBS-based composites. The composites were cycled with a constant strain amplitude of 1% at different pre-strained levels (referred to as static tests in this report). Additionally, so-called dynamic tests were performed without any pre-strain 242 Polymers 2021,13, 1597 conditions. Both tests were made with a constant frequency of 10 Hz. The MRE was calculated using: MRef f ect =Gs−G0 G0x100% (1) where G 0 represents the initial shear modulus and G s the shear modulus upon application of an external magnetic field. Static strains were set at 0.00%, 0.20%, 0.67%, 1.09%, 1.50%, 1.93%, 2.35%, and 2.75%, whereas the dynamic strains were set at 0.04%, 0.25%, 0.45%, 0.65%, and 0.85%. The test conditions for both static and dynamic experiments are shown in Figure 4. (a) (b) Figure 4. (a) Static and (b) dynamic tests procedure to investigate the MR effect of SEBS-based composites. 3. Results 3.1. Effect of CIP Concentration To investigate the influence of CIP concentration on the MRE STL-based composites, we investigated volume fractions between 10 and 60% of HS. For the static tests, a strain amplitude of 1% and a pre-strain of 1.58% were selected. For the dynamic measurements, a strain amplitude of 0.66% was chosen. All tests were performed with an applied magnetic field of 0.54 T and a constant frequency of 10 Hz. The results of the static and dynamic tests are illustrated in Figure 5. For the static and dynamic measurements, a linear relationship between the MR effect and the CIP content was observed below 50 vol.-%. As is illustrated in Figure 5, above 50 vol.-%, the magnetorheological effect decreases significantly. This behavior can be explained by the critical particle volume concentration (CPVC), above which linearity approximation, as described by Jolly et al. [ 18 ] and Ginder [ 19 ], is no longer valid. The point at which there is not enough matrix present to fill all the space between the particles is known as the CPVC. To verify the CPVC point, density measurements can be used (Figure 6). Based on the mixing rule, the theoretical calculated density of a composite linearly increases when filler content increases. At the point where the theoretical and measured density shows a discrepancy, the CPVC point can be defined because air voids in the composite will lower the measured density significantly. Density measurements of samples with a concentration above 50 vol.-% confirmed this prediction. The theoretical density of the composite with 60 vol.-% CPI was higher than the actual, measured values. Based on the discrepancy, it can be assumed that for 60 vol.-% CIP, air voids between the CIP particles are present (Figure 6). This behavior is in good agreement with the theory of critical particle volume concentration (CPVC). Filling a polymeric material above the CPVC results in a porous structure which causes a significant change in composite properties. Static and dynamic MR effects show similar results [ 20 , 21 ]. In Figure 6, it can be seen that the torque for the 60 vol.-% CIP is almost three times higher in comparison to 50 vol.-%. This implies that the processing behavior 243 Polymers 2021,13, 1597 (e.g., viscosity) will significantly increase, and significantly higher machine power during thermoplastic shaping is required. (a) (b) Figure 5. The influence of the CIP content on the MR effect ( a ) for the static test, a strain amplitude of 1% and pre-strain of 1.53% was used; ( b ) for the dynamic test, a strain amplitude of 0.66% was investigated. STL matrix with different concentrations of HS-type CIP; a constant frequency of 10 Hz and field strength of 0.54 T was selected for all tests. Figure 6. The torque at the end of the mixing process and density of the SEBS-based composite in relation to the volume concentration of CIP. The red line shows the calculated density based on the mixing rule. The blue line presents the measured density, and the stars are torque measured in Nm. 3.2. Flow Behavior at Meting Temperature of TPE Composites The equation of motion of spherical CIP in a liquid is inversely proportional to the viscosity of the liquid [ 22 ]. For comparison reasons, complex viscosity on two different SEBS composites with 30 vol.-% CIP was investigated with a constant oscillating frequency of 10 Hz and 1% strain amplitude. Table 3shows the complex viscosity values above the melting point of 160 ◦C. Table 3. A comparison of the complex viscosity and torque at the end of the mixing process for two different SEBS composites with 30 vol.-% CIP. Complex Viscosity at 170 ◦C (Pas) Torque at the End of Mixing 170◦(Nm) STT30HS 184 0.1 ±0.1 STL30HS 30,000 1.6 ±0.2 244 Polymers 2021,13, 1597 The MRE based on STT has a significantly lower complex viscosity in comparison to the one based on STL. The torque after the mixing for both matrix materials confirmed these results. Therefore, it can be expected that the orientation of the carbonyl iron particles in a magnetic field will be less restrained. 3.3. Magnetization Saturation of the CIPs High saturation magnetization is a crucial factor in maximizing the MR effect, as discussed by Jolly et al. [ 18 ]. In Figure 7a, the saturation magnetization in relation to filler content is shown. The results for STL type SEBS composites with three different types of CIP filler (30 vol.-% filler content) are shown in Figure 7b. (a) (b) Figure 7. The magnetization loop for ( a ) STL type styrene-ethylene-butadiene-styrene thermoplastic elastomers with HS filler content between 10 and 60 vol.-%, ( b ) 30 vol.-% filler content of three different CIP grades in STL type styrene-ethylene- butadiene-styrene thermoplastic elastomers. The saturation magnetization for the composites with different filler content is shown in Figure 8. It can be observed that by extrapolation of the composite values, 2 T of pure carbonyl iron can be achieved. This value is very close to the theoretical value of iron (2.2 T). Figure 8. The saturation magnetization in relation to the CIP content and extrapolation of the results to 100% carbonyl iron. In Figure 7b, a saturation magnetization of 0.58 T, 0.65 T, and 0.62 T for STL30HS, STL30CC, and STL30EW-I were observed, respectively. Therefore, the highest saturation magnetization was obtained for the composites based on the carbonyl iron particles with a SiO2coating. This is the CC grade type of CIP used in this study. 245 Polymers 2021,13, 1597 3.4. Effect of the Matrix Material To evaluate the influence of the Shore hardness of styrene-ethylene-butadiene-styrene (SEBS) elastomers on the magnetorheological effect, composites with a constant filler content (30 vol.-%) carbonyl iron particles were selected. Based on the results of the magnetic saturation, the carbonyl iron particles with surface-treatment of SiO 2 (CC grade) were used for this investigation. According to G. Bossis et al. [ 22 ] and Hass [ 23 ], the time required to induce alignment of particles along an external field is proportional to the viscosity of the melted matrix. Moreover, alignment of CIP in the MRE has been proven to positively influence the MR effect [ 22 , 23 ]. As shown in Table 3, the viscosity of 30 vol.-% STL is much higher than 30 vol.-% STT, meaning that full alignment of particles is faster in the former composite. This is expressed in the MR effect: MRE sample based on STT showed a larger static MR effect than the one obtained from STL type, as shown in Figure 9a. However, for dynamic strain, the influence of shore hardness of the thermoplastic elastomer matrix becomes more important. Therefore, the softer STL-based composite shows a higher MR effect, as illustrated in Figure 9b. (a) (b) Figure 9. The influence of matrix material during static ( a ) and dynamic ( b ) strain on the MR effect. Composites STL30CC and STT30CC were used for this analysis. A constant strain amplitude of 1%was used for the static tests. A constant frequency of 10 Hz and field strength of 0.54 T were used for both test conditions. These results indicate that for thermoplastic elastomers, a low melt flow behavior increases the static MR effect thanks to the improved orientation of CIP within the matrix. On the other hand, low matrix Shore hardness resulted in a high MR dynamic effect due to the low initial stiffness of the composite. Based on these results, further experiments were continued with softer STL type styrene-butadiene thermoplastic elastomer. 3.5. Effect of the CIP Surface Treatment Małecki et al. [ 17 ] studied the magnetorheological effect of isotropic MRE based on an SEBS matrix. Coating the CIP with SiO 2 proved to negatively impact the MR effect due to the increased affinity between particles and matrix. This reduced the mobility of CIP within the composite, which in turn hindered the buildup of CIP chains in a magnetic field which results in a lower stiffening effect of the composite. In our experiments on anisotropic SEBS-based MRE, we observed that a SiO 2 -coating on CIP particles increased the MR effect significantly under static and dynamic strain. It can be assumed that for anisotropic MRE, mobility is less important because CIP particles 246 Polymers 2021,13, 1597 are already aligned in the matrix. Figure 10a,b illustrate the effect of the surface treatment on the MR effect. (a) (b) Figure 10. The effect of static ( a ) and dynamic ( b ) strain on MR effect. For this investigation, STL30HS, STL30CC, and STL30EW-I composites were used. All samples contain 30 vol.-% CIP in an STL matrix. A constant strain amplitude of 1%was used for the static tests. A constant frequency of 10 Hz and field strength of 0.54 T were used for both test conditions. The higher static MR effect could be obtained for the STL30CC (SiO 2 -surface) MRE composite. This can be explained by the higher saturation magnetization of these particles (Figure 4). As already mentioned, the effect of the filler-matrix affinity on the MR effect is important and has been already reported in literature [ 17 ]. Both composites STL30HS and STL30CC show low G 0 values of 0.6 MPa, while G 0 of STL30EW-I composite increased to 0.8 MPa. The higher shear modulus can be explained by the higher filler-matrix interfacial adhesion. These results are in agreement with the studies of Wang et al. [ 24 ]. Young’s modulus of composites is affected by the polymer-filler interfacial adhesion. When the polymer–filler interfacial adhesion is weak, the composites exhibit lower modulus. Therefore, it can be assumed that the phosphate coating on top of the silica layer results in a higher interfacial adhesion. This results in a higher G 0 . Based on Figure 10, we can conclude surface treatment of CIP of static and dynamic strain result in slightly different results, especially for the combined SiO2and phosphate coating. 4. Conclusions Styrene-ethylene-butylene-styrene, also known as SEBS, is an important thermoplastic soft elastomer (TPE) that behaves like rubber without undergoing vulcanization. In this study, we reported the magnetorheological effect (MR effect) of two different SEBS-based composites filled with different kinds of carbonyl iron fillers. Carbonyl iron powder without surface treatment and with SiO 2 and SiO 2 -phosphate surface treatment were used as magneto active fillers. The main application of MR elastomers is found in the damping structures. The structure will generate shear stress on the MR elastomers, which results in a static shear strain. Based on this background, we investigated the MR effect under static and dynamic strain. The source of SEBS and carbonyl iron filler affected the static and dynamic magnetorheological performance of the magnetorheological elastomer composite (MREC) significantly. Increasing the carbonyl iron content from 10 to 60 vol.-%, we could observe a significant drop in the MR effect above 50 vol.-% of filler. This behavior is in good agreement with the theory of critical particle volume concentration (CPVC). Filling a polymeric ma- 247 Polymers 2021,13, 1597 terial above the CPVC results in a porous structure which causes a significant change in composite properties. Static and dynamic MR effects show similar results. For anisotropic MR elastomers, changing the SEBS matrix affected the static MR effect differently. Higher static MR properties can be achieved with low complex viscosity above the melting point, and the shore hardness of the SEBS plays a minor role. With a lower complex viscosity, the resistance to generate alignment of the carbonyl iron particles is lower; thus, the chain formation under a magnetic field is faster. For the dynamic MR effect, the shore hardness of the SEBS is an important parameter, as expected. The surface treatment of the CIPs resulted in a higher magnetization saturation of MRECs with a filler content of 30 vol.-%. The highest MR effect could be obtained with a surface treatment of SiO 2 coating. We assume that higher static and dynamic MR effect with SiO 2 coating is caused by two phenomena: (1) SiO 2 coated CPI show higher saturation magnetization, which will result in higher MR-effect. (2) Higher matrix-filler affinity reduced the initial shear modulus, positively affecting the MR-effect. Author Contributions: The idea for this work came from F.C.; The work mixing and characterization were done by F.E.; the manuscript was written by A.T. and F.C.; presentation of the results and the draft structure were discussed by all authors. All authors have read and agreed to the published version of the manuscript. Funding: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 860108 (SMART PROJECT). Institutional Review Board Statement: The study was conducted according to the guidelines of the Declaration of Helsinki and comply with the research integrity guidelines of EMPA (https: //www.empa.ch/web/empa/integrity, accessed on 10 May 2021). Data Availability Statement: The data presented in this study are available on request from the corresponding author. Conflicts of Interest: The authors declare no conflict of interest. Appendix A Figure A1. Calculated magnetic field intensity using ( a ) 2 pairs of permanent magnets and ( b ) 3 pairs of permanent magnets. References 1. Molchanov, V.S.; Stepanov, G.V.; Vasiliev, V.G.; Kramarenko, E.Y.; Khohlov, A.R.; Xu, Z.D.; Guo, Y.Q. Viscoelastic properties of magnetorheological elastomers for damping applications. Macromol. Mater. Eng. 2014,299, 1116–1125. [CrossRef] 2. Lee, C.W.; Kim, I.H.; Jung, H.J. Fabrication and Characterization of Natural Rubber-Based Magnetorheological Elastomers at Large Strain for Base Isolators. Shock Vib. 2018,2018.[CrossRef] 3. Yuan, L.; Sun, S.; Pan, Z.; Ding, D.; Gienke, O.; Li, W. Mode coupling chatter suppression for robotic machining using semi-active magnetorheological elastomers absorber. Mech. Syst. Signal Process. 2019,117, 221–237. [CrossRef] 4. Christie, M.D.; Sun, S.S.; Ning, D.H.; Du, H.; Zhang, S.W.; Li, W.H. A torsional MRE joint for a C-shaped robotic leg. Smart Mater. Struct. 2017,26.[CrossRef] 248 Polymers 2021,13, 1597 5. Tian, T.F.; Li, W.H.; Alici, G. Study of magnetorheology and sensing capabilities of MR elastomers. J. Phys. Conf. Ser. 2013 , 412, 012037. [CrossRef] 6. Li, W.; Kostidis, K.; Zhang, X.; Zhou, Y. Development of a Force Sensor Working with MR Elastomers. IEEE/ASME Int. Conf. Adv. Intell. Mechatron 2009, 233–238. [CrossRef] 7. Phan, K.L.; Mauritz, A.; Homburgb, F.G.A. A novel elastomer-based magnetoresistive accelerometer. Sens. Actuators A Phys. 2008,145.[CrossRef] 8. Lokander, M. Performance of Magnetorheological Rubber Materials. Ph.D. Thesis, KTH, Fiber and Polymer Technology, Stockholm, Sweden, 2004; p. 54. 9. Davis, L.C. Model of magnetorheological elastomers. J. Appl. Phys. 1999,85, 3348–3351. [CrossRef] 10. Mattias Lokander, B.S.; Reitbergerb, T. Oxidation of natural rubber-based magnetorheological elastomers. Polym. Degrad. Stab. 2004,86, 467–471. [CrossRef] 11. Lu, X.; Qiao, X.; Watanabe, H.; Gong, X.; Yang, T.; Li, W.; Sun, K.; Li, M.; Yang, K.; Xie, H.; et al. Mechanical and structural investigation of isotropic and anisotropic thermoplastic magnetorheological elastomer composites based on poly(styrene-b- ethylene-co-butylene-b-styrene) (SEBS). Rheol. Acta 2012,51, 37–50. [CrossRef] 12. Yu, M.; Qi, S.; Fu, J.; Zhu, M.; Chen, D. Understanding the reinforcing behaviors of polyaniline-modified carbonyl iron particles in magnetorheological elastomer based on polyurethane/epoxy resin IPNs matrix. Compos. Sci. Technol. 2017 ,139, 36–46. [CrossRef] 13. Zajac, P.; Kaleta, J.; Lewandowski, D.; Gasperowicz, A. Isotropic magnetorheological elastomers with thermoplastic matrices: Structure, damping properties and testing. Smart Mater. Struct. 2010.[CrossRef] 14. Burgaz, E.; Goksuzoglu, M. Effects of magnetic particles and carbon black on structure and properties of magnetorheological elastomers. Polym. Test. 2020,81, 106233. [CrossRef] 15. Jin, Q.; Xu, Y.G.; Di, Y.; Fan, H. Influence of the particle size on the rheology of magnetorheological elastomer. Mater. Sci. Forum 2015,809, 757–763. [CrossRef] 16. Li, W.; Chen, J.; Gong, X.; Yang, T.; Sun, K.; Chen, X.; Li, W. Microstructure and magnetorheological properties of the thermoplastic magnetorheological elastomer composites containing modified carbonyl iron particles and poly(styrene-b-ethylene- ethylenepropylene-b-styrene) matrix. Smart Mater. Struct. 2012.[CrossRef] 17. Małecki, P.; Królewicz, M.; Krzak, J.; Kaleta, J.; Pigłowski, J. Dynamic mechanical analysis of magnetorheological composites containing silica-coated carbonyl iron powder. J. Intell. Mater. Syst. Stuct. 2015.[CrossRef] 18. Jolly, M.R.; Carlson, J.D.; Muñoz, B.C. A model of the behaviour of magnetorheological materials. Smart Mater. Struct. 1996 ,5, 607–614. [CrossRef] 19. Ginder, J.M. Rheology Controlled by Magnetic Fields. Digit. Encycl. Appl. Phys. 2003.[CrossRef] 20. Howard, K.W.; Hodgson, K.T. Influence of pigment packing behavior on the adhesive requirements of aqueous paper coatings. J. Coat. Technol. Res. 2015,12, 237–245. [CrossRef] 21. Khorassani, M.; Pourmahdian, S.; Afshar-Taromi, F.; Nourhani, A. Estimation of critical pigment volume concentration in latex paint systems using gas permeation. Iran. Polym. J. 2005,14, 1000–1007. 22. Bossis, G.; Lançon, P.; Meunier, A.; Iskakova, L.; Kostenko, V.; Zubarev, A. Kinetics of internal structures growth in magnetic suspensions. Phys. A. 2013,392.[CrossRef] 23. Hass, K.C. Computer simulations of nonequilibrium structure formation in electrorheological fluids. Phys. Rev. E 1993 ,47, 3362–3373. [CrossRef][PubMed] 24. Wang, K.; Wu, J.; Ye, L.; Zeng, H. Mechanical properties and toughening mechanisms of polypropylene/barium sulfate composites. Compos. Part A Appl. Sci. Manuf. 2003,34, 1199–1205. [CrossRef] 249 Sustainability 2021,13, 4010 3. Improve gross and fine motor skills (N3). 4. Promote shared actions linked to feelings of enjoyment, interest and common goals (N4). 5. Reduce levels of frustration in the game (N5). 6. Increase emotional and social reciprocity (N6). 7. Facilitate learning about changes in game turns (N7). Subsequently, these needs, as well as the way of working with them, were evaluated in detail; they were subdivided and defined by analyzing early intervention therapies, specifically, applied behavior analysis [ 43 ], Denver [ 44 ], pivotal response treatment [ 45 ], sensory integration [ 46 ], physical therapy [ 47 ], auditory integration [ 48 ], music therapy [ 49 ] and use of high assistive technology [ 50 ]; other techniques to improve attention with imitation tasks and environmental resources were also analyzed and considered; their efficiency is contrasted with the neural activation of the subject [ 51 – 55 ]. Due to the heterogeneous group of signs and symptoms linked to sub-needs, it was necessary to direct the design of assistive technology toward multifunctionality; this implies including in the same product the satisfaction of various needs, making it possible for it to adapt to the disorder in a flexible way and therefore being adequate for a greater number of users (different ages and degrees of autism [ 1 ]). This exhaustive research study on psychological, physiological and neuronal factors resulted in conclusions on the most suitable design variables (motor, visual, auditory, tactile, olfactory, imitation, interaction, context, cognitive and temporal) contrasted by their relationship with (1) neuronal activity, (2) the subject’s adequate and comfortable reaction to environmental stimuli and (3) the use of imitation tasks to improve the child’s attention. In addition, the most appropriate functions and design variables were defined (see synthesis in Table 1), since the relationships between the need and the design variables were evaluated as “excellent, medium, low or null”, and only those relationships rated as “excellent” were taken into account in the conceptual design. It must be taken into account that, unlike typically developing children, a user with ASD may present deficiencies in social interaction, communication, behavior or executive function. This implies that the execution of tasks while using the product implies an added difficulty when having to pay attention to the object, to other people or to the actions that they carry out. To improve usability, comfort, reduce frustration and minimize errors, the toy should have motivational and behavioral elements or facilitators as secondary functional requirements. They will make the product adapt to the user in three dimensions: (1) Physical accessibility: it includes all anthropometric adaptations (static or dimensional and dynamic or mobility). (2) Cognitive accessibility: functions and elements of the product are adapted to the processing, understanding, semiotics, learning and decision-making of a user with ASD. (3) Sensory accessibility: it assures that the perception of the product is comfortable, intuitive and adapted to the perceptual peculiarities of a user with ASD. Table 2details and classifies the set of functional requirements necessary to improve the accessibility of the product; they are derived from the general requirements defined in Table 1(numbered from 1 to 10). These allowed to carry out the conceptual design in the project were defined. 257 [Document text truncated for crawler view.]