Development of an intelligent computer vision system for identification, characterization and analysis of yarn quality

July 2024 Universidade do Minho Escola de Engenharia Filipe Alexandre de Sousa Pereira Development of an intelligent computer vision system for identification, characterization and analysis of yarn quality Development of an intelligent computer vision system for identification, characterization and analysis of yarn quality Filipe Alexandre de Sousa Pereira UMinho | 2024 Filipe Alexandre de Sousa Pereira Development of an intelligent computer vision system for identification, characterization and analysis of yarn quality PhD Thesis Doctoral Program in Electronics and Computers Engineering specializing in Control, Automation and Robotics Work conducted under the guidance of Professor Doctor Filomena Maria da Rocha Menezes de Oliveira Soares Professor Doctor Vítor Hugo Mendes da Costa Carvalho Professor Doctor Rosa Maria Castro Fernandes Vasconcelos July 2024 i DIREITOS DE AUTOR E CONDIÇÕES DE UTILIZAÇÃO DO TRABALHO POR TERCEIROS Este é um trabalho académico que pode ser utilizado por terceiros desde que respeitadas as regras e boas práticas internacionalmente aceites, no que concerne aos direitos de autor e direitos conexos. Assim, o presente trabalho pode ser utilizado nos termos previstos na licença abaixo indicada. Caso o utilizador necessite de permissão para poder fazer um uso do trabalho em condições não previstas no licenciamento indicado, deverá contactar o autor, através do RepositóriUM da Universidade do Minho. Licença concedida aos utilizadores deste trabalho Atribuição CC BY https://creativecommons.org/licenses/by/4.0/ ii ACKNOWLEDGEMENTS I am using this opportunity to express my gratitude to everyone who supported me throughout the development and writing of the present work. I am thankful for their aspiring guidance, invaluably constructive criticism, and friendly advice during the project work. In particular, I would like to give my thanks to: o My supervisor Professor Doctor Filomena Soares, for her unconditional support, guidance, and freedom to pursue my ideas. o My advisor Professor Doctor Vitor Carvalho, for his help, advice, supervision, and excellent feedback. He constantly allowed this thesis to be my own work but guided me in the right direction whenever necessary. All the advice and guidance he gave me was invaluable and I sincerely thank him. o My co-supervisor Professor Doctor Rosa Vasconcelos, for her help, advice, supervision, and excellent feedback. All the advice she gave me was very important. o My former student and colleague Leandro Montenegro for his valuable knowledge in machine learning. I am eternally grateful for his valuable observations regarding the work carried out. o Professor Doctor José Mendes Machado for his help, advice, supervision, and excellent feedback. o Professor Doctor Manuel Carlos Felgueiras for his vision and enthusiasm, and for sharing his opinion on aspects related to the topic. o My parents for providing unconditional support and continued encouragement throughout my years. o My wife and children for their unconditional support in the study and also in the research and writing process of the Thesis. This achievement would not have been possible without them. FUNDING This work has been supported by FCT – Fundação para a Ciência e Tecnologia in the scope of the project: UIDB/04077/2020 and UIDB/00319/2020. iii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the University of Minho. University of Minho, 30th July 2024 Full name: Filipe Alexandre de Sousa Pereira Signature: ____________________________________________________________________ iv RESUMO Portugal é um país com uma forte tradição da indústria têxtil e, mesmo no contexto de mudanças no mercado, as empresas portuguesas continuam a perseguir o nível de excelência que lhes permite sobreviver num mundo mais complexo e exigente. Na indústria têxtil, a qualidade do produto final está diretamente relacionada com a qualidade do fio e, portanto, é essencial fazer uma avaliação precisa das características de acordo com certos parâmetros predefinidos. Há uma evolução dos dispositivos que avaliam a qualidade do fio, no entanto, ainda têm limitações, como alto custo, dimensão e peso, assim como resolução e precisão limitadas na determinação de certos parâmetros do fio. O objetivo principal desta tese é desenvolver algoritmos de deep learning para identificar e caracterizar a pilosidade do fio, além de criar algoritmos para caracterizar e analisar outros parâmetros de qualidade do fio usando visão computacional. A estratégia foi, em primeiro lugar, projetar um protótipo mecatrónico que permitisse a captura direta de imagens ou vídeos de alta qualidade do enrolamento do fio, e também uma análise e classificação das pilosidades do fio. O protótipo permite obter outras características inerentes à análise da qualidade do fio, como: massa linear, diâmetro, volume, direção da torção, passo da torção, desvio médio de massa, coeficiente de variação, coeficiente de pilosidade, desvio médio de pilosidade e desvio padrão. Esta tese de doutoramento introduz, como uma das principais contribuições, uma nova abordagem de deep learning utilizando um algoritmo otimizado baseado no YOLOv5s6 (You only look once) para caracterizar diferentes tipos de pilosidade do fio. Os resultados mostram que o algoritmo proposto melhora significativamente o desempenho do modelo, com um aumento de 5-6% na métrica mAP0.5 ( mean average precision at 0.5 intersection over union (IoU)) e um aumento de 11-12% na métrica mAP0.5:0.95 em comparação com o algoritmo YOLOv5s6 padrão. A abordagem melhora efetivamente todas as métricas analisadas para a caracterização da pilosidade do fio. A implementação bem-sucedida deste trabalho pode aumentar a eficiência produtiva da indústria têxtil e contribuir para a criação de produtos de alto valor acrescentado. PALAVRAS-CHAVE: Deep Learning , Defeitos do Fio, Pilosidades, Processamento de Imagem, Qualidade do Fio. v ABSTRACT Portugal has a strong tradition in the textile industry, even under the context of market and demands changes, the Portuguese companies continue pursuing the excellence level that makes them survive in a more complex and challenging world. In the textile industry, the quality of the final product is directly related to the quality of the yarn and therefore it is essential to make an accurate assessment of the yarn characteristics, according to certain pre-established parameters. There is a significant evolution of the devices that evaluate the quality of the yarn, however, these devices still have several limitations such as high cost, large dimension, and weight, as well as limited resolution and precision in determining certain parameters of the yarn. The main goal of this thesis is to develop deep learning algorithms to identify and characterize yarn hairiness, as well as to create algorithms for characterizing and analyzing other yarn quality parameters using computer vision. The strategy was, first, to design a mechatronic prototype that allows for the direct capture of high-quality images or videos of yarn winding, and also for the analysis and classification of yarn hairiness. It also allows to obtain other characteristics inherent to the analysis of the yarn quality, such as: linear mass, diameter, volume, twist direction, twist step, average mass deviation, coefficient of variation, hairiness coefficient, average hairiness deviation and standard deviation. This thesis introduces, as one of the main achievements, a novel deep learning approach using an optimized algorithm based on YOLOv5s6 (You only look once) to characterize different types of yarn hairiness. The results show that the proposed algorithm significantly improves the model performance, with a 5-6% increase in mAP0.5 (mean average precision at 0.5 intersection over union (IoU)) metric and an 11-12% increase in the mAP0.5:0.95 metric compared to the standard YOLOv5s6 algorithm. The approach effectively enhances all analyzed metrics for yarn hairiness characterization. The successful implementation of this work can increase the productive efficiency of the textile industry and contribute to the development of high added value products. KEYWORDS: Deep Learning, Faults, Hairiness, Image Processing, Yarn Quality. vi LIST OF CONTENTS Acknowledgements .............................................................................................................................. ii Resumo.............................................................................................................................................. iv Abstract............................................................................................................................................... v List of Contents .................................................................................................................................. vi List of Figures ..................................................................................................................................... xi List of Tables ..................................................................................................................................... xix Abbreviations ..................................................................................................................................... xxi 1. Introduction ................................................................................................................................ 1 1.1 Framework .......................................................................................................................... 2 1.2 Objectives ........................................................................................................................... 6 1.3 Development methodology .................................................................................................. 7 1.4 Main contributions of the work ............................................................................................. 8 1.5 Thesis Structure ................................................................................................................ 13 2. Theoretical Concepts ................................................................................................................ 15 2.1 Yarn characterization ......................................................................................................... 16 2.1.1 Linear mass and diameter ......................................................................................... 16 2.1.2 Specific volume ......................................................................................................... 17 2.1.3 Hairiness ................................................................................................................... 17 2.1.4 Yarn twist, twist orientation, and twist step ................................................................. 18 2.1.5 Types of faults in yarn ................................................................................................ 19 2.2 Statistical Parameters of Yarn ............................................................................................ 20 2.2.1 Mean Deviation of Mass ............................................................................................. 20 2.2.2 Coefficient of Variation ............................................................................................... 20 2.2.3 Hairiness Coefficient .................................................................................................. 20 2.2.4 Hairiness Mean Deviation .......................................................................................... 21 2.2.5 Hairiness Standard Deviation ..................................................................................... 21 2.3 Spectral Analysis ............................................................................................................... 21 2.4 Image Processing & Computer Vision ................................................................................ 22 2.4.1 Unitary element and color scales ............................................................................... 22 xiii Figure 59 –Confusion Matrix for Binary Classification [63]. ................................................................ 58 Figure 60 – An example of the precision-recall curve [64]. ................................................................. 59 Figure 61 – An example of the precision-recall curve [65]. ................................................................. 60 Figure 62 – An example of the precision-recall curve [69]. ................................................................. 62 Figure 63 – Sub-variants of YOLOv5 [56]. ......................................................................................... 64 Figure 64 – Network structure for YOLOv5s6 - (A): The complete structure with backbone, neck, and head modules. (B, C): Two unique variations of CSP blocks (C3); (D): CBS - Convolutional Batch Normalization layer and SILU (sigmoid linear units) activation function; (E): Other blocks with CBS; (F) - Two distinct bottleneck blocks] [76]. .................................................................................................................... 66 Figure 65 – C3 Module from YOLOv5s6 [76]. ................................................................................... 67 Figure 66 – SPPF Module from YOLOv5s6 [76]. ................................................................................ 67 Figure 67 – Schematic representation of the activation function [79]. ................................................ 68 Figure 68 - SiLU activation function [79] . .......................................................................................... 69 Figure 69 - Activation function GeLU [80]. ......................................................................................... 71 Figure 70 - Comparison of activation functions [81]. .......................................................................... 72 Figure 71 – YOLOv5s6 Image Augmentation [82]. ............................................................................. 73 Figure 72 - Flowchart of the method for detecting yarn parameters and evaluating yarn quality [83]. .. 77 Figure 73 - System device to detect yarn parameters [83]. ................................................................ 77 Figure 74 - a) New foreground image, b) yarn core, c) hairiness [83]. ................................................ 78 Figure 75 - An example of a yarn image with a nep defect obtained using the image acquisition system [87]. ................................................................................................................................................. 79 Figure 76 - An example of a yarn image with a nep defect obtained using the image acquisition system [87]. ................................................................................................................................................. 80 Figure 77 - Confusion matrices that offer insights into the performance of the model system [87]. ..... 81 Figure 78 - Cotton yarns: the architecture of neural network traction properties [89]. ......................... 83 Figure 79 - Blended yarns: the architecture of neural network traction properties [89]. ....................... 83 Figure 80 - Blended yarns: the structure of neural network traction properties [91]. ........................... 84 Figure 81 - The structural composition of the neural network, including its tenacity and elongation percentages: (a) Ring yarn system (left), (b) compact yarn system (right) [91]. ................................... 84 Figure 82 - The CVm% neural network architecture for ring-spun and compact yarn systems [91] ....... 85 Figure 83 - The neural network architectures for imperfections: a) Ring-spun yarn system, b) Compact yarn system [91]. ............................................................................................................................. 85 xiv Figure 84 – a) A yarn in grayscale (left); b) Image after linearization using the OTSU algorithm (right) [97]. ........................................................................................................................................................ 92 Figure 85 - a) Gray scale image (left); b) Image with Gaussian filter (right) [97]. ................................. 93 Figure 86 - a) Image prior to the application of the opening morphological operation (left); b) Image subsequent to the implementation of the opening morphological operation (right) [97]. ..................... 93 Figure 87 - a) Image after the closing process (left); b) Isolated core image (right) [97]. ..................... 94 Figure 88 - Isolated outlines of the yarn. ............................................................................................ 96 Figure 89 - Contours of the isolated core. .......................................................................................... 96 Figure 90 - Contours of the isolated hairiness. ................................................................................... 97 Figure 91 - Separation of particles in single cable or yarn. ................................................................. 98 Figure 92 - Separation of particles in Separation of particles in 2 cables or folded yarns. .................... 98 Figure 93 - Image after the opening process. ..................................................................................... 98 Figure 94 - Image of isolated particles. .............................................................................................. 99 Figure 95 – StViewer software executable [92]. ............................................................................... 100 Figure 96 –Available parameter categories in StViewer [92]. ............................................................ 100 Figure 97 – OMRON STC-MBS163U3V Camera [92]. ...................................................................... 101 Figure 98 – Lens set already equipped with a yarn holder [92]. ....................................................... 101 Figure 99 – User interface [92]. ...................................................................................................... 102 Figure 100 – Algorithm Analysis Methodology Flowchart [92]. ......................................................... 105 Figure 101 - Improved architecture based on YOLOv5s6 – YOLOv5s6-Hairiness [76]. ...................... 109 Figure 102 - The CBG module in the optimized YOLOv5s6-Hairiness [104]. ..................................... 110 Figure 103 - The CBG module employed in various other blocks [104]. ........................................... 110 Figure 104 - The CBG module is used in two different types of bottleneck blocks [104]. ................... 110 Figure 105 - Two distinct types of CSP blocks (C3) [104]. ............................................................... 111 Figure 106 - Difference between C3_x (top) and C2f module (bottom) [104]. ................................... 113 Figure 107 - The architecture of the Bot-Transformer block. (a) BottleneckTransformer*x signifies that a total of x Bottleneck Transformer blocks are stacked, and each Bottleneck Transformer is depicted in (b) [104].............................................................................................................................................. 115 Figure 108 - MHSA Layer used in the Bot-transformer Block [106]. ................................................. 116 Figure 109 – V-Model for the Design of a Mechatronic System - VDI 2206 [110]. ............................. 125 Figure 110 – a) Over end withdrawal; b) Side withdrawal [97]. ........................................................ 131 xv Figure 111 – a) Yarn ballooning representation; b) Yarn guide position above conical package apex representation [97]. ........................................................................................................................ 131 Figure 112 – a) Parallel package; b) Cross wound package [97]. ..................................................... 132 Figure 113 – Random winding: a) grooved drum; b) yarn guide [97]................................................ 133 Figure 114 – Illustration of precision winding [97]. .......................................................................... 133 Figure 115 – Step-precision winding and depiction of the fluctuation in coil angle across phases [97]. ...................................................................................................................................................... 134 Figure 116 – Illustration of a flexible winding system [97]. ............................................................... 135 Figure 117 – Different types of ceramic yarn guides [97]. ............................................................... 136 Figure 118 – Yarn guide roller [97]. ................................................................................................ 136 Figure 119 – Methods of applying tension to a yarn [97]. ................................................................ 137 Figure 120 – Yarn tensioners: a) additive; b) multiplicative; c) multiplicative self-adjusting (lever tension); d) multiplicative self-adjusting (ball tension) [97]. ............................................................................ 138 Figure 121 – Automatic tension control method employing a yarn positive over feed system [97]. .... 139 Figure 122 – Frontal schematic view of the machine: 1) Box; 2) Bobbin support; 4) Guide support; 6) Bobbin; 9) First positive yarn feeder (PYF); 11) Second PYF; 12) PYF support; 13) Toothed belt; 14) Toothed belt; 19) Camera; 20) Servomotor support; 21) Ring light; 22) Guide support on lens; 24) Lens; 26) Camera support; 27) Sliding base servomotor; 28) Sliding base PYF; 31) Support block; 33) Grooved drum; 34) Flat belt; 38) Grooved drum support; 40) Flat pulley; 43) Structural beam; 47) Lens support; 49) Yarn guide; 50) Guidance subsystem cover sensor [97]. ........................................................... 141 Figure 123 – Path of the yarn during winding [97]. ......................................................................... 142 Figure 124 – Cinematic chain: 1) Toothed belt and pulley set between servomotor and second PYF; 2) Toothed belt and pulley set between first and second PYF; 3) Flat belt and pulley set between servomotor and winding subsystem; A Sliding support of first PYF; B Sliding support of servomotor [97]. ........... 143 Figure 125 – Automation system representation: 1) 2 pole circuit breaker; 2) 1 pole circuit breaker (× 2); 3) Power socket to yarn illumination; 4) Power supply (24 V DC); 5) Servomotor driver; 6) Servomotor; 7) Positive yarn feeder 1; 8) yarn unwinding sensor relay (24 V DC); 9) Yarn guidance sensor relay; 10) Positive yarn feeder 2; 11) Yarn winding sensor relay; 12) Guidance subsystem cover sensor; 13) Emergency button; 14) PLC; 15) Local HMI (Human-Machine Interface); 16) Local Computer; 17) NI OPC Server; 18) LabVIEW [97]. .............................................................................................................. 144 Figure 126 – a) Local HMI location (left); b) Local HMI screen for yarn quality test defined by time and velocity (right) [97]. ........................................................................................................................ 145 xvi Figure 127 – Yarn quality test saved data [97]. ............................................................................... 146 Figure 128 – Kinematic chains ((A) and (B) are Servomotors [92]. .................................................. 146 Figure 129 – Operating Modes Coordination Flowchart. .................................................................. 150 Figure 130 – Test preparation mode flowchart. ............................................................................... 153 Figure 131 – Flowchart of the test mode (part 1). ........................................................................... 155 Figure 132 – Flowchart of the test mode (part 2). ........................................................................... 156 Figure 133 – Flowchart of the test termination mode. ..................................................................... 158 Figure 134 – Flowchart of the stop mode (part 1). .......................................................................... 159 Figure 135 – Flowchart of the stop mode (part 2). .......................................................................... 160 Figure 136 – Flowchart of the manual mode. .................................................................................. 162 Figure 137 – Outline of the process of translating Grafcet into Ladder Diagram. .............................. 164 Figure 138 - Industrial cloud Talk2M with Flexy 205 [120]............................................................... 165 Figure 139 - Prototype’s mechanical system [97]. ........................................................................... 167 Figure 140 - Automation system’s hardware [97]. ........................................................................... 167 Figure 141 - Three types of yarn tested [92]. ................................................................................... 174 Figure 142 - Contours of the yarn that have been isolated (Cotton yarn with an average linear mass of 56.4tex) [92]. ................................................................................................................................. 177 Figure 143 - Image of purple thread with isolated particles. ............................................................. 178 Figure 144 - Mass spectrogram by prototype [92]. .......................................................................... 183 Figure 145 - Mass spectrogram graph generated by the USTER TESTER 3 [92]. .............................. 183 Figure 146 – Hairiness spectrogram by prototype [92]. ................................................................... 184 Figure 147 - Hairiness spectrogram graph generated by the USTER TESTER 3 [92]. ........................ 184 Figure 148 - Annotations in a yarn image were made using LabelME, with loop fibers marked in green and protruding fibers marked in red. The image is of cotton yarn with an average linear mass of 56.4 tex and a magnification factor of 20x. ................................................................................................... 185 Figure 149 - Visualization of the dataset includes: (a) The number of annotations per class; (b) Visualization of the location and size of each bounding box; (c) The statistical distribution of the bounding box positions; (d) The statistical distribution of the bounding box sizes. ............................................ 187 Figure 150 - An example of a yarn image with the presence of loop fibers and a confusing appearance. This image is of cotton yarn with an average linear mass of 56.4 tex, taken with a magnification factor of 20x. ............................................................................................................................................... 188 xvii Figure 151 - Data augmentation of the dataset used in the study (Cotton yarn with an average linear mass of 56.4 tex, with a magnification factor of 20x). ............................................................................... 189 Figure 152 – Number of epochs - mAP_0.5. ................................................................................... 191 Figure 153 – Number of epochs - mAP_0.5:0.95. ........................................................................... 192 Figure 154 – Number of epochs - precision. .................................................................................... 192 Figure 155 – Number of epochs - recall. ......................................................................................... 193 Figure 156 – Various values of learning rate in Yolov5s6 – Hairiness Improved - mAP_0.5. ............. 195 Figure 157 – Various values of learning rate in Yolov5s6 – Hairiness Improved - mAP_0.5:0.95. ..... 195 Figure 158 – Various values of learning rate in Yolov5s6 – Hairiness Improved - mAP_0.5. ............. 196 Figure 159 – Various values of learning rate in Yolov5s6 – Hairiness Improved - mAP_0.5. ............. 196 Figure 160 - Performance metrics for the YOLOv5s6-Hairiness approach. ........................................ 199 Figure 161 - Performance metrics for the proposed YOLOv5s6-Hairiness approach. ......................... 200 Figure 162 - Performance metrics for the proposed YOLOv5s6-Hairiness approach. ......................... 202 Figure 163 - Proposed YOLOv5s6-Hairiness architecture approach. ................................................. 211 Figure 164 - Test results comparing the proposed optimized YOLOv5s6-Hairiness (right image) with the default YOLOv5s6 (left image) at a 0.2 confidence threshold (Cotton yarn with an average linear mass of 56.4 tex, magnified by a factor of 20x). ........................................................................................... 212 Figure 165 - Test results comparing the proposed optimized YOLOv5s6-Hairiness (right image) with the default YOLOv5s6 (left image) at a 0.3 confidence threshold (Cotton yarn with an average linear mass of 56.4 tex, magnified by a factor of 20x). ........................................................................................... 213 Figure 166 - Test results comparing the proposed optimized YOLOv5s6-Hairiness (right image) with the default YOLOv5s6 (left image) at a 0.5 confidence threshold (Cotton yarn with an average linear mass of 56.4t ex, magnified by a factor of 20x). ........................................................................................... 213 Figure A.1-1 - Field of View Visualization [128] ................................................................................ 241 Figure A.2-1 - Higher Hierarchy Grafcet - Coordination of Operating Modes [128] ............................. 243 Figure A.2-2 - Grafcet A – Part 1 ..................................................................................................... 244 Figure A.2-3 - Grafcet A – Part 2 ..................................................................................................... 245 Figure A.2-4 - Grafcet A – Part 3 ..................................................................................................... 245 Figure A.2-5 - Grafcet B .................................................................................................................. 246 Figure A.2-6 - Grafcet A - Test Mode ................................................................................................ 246 Figure A.2-7 - Grafcet B - Test Mode – Part 1 .................................................................................. 247 Figure A.2-8 - Grafcet B - Test Mode – Part 2 .................................................................................. 247 xviii Figure A.2-9 - Grafcet C - Test Mode ................................................................................................ 248 Figure A.2-10 - Grafcet of Test Termination Mode – Part 1 .............................................................. 249 Figure A.2-11 - Grafcet of Test Termination Mode – Part 2 .............................................................. 250 Figure A.2-12 - Grafcet of Test Termination Mode – Part 3 .............................................................. 251 Figure A.2-13 - Grafcet of Stop Mode – Part 1 ................................................................................. 252 Figure A.2-14 - Grafcet of Stop Mode – Part 2 ................................................................................. 253 Figure A.2-15 - Manual Mode Grafcet – Grafcet A – Part 1 ............................................................... 254 Figure A.2-16 - Manual Mode Grafcet – Grafcet A – Part 2 ............................................................... 254 Figure A.2-17 - Manual Mode Grafcet – Grafcet A – Part 3 ............................................................... 255 Figure A.2-18 - Manual Mode Grafcet – Grafcet A – Part 4 ............................................................... 255 Figure A.2-19 - Manual Mode Grafcet – Grafcet C ............................................................................ 255 Figure A.3-1Higher Hierarchy Grafcet - Coordination of Operating Modes ....................................... 256 Figure A.3-2 - Grafcet of Test Mode - Grafcet A – Part 1 ................................................................... 256 Figure A.3-3 - Grafcet of Test Mode - Grafcet A – Part 2 ................................................................... 257 Figure A.3-4 - Grafcet of Test Mode - Grafcet A – Part 3 ................................................................... 258 Figure A.35 - Grafcet of Test Mode - Grafcet A – Part 4 .................................................................. 258 Figure A.36 - Grafcet of Test Mode - Grafcet A – Part 3 .................................................................. 258 Figure A.37 - Grafcet of Test Mode - Grafcet C ............................................................................... 259 Figure A.38 - Grafcet of Stop Mode – Part 1 .................................................................................. 259 Figure A.39 - Grafcet of Stop Mode – Part 2 .................................................................................. 260 Figure A.310 - Grafcet of Manual Mode ......................................................................................... 261 Figure A.41 - Predefined screens for HMI installed on the machine body – Part 1 .......................... 262 Figure A.42 - Predefined screens for HMI installed on the machine body – Part 2 .......................... 263 Figure A.43 - Predefined screens for HMI installed on the machine body – Part 3 .......................... 264 Figure A.44 - Predefined screens for HMI installed on the machine body – Part 4 .......................... 265 Figure A.45 - Predefined screens for HMI installed on the machine body – Part 5 .......................... 265 Figure A.51 - LabVIEW interface - Local computer .......................................................................... 266 xix LIST OF TABLES Table 1 - A Comparison of Characteristics Between Previously Proposed Systems [92] ...................... 86 Table 2 - Camera technical specifications (Adapted from [92]). ........................................................ 101 Table 3 - Project Performance Specifications ................................................................................... 128 Table 4 - Morphological chart - green cells: solutions adopted in the final prototype [97]. ................. 140 Table 5 - A Comparison of Characteristics Between Previously Proposed Systems and the Case Study [92] ................................................................................................................................................ 168 Table 6 - Results obtained from the USTER TESTER 3 equipment [92]............................................. 175 Table 7 - Purple Cotton Yarn Image Analysis Results. ...................................................................... 175 Table 8 - Yellow Cotton Yarn Image Analysis Results. ....................................................................... 176 Table 9 - Image Analysis Results of White Polyester Yarn. ................................................................ 176 Table 10 - Results of the video analysis for 1km of purple cotton. .................................................... 180 Table 11 - Classification of the dataset based on various experimental conditions - without using data augmentation techniques [113]. ..................................................................................................... 186 Table 12 - The development environment. ....................................................................................... 189 Table 13 - Software tools environment. ........................................................................................... 190 Table 14 - Parameters settings of the training configuration. ............................................................ 190 Table 15 - Parameters settings of the training configuration. ............................................................ 193 Table 16 - Parameters settings of the learning rate training configuration in Yolov5s6 – Hairiness Improved algorithm. ....................................................................................................................................... 194 Table 17 - Parameters settings of the training configuration. ............................................................ 198 Table 18 - Performance metrics increase (%) of the optimized YOLOv5s6-Hairiness algorithm with augmentation and the YOLOv5s6 Default model with augmentation. ................................................ 198 Table 19 - Metrics comparation results between YOLOv5s6 Default with data Augmentation and improved YOLOv5s6-Hairiness with C2f and Bot-Transformer Module. ............................................................ 204 Table 20 - Performance comparison of introducing the C2f module in the different locations of the algorithm. ....................................................................................................................................... 205 Table 21 - Metrics comparation results between Activation Functions with data Augmentation. ......... 206 Table 22 - Improved YOLOv5s6 with activation function GeLU. ........................................................ 207 Table 23 - Performance comparison of different head numbers of MHSA. ........................................ 208 Table 24 - Performance comparison of different High Hyperparameters with YOLOv5s6 Default. ...... 209 xx Table 25 - Performance metrics comparison between proposed optimized YOLOv5s6-Hairiness and other models with data Augmentation. ..................................................................................................... 212 Table 26 - k-fold Cross Validation with k = 10 in optimized YOLOv5s6-Hairiness with Augmentation.. 214 xxi ABBREVIATIONS ADAM Adaptive Moment Estimation ANN Artificial Neural Network C2F Coarse to Fine CV Computer Vision DL Deep Learning ELAN Efficient Layer Aggregation Network GELU Gaussian Error Linear Unit HMI Human Machine Interface HSV Hue Saturation Value IP Image Processing MAE Mean Absolute Error PLC Programmable Logic Controller RELU Rectified Linear Unit R-CNN Region Convolutional Neural Network R-FCN Region Fully Convolutional Network RGB Red Green Blue SCADA Supervisory Control and Data Acquisition SGD Stochastic Gradient Descent SILU Sigmoid Linear Units SPPF Spatial Pyramid Pooling Fast TEX Linear Mass of Yarn YOLO You Only Look Once 1. INTRODUCTION Overview This chapter presents the framework and importance of the yarn quality analysis problem under study. The objectives of the work are highlighted, along with the development methodology and the scientific contributions of the work. Finally, the structure of this thesis is presented. 1.1 Framework 1.2 Objectives 1.3 Development methodology 1.4 Main contributions of the work 1.5 Thesis structure Chapter 1 – Introduction ___________________________________________________________________________ 8 To answer the research questions, the method used was quantitative, since readings, measurements and comparisons of results were carried out regarding the quality and performance of the system developed throughout the various research stages [14]. Data collection occurred through the proposed system developed. Thus, the quantitative method was considered the most appropriate because it was intended to determine and evaluate the continuous improvement of the developed proposed system. Based on an innovative result, where there was a transfer of knowledge to technology, grounded on a real context, an action research strategy was adopted [15]. To evaluate the performance of the system developed, quantitative data was used, and comparisons were made with existing systems on the market and reference systems in the textile industry. 1.4 Main contributions of the work The scientific contributions achieved while developing this work are: 1. Development/optimization of yarn characterization methods: • Innovative Methodology: This work introduces an innovative methodology that offers a more precise method of examining/determining defects in yarn, particularly focusing on yarn hairiness. • Deep Learning: Deep learning was integrated into the existing algorithms for the detection and automatic classification of yarn hairiness types. This integration substantially enhanced the accuracy and classification capabilities of the algorithms. • Enhancements in Algorithm Design: The developed algorithm, YOLOv5s6 (You Only Look Once – version 5, model s6) - Hairiness [16], introduced several key enhancements, including the integration of the C2f (coarse to fine) function, a BotTransformer module, the adoption of the GeLU (Gaussian Error Linear Unit) activation function, and fine-tuning of hyperparameters. These improvements aimed to address scale, contextual challenges, capture spatial information, understand complex relationships within images, and refine the training data for better pattern capture. • Performance Improvement: Experimental results showcased an increased performance with the proposed YOLOv5s6-Hairiness algorithm. It outperformed the original YOLOv5, demonstrating a 5-6% improvement in the mAP0.5 (mean Average Chapter 1 – Introduction ___________________________________________________________________________ 9 Precision at 0.5 Intersection over Union (IoU)) metric and an 11-12% increase in mAP0.5:0.95 metric. This performance enhancement establishes it as a more efficient and high-performing neural network for yarn hairiness detection, particularly excelling in detecting complex classes like loop fibers. 2. Yarn image dataset: • Dataset Creation: The dataset creation process was complete and meticulous, consisting of 684 images specifically captured for yarn hairiness analysis. These images were annotated and classified into two categories: protruding fibers and loop fibers. The number of annotations (11037) and images (684) provides a rich resource for training and validating deep learning algorithms for yarn defect classification. • Annotation and Classification Methodology: Utilization of LabelME software [17], [18] in polygonal mode facilitated precise annotation of loop fibers (marked in green) and protruding fibers (marked in red) in the dataset images. This allowed for meticulous labeling, resulting in an average of 16.1 annotations per image. Additionally, the split of the dataset into training (70%), validation (20%), and test (10%) sets ensures a robust evaluation of the model's performance without the use of augmentation techniques. • Visualization and Statistical Analysis of the Dataset: The dataset characterization provides comprehensive insights into the distribution of annotations, bounding box positions, sizes, and statistical information regarding loop and protruding fibers. Visualizations of hairiness offer a clear understanding of the diversity in bounding box positions and sizes, essential for training models to recognize objects with varying characteristics and placements. • Imbalance Identification and Addressing: The dataset analysis highlighted an imbalance between loop fibers and protruding fibers. The complexity of loop fibers, their diverse appearances, sizes, and occlusions contribute to the challenges in detection. This insight can guide future research to focus on techniques that specifically address the complexities associated with detecting loop fibers. • Data Augmentation Techniques: To address the challenges identified, data augmentation techniques were employed using various such as flipping, saturation adjustment, blur, and noise addition were applied to augment the dataset, resulting in an expanded dataset of 1644 images. This augmented dataset provides a diverse range of image variations for robust model training and evaluation. Chapter 1 – Introduction ___________________________________________________________________________ 10 • Dataset as a Platform for Improvements: This dataset, annotated with 11037 detailed markings across 684 images, presents a valuable resource. It not only facilitates the development of more robust machine learning models specifically designed for yarn hairiness detection but also opens paths for advancing the field of deep learning in textile sciences. By providing clear annotations, statistical insights, and diverse augmentation techniques, this dataset stands as a resource for future research in textile engineering fostering advancements in the analysis and understanding of textile defects. 3. Development of algorithms for image acquisition, processing, and analysis: • Innovative Image Capture: The algorithms enable the capture of in-motion images of the yarn production process, a capability not previously available in existing commercial systems. This advancement may provide a dynamic and continuous view of the yarn production process. • Advanced Image Pre-processing: The algorithms incorporate advanced preprocessing techniques to enhance the quality of captured images. This includes distortion correction, noise elimination, and contrast improvement, ensuring precise data collection for subsequent analysis. • Segmentation and Detection: The algorithms can precisely segment specific areas of interest in yarn images, such as the identification and differentiation of loop fibers, protruding fibers, or other characteristics, enabling accurate detection and analysis of these areas. • Spectral Analysis: Performs spectral analysis based on yarn linear mass variation and hairiness data, utilizing the Fast Fourier Transform (FFT) to identify periodical irregularities in yarn. Results are available in the user interface. The technological contributions achieved while developing this work are: 1. Development of a mechatronic device: • Development of an Automatic Unwinding and Winding System: This system introduces an innovative approach by creating an automatic unwinding and winding system aimed at verifying yarn quality through image processing. This non-destructive prototype allows for the yarn winding and unwinding without damaging the yarn/bobbins. It has the ability to accurately move the yarn in front of the camera lens, minimizing Chapter 1 – Introduction ___________________________________________________________________________ 11 vibrations and supporting different types of yarn and bobbins without causing any destructive effects to them. • Enhanced Versatility and Portability: Unlike existing commercial yarn quality testing machines that are large, complex, and expensive, non-destructive prototype offers enhanced versatility and portability (weighs less than 30 kg and fits within a compact box of less than 0.27 m³). Its design facilitates the testing of various types of yarns and bobbins with diameters up to 160mm (obtained in project performance specifications), enabling textile companies to conduct on-site analysis without relying on external laboratories. • Monitoring and Data Storage: The system provides an intuitive local HumanMachine Interface (HMI) that allows users to control yarn quantity and velocity. It enables monitoring and storage of data, ensuring the access to statistical information and assuring user safety and system security. • Image Analysis for More Parameters: Through the application of image analysis techniques, the non-destructive prototype allows the analysis of more parameters compared to existing commercial alternatives. Additionally, the proposed system captures images and videos in motion and collects parameters like number of cables, twist step, twist orientation, and loose fibers – features which are not present in the existing systems. • Incorporation of Mechatronic Principles: The design and functioning of the system involves mechatronic principles, combining mechanical, automation, and electronic engineering aspects to support the analysis of yarn quality. It integrates a control system that synchronizes yarn movement, maintaining stability, and ensuring accuracy during the unwinding and winding processes. • Improved System Control and User Interaction: The controller architecture of the system incorporates a Programmable Logic Controller (PLC) for system control to provide efficient and user-friendly operation. Additionally, it includes a Supervisory Control and Data Acquisition (SCADA) system with a Local Human Machine Interface (HMI) for monitoring, system variable control, and immediate notification of significant events, such as emergency, servomechanism anomaly, yarn break/absence. Chapter 1 – Introduction ___________________________________________________________________________ 12 2. Enhanced Yarn Analysis Capabilities: • Acquisition of Various Yarn Parameters: Facilitates acquisition of various yarn parameters like diameter, linear mass, volume, twist direction, twist step, number of cables, hairiness index, loose fibers, thin places (-50%), thick places (+50%), neps (+200%), U (%), CV (%), and sH (%) – even in motion, a feature absent in existing commercial systems. • Customizable Analysis: Allows users to tailor analysis parameters (yarn material, dimensional tolerance, number of yarn sections). • Data Collection and Storage: Collects and stores comprehensive data for future statistical processing. • Image and Video Visualization: Allows visualization of both images and videos under analysis. • User Monitoring of Image Processing Operations: Provides user monitoring of image processing operations. 3. Development of an application for report generation: The proposed system includes an application that provides statistical data and generates comprehensive reports. This application plays a fundamental role both in practical use by the user and for future use in textile companies, offering access to statistical information and facilitating informed decision-making. These functionalities enhance the usability of the system, supporting efficient quality control with the potential in optimizing the production processes within the textile industry. In summary, scientific contributions focus on deep learning algorithms (YOLOv5s6-Hairiness) and the build of a. yarn image dataset. Technological contributions focus on the development of a mechatronic device, and practical applications that enable the effective implementation of these technologies aiming to improve the quality of the final product in the textile industry. Both types of contributions can have a significant impact in the industry. Chapter 1 – Introduction ___________________________________________________________________________ 13 1.5 Thesis Structure This thesis is structured into eight chapters. The second chapter, Theoretical Concepts, presents the textile theoretical concepts and the deep learning concepts to understand the work. The third chapter, Literature Review, encompasses a literature review in which a survey of research projects centered on the analysis of yarn is conducted. This chapter is subdivided into three parts, the first presenting a state of the art of systems that use image processing, computer vision and artificial intelligence, the second part presenting a comparison between the various systems and the third part a critical review analysis. The fourth chapter, Image Processing and Computer Vision System for Yarn Analysis, presents the techniques used in the developed system regarding image processing and computer vision for yarn analysis. Furthermore, the chapter delves into the hardware employed for image acquisition, alongside a description of the interface software developed. The fifth chapter, Deep Learning Techniques for Yarn Hairiness Analysis, describes the deep learning techniques used for analyzing and classifying yarn hairiness. To this end, the improved architecture of the YOLOv5s6 neural network is presented, including a specific discussion of the composition of the modules with an emphasis on the modified blocks. The sixth chapter, Mechatronic Prototype Development, presents the design of the mechatronic prototype and the details of each part that constitutes it, as well as the tests and results obtained. In the seventh chapter, Experimental Results and Discussion, the experimental verification is presented, and the results obtained are discussed, including a detailed analysis of the created dataset and the effects of the changes implemented in the proposed neural network. The eighth chapter, Conclusions and Future Work, presents the main conclusions of the study, as well as some proposals for future steps, with the aim of continuing the work, which may allow new lines of investigation and/or possible technological developments. Summary of the chapter: This chapter provided an overview of the research developed, as well as its relevance to society, which served as the motivation behind this work. In this chapter, the methodology that aligns with the Chapter 1 – Introduction ___________________________________________________________________________ 14 research questions was presented, following a positivist philosophy with a deductive approach. Rooted in an innovative outcome with knowledge transfer to technology within a real-world context, an action research strategy is adopted. This strategy involves an active dynamic, aimed at achieving the desired prototype, and quantitative data was used to assess the performance of the system. In this chapter, the framework, objectives, development methodology, primary scientific/technological contributions of the work, and the structure of the thesis were presented. 2. THEORETICAL CONCEPTS Overview In this chapter, an overview of yarn characterization parameters is presented, providing a theoretical foundation to understand the developed system. Additionally, artificial intelligence concepts and the deep learning principles necessary for comprehending their application within the work are discussed. 2.1 Yarn Characterization 2.2 Statistical Parameters of Yarn 2.3 Spectral Analysis 2.4 Image Processing & Computer Vision 2.5 Deep Learning Chapter 2 – Theorical Concepts ___________________________________________________________________________ 16 2.1 Yarn characterization This subchapter covers various aspects of yarn properties, including linear mass, diameter, specific volume, hairiness, twist orientation and step, defects, as well as several statistical parameters [19], [20], [21], [22]. 2.1.1 Linear mass and diameter The connection between the diameter of a yarn and its linear mass represents a crucial basis in yarn analysis, offering a simplified way of characterization. When the yarn assumes a cylindrical form, it is geometrically established that the yarn diameter is directly related to the square root of its linear mass [19], [20], [21], [22], [23]. Figure 2 provides a visual representation of a yarn configuration, demonstrating the observable correlation between these two parameters. Figure 2 - Yarn configuration [19]. The linear mass is expressed in tex (g/km), and can be determined by Equation 1 [19], [20], [21], [22], [23]: 𝑡𝑒𝑥= 𝜙× 𝜌 × 𝜋 (𝑑 2)2×105 ↔𝑑= √4 × 𝑡𝑒𝑥/(𝜋× 105×𝜙×𝜌) (1) Where: • d is the yarn diameter (cm); • tex is the linear mass of the yarn (g/km); • ϕ is the porosity; • ρ is the density of the yarn material (expressed in g/cm3). Chapter 2 – Theorical Concepts ___________________________________________________________________________ 17 2.1.2 Specific volume The arrangement of the fibers is reflected in the specific volume of the yarn, which is determined by the volume occupied by the fibers and the interstitial spaces between them, as depicted in Equation 2 [19], [20], [21], [22], [23]: 𝑉𝑡=𝜋 𝑅2 𝑁 10−5 (2) where: • R is the yarn radius (mm); • N is the linear mass of the yarn (g/km); • Vt is the specific volume (cm3/g). 2.1.3 Hairiness Hairiness occurs when certain fibers extend outward from the main body of the yarn, resulting in some fibers forming loops and/or protruding from the yarn core. This property significantly impacts the appearance, texture, and usability of the textile, making it a critical factor for the final product quality. Detecting and measuring this parameter requires a complex analysis and generally cannot be defined by a single indicator. Typically, it is assessed by considering the number of fibers extending beyond the yarn core per unit length. Furthermore, measuring hairiness allows for the quantification of various statistical parameters essential for characterizing yarn quality and, consequently, the fabrics quality. These parameters include, but are not limited to, the standard deviation of hairiness, hairiness index, and hairiness mean deviation. Figure 3 provides a visual representation of these fibers along the yarn, which can be further categorized into protruding fibers and loop fibers (hairiness wound around the yarn body) [19], [20], [21], [22]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 24 Figure 9 – Example of a complex color in the RGB scale [32]. The HSV scale is also an additive polychromatic scale where pixels are grouped in trios. However, unlike RGB, each pixel represents one of three color properties: hue, saturation, and brightness [33], [34], [35]. • Hue: value determines the type of color being represented (green, red, orange, purple, etc.). • Saturation: indicates how vivid or intense the color will be. For instance, a saturation value of 255 with a red hue represents a vibrant and pronounced red, while a value of 50 displays a more muted red. • Value: determines how light or dark the represented color will be. A high value, for example, represents a lighter shade of red, while a lower value indicates a darker red [33], [34], [35]. Figure 10 provides a visual representation of this scale. Figure 10 – Visual representation of the HSV scale [34]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 25 2.4.2 Logic Operations To begin analyzing image processing operations, one of the starting points can be logical operations. These operations aim to compare two images of equal size and subsequently combine them into a single image. There are two main groups of logical operations: arithmetic operations and comparison operations. Arithmetic operations aim to create a resulting image that is the result of adding or subtracting the elements present in two initial images. These operations involve pairing the corresponding pixels in the two images that occupy the same position, and these pixel pairs are then summed or subtracted to obtain the final image [36]. Figure 11 provides examples of subtraction and addition, respectively. Figure 11 – Examples of image subtraction and addition [36]. These operations can serve various purposes, but the most common use is to enhance or eliminate specific aspects of an image. For instance, when identifying changes between two images in a video, a commonly used process involves subtracting these two images. This results in an image that highlights only the differences between the pair of images, as exemplified in Figure 12. Figure 12 – Example of using subtraction to highlight changes [36]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 26 Comparison operations, on the other hand, aim to compare aspects between a pair of images and generate a resulting image that combines the two originals in various ways. These operations are mostly used with binary pixel value images, i.e., black (1) and white (0). The most common types of comparisons are: • "AND" Comparison: Compares the two images and creates an image where the only pixels with a value of 1 are those that have this value in both images, assigning a value of 0 to those that have it in only one of the images (Figure 13) [36]. Figure 13 – "AND" comparison demonstration [36]. • "OR" Comparison: Compares the two images and creates an image where the pixels with a value of 1 are those that have this same value in at least one of the original images (Figure 14) [36]. Figure 14 – "OR" comparison demonstration [36]. • "XOR" Comparison: Performs an operation similar to the "OR" comparison; however, all pixels with a value of 1 in both compared images will have a value of 0 in the resulting image (Figure 15) [36]. Figure 15 – "XOR" comparison demonstration [36]. There is also an operation that, while not belonging to the comparison group, is commonly used in conjunction with them. This operation is inversion or "NOT," where all pixels in an image take the Chapter 2 – Theorical Concepts ___________________________________________________________________________ 27 opposite value, i.e., all pixels with a value of 1 become 0, and vice versa, as demonstrated in Figure 16 [36]. Figure 16 – Demonstration of the inversion operation [36]. 2.4.3 “Thresholding” Operation One of the most versatile groups of processes in image processing operations is the "Thresholding" group. Its objective is to separate various details in the image to facilitate their isolation or enhance their prominence. The simplest processes within this group are binary thresholding methods, which involve dividing all the pixels in the image into two classes. A pixel value threshold is used to create these classes. Pixels with values greater than the threshold are grouped into one class, while those with values below the threshold are grouped into another [36], [37]. In regular binary thresholding (Equation 10), pixels in the class with values above the threshold are assigned the maximum value of 255, while pixels in the class with values below the threshold are assigned the minimum value of 0. In the case of inverse binary thresholding (Equation 11), the same process occurs, but the classes are swapped. In other words, pixels in the class with values above the threshold are assigned the minimum value [36], [37]. 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)= {255 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙 (𝑥,𝑦)>𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 0 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)≤𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 (10) 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)= {255 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙 (𝑥,𝑦)≤𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 0 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)>𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 (11) In addition to these processes, there are also "Thresholding to zero" processes and their inverses. These processes function similarly to binary thresholding; however, only one of the classes is affected. For example, in the regular "Thresholding to zero" case (Equation 12), only the class containing values equal to or less than the threshold value is modified, assigning all its pixels the minimum value of Chapter 2 – Theorical Concepts ___________________________________________________________________________ 28 0. The opposite occurs in the case of inverse "Thresholding to zero" (Equation 13), where only pixels in the class above the threshold value are affected, assigning them the maximum value of 255 [36], [37]. 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)= {𝑉𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦) 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙 (𝑥,𝑦)>𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 0 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)≤𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 (12) 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)= { 0 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙 (𝑥,𝑦)>𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑉𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦) 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)≤𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 (13) The last of the binary thresholding processes is truncation (Equation 14). It acts in a very similar way to inverse "Thresholding to zero"; however, the values assigned to the pixels in the affected class are the threshold value itself, rather than the maximum value. Figure 17 provides a visualization of the effect of each type of thresholding described here [36], [37]. 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)= { 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙 (𝑥,𝑦)>𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦) 𝑖𝑓 𝑣𝑎𝑙 𝑝𝑖𝑥𝑒𝑙(𝑥,𝑦)≤𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 (14) Figure 17 – Visual representation of all types of binary thresholding [38]. All of these processes belong to the group of binary thresholding, as they separate the pixels in the image into two classes, specifically for grayscale images. However, there are more advanced processes Chapter 2 – Theorical Concepts ___________________________________________________________________________ 29 that can isolate pixels in an image into 3 or more classes, which obviously involves a greater number of threshold values and a higher level of separation and feature enhancement in the image. It is also possible to perform these processes on images in other color spectrums, such as RGB or HSV, which can not only separate classes by color but also by brightness and saturation [36], [37]. These processes are considerably more complex and will not be further explored here as they are beyond the scope of this work. 2.4.4 OTSU method The OTSU method is commonly referred to as another type of thresholding; however, it is more accurate to state that it is a method that can be associated with any of the other thresholding processes. In other words, when the OTSU method is combined with, for example, a binary thresholding process, it assists in determining, through mathematical algorithm methodology, the best threshold value choice [39], [40]. The way the algorithm operates is by testing all possible values for the threshold, seeking the one that yields the maximum variance between classes (in this case, classes refer to the image background and subject). This variance can be calculated using equation 15 [39], [40]: 𝜎𝑊 2= 𝑊𝑠×𝜎𝑠2+𝑊𝑓×𝜎𝑓2 (15) Where: • W corresponds to the relative weight of each class, i.e., the probability that any pixel belongs to the respective class. This means that the greater the number of pixels belonging to each class, the higher their relative weight. • s and f correspond to the nomenclature of the subject and background classes, respectively. Figure 18 provides a visualization of the application of this method to an image. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 30 Figure 18 – Graphical representation of the OTSU method [39]. 2.4.5 Linear Filters Another commonly used process in image processing methodologies is image transformation using linear filters [36], [41], [42]. This process differs from the previous ones in that the final value of a pixel depends not only on its initial value but also on the initial values of its neighbors. It is a particularly useful process for detecting contours, points, geometric shapes, and objects in general in an image. For example, it is a process that can be used to detect the presence of a person in an image. It can also be used to apply various filters to the image, such as blurring (Figure 19) or adding noise. Figure 19 – Example of applying a linear filter for blurring [36]. Before explaining the process by which the transformation occurs, it is important to clarify the term "linear filters" in this context. A linear filter refers to a square matrix of order 3 or higher. For reasons that will be mentioned later, the order of a mask matrix must always be an odd number. Figure 20 shows an example of a simple linear filter. 1 9[1 1 1 1 1 1 1 1 1] Figure 20 – Example of a linear filter matrix. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 31 This transformation begins by overlaying the filter matrix onto the pixels of the image, as illustrated in Figure 21. Figure 21 – Overlaying a filter on image pixels [36]. Next, a multiplication operation is performed between the matrices. However, this operation is not a typical mathematical matrix multiplication. The operation involves multiplying each term in the filter matrix by the term in the same position in the image matrix. For example, initially, the terms at position (1, 1) in the filter matrix are multiplied by the term at position (1, 1) in the image matrix. This process is repeated for all terms in the filter matrix. After executing this operation, a new matrix is obtained. Figure 22 represents the new matrix for the example of the previous image. It is important to note that, when dealing with pixel values, decimal numbers are always rounded. In this context, only integers between 0 and 255 are used. [0 0 0 0 0 0 0 0 3] Figure 22 – The resulting matrix from the multiplication of matrices. Next, the sum of all the terms in the resulting matrix is calculated. The result is the final value of the central pixel in the image matrix, as shown in Figure 23. Figure 23 – Replacement of the final value in the matrix [36]. This process of application of the linear filter, can also be described by Equation 16. 𝑁𝑒𝑤[𝑖,𝑗]= ∑ ∑ 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙[𝑖+𝑘,𝑗+𝑙]×𝐹𝑖𝑙𝑡𝑒𝑟[𝑘,𝑙] 𝑙𝑘 (16) Chapter 2 – Theorical Concepts ___________________________________________________________________________ 32 This entire process is repeated as many times as the filter fits into the original image, always with a different pixel in the central position of the overlapping matrices. The fact that this process requires the existence of a central term (a term at the geometric center of the matrix) in the resulting matrix means that the linear filter matrix must have a central term. This requirement necessitates that the linear filter be a square matrix of odd order, as even-order matrices do not have a single central term. One of the most common and versatile linear filters is the Gaussian filter. Its purpose is to reduce noise in an image by applying a slight blur to the image. Applying a Gaussian filter is one of the first steps in many image processing methodologies. Although counterintuitive, blurring the image can result in a clearer final image because removing initial noise leads to a clearer final image. It is important to note that the larger the order of the Gaussian filter applied, the greater the effect on the image. Figure 24 provides an example of the application of a Gaussian filter. The contrast of this type of blur is compared with that of Figure 19. Figure 24 – Application of a Gaussian filter [36]. The term "Gaussian filter" does not refer to a single linear filter but rather to a type of filter that follows the same rule. These filters are named after the normal distribution in two dimensions. Figure 25 gives an example of a 7x7 order Gaussian filter matrix. Figure 25 – Example of a 7x7 Gaussian matrix [36]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 33 The second type of filters to discuss are line detection filters. Their goal is to identify the presence of lines with the desired orientation. They are widely used in applications that seek to identify objects in an image. The filter matrix used attempts to match the shape of the object to be found. Figure 26 provides an example of a vertical line detection filter as well as how it transforms the original image. Figure 26 – Example of applying a vertical line detection filter [36]. The example shown is a simple application of these filters. More commonly, multiple filters are used to simultaneously detect vertical, horizontal, and diagonal lines, revealing the presence of more complex shapes. Figure 27 demonstrates the application of three filters to identify a complex object. Figure 27 – Using multiple filters to detect a complex shape [36]. For the last example, a filter matrix of order 3 was used, but a higher level of detail can be achieved by increasing the order of the filter matrix. Figure 28 shows the result of performing the same operation but with a 5x5 order filter. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 40 A ‘dataset’, in broad terms, refers to a collection of data that has been gathered and organized in some way. These data can represent information about a variety of phenomena or objects, such as images, text, numbers, or any other form of measurable information. 'Datasets' are essential for feeding machine learning algorithms and enabling them to acquire knowledge and make informed decisions based on the available data. Before delving into the various types of machine learning, it is important to understand that 'datasets' play a pivotal role as they serve as the raw material upon which algorithms learn and develop their capabilities [46], [47], [48]. From this point onward, the different types of machine learning are introduced, including supervised learning, unsupervised learning, and reinforcement learning, and provide an explanation of how these algorithms utilize datasets in specific ways to achieve their respective objectives [46], [47], [48]. Figure 39 – Subsets of artificial intelligence [46]. • Supervised learning: When the algorithm has a labeled dataset (data with their respective correct outputs) and learns by comparing the model output with the expected output, and its parameters are readjusted, if necessary, until reaching an acceptable threshold and predetermined. Figure 40 shows the processes in detail. Figure 40 – Supervised learning. Steps: (1) made annotations in the dataset; (2) dataset separation; (3) Chapter 2 – Theorical Concepts ___________________________________________________________________________ 41 algorithm training; (4) dataset test for algorithm evaluation and performance [48]. Supervised learning can be further categorized into two main types: classification and regression. Classification involves predicting a discrete value where the algorithm identifies input data as belonging to a specific class (for example, determining whether a photo contains a pre-identified animal). The algorithm's performance is evaluated based on how accurately it can correctly classify new images according to the provided annotations. Regression, on the other hand, deals with continuous data [46], [47], [48]. • Unsupervised learning operates without prior annotations on the data. In this category, the algorithm is provided with a dataset lacking annotations and seeks to discover similarities among groups of data. Within unsupervised learning, two primary tasks exist: data clustering and information extraction. Data clustering, also known as clustering algorithms (Figure 41), involves grouping data based on their similarities. Information extraction (Figure 42), on the other hand, entails the algorithm associating new information with previously obtained data, such as providing movie recommendations on a website based on a user's past viewing history [46], [47], [48]. Figure 41 – Clustering algorithm [49]. Figure 42 – Unsupervised learning–Information extraction [50]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 42 • Reinforcement learning involves training machine learning models to make a series of decisions based on a reward/punishment framework. In essence, this technique aims to accomplish a specific task, initially employing a trial-and-error approach. Subsequently, the outcomes of each attempt, regardless of success, are utilized for training by means of a reward/punishment system, as depicted in Figure 43 [50]. Figure 43 – Unsupervised learning–Information extraction [50]. As previously referred a dataset is a collection of data used as the foundation for developing artificial intelligence algorithms or various other types of data science projects. Typically, creating a dataset is a critical and time-intensive phase that significantly impacts the performance of the system. It is often divided into three subsets: training, validation, and test datasets, as illustrated in Figure 44 [50]. Figure 44 - Unsupervised learning—information extraction [50]. • Training dataset: This set consists of samples used for model development and convergence. It serves as the basis for training the algorithm and enabling it to learn from the provided data. • Validation dataset: In this phase, samples from the validation dataset are employed to objectively assess a model's performance after the training process. This stage also allows for fine-tuning Chapter 2 – Theorical Concepts ___________________________________________________________________________ 43 the algorithm's hyperparameters, which can lead to adjustments and improvements in the training phase. • Test dataset: The test dataset comprises samples used for rigorously testing the algorithm. It provides a final evaluation of the model's performance. It is crucial that the samples in the test dataset are not used in either the training or validation phases to maintain the integrity of the evaluation results. 2.5.2 Neural Networks for Object Detection A significant advancement has been made in the field of computer vision. Thus far, the emergence of algorithms designed for image classification has been witnessed, enabling the discernment of the presence of objects such as dogs, cats, cars, or bicycles within an image. However, the next frontier in this domain is object detection. Distinct from classification algorithms, object detection algorithms aim to precisely locate objects of interest within an image by drawing bounding boxes (referred to as object detectors) or bounding curves (instant segmentation) around them [51]. Notably, a single image can contain multiple distinct objects requiring detection. One possible solution would involve partitioning the image into various regions of interest and employing convolutional neural networks to classify the presence of objects within these regions. Nonetheless, this approach presents challenges, particularly when objects of interest exhibit varying spatial locations and proportions within the image. Consequently, selecting numerous regions may lead to excessive computational costs, especially in scenarios such as highway surveillance where numerous vehicles need to be detected [51]. To address these issues, a family of algorithms was developed, including R-CNN (Region-based Convolutional Neural Network), Fast R-CNN (improvement over R-CNN), Faster R-CNN (extension of Fast R-CNN), R-FCN (Region-based Fully Convolutional Networks), YOLO (You Only Look Once), and others [51]. YOLO YOLO (You Only Look Once) was introduced by Joseph Redmon and Ali Farhadi in 2015 during their doctoral studies [52]. Since its initial release, YOLO has undergone several iterations, with the latest version being YOLOv8. YOLO represents a departure from the region-based algorithms discussed earlier. It is a convolutional network that predicts both bounding boxes and class probabilities for these boxes. YOLO divides the image into N grids, each of which has a uniform SxS dimension. Each grid is responsible for detecting and pinpointing the objects it encompasses. These grids predict the coordinates Chapter 2 – Theorical Concepts ___________________________________________________________________________ 44 of bounding boxes (relative to cell coordinates), as well as the object label and the probability of its presence within the cell. By having multiple cells predict the same object with varying bounding box predictions, YOLO significantly reduces computational requirements, as both detection and recognition tasks are efficiently handled within the image (see Figure 45). Figure 45 - Models the detection as a regression problem [52]. In Figure 45, the image is divided into an S×S grid, and for each cell of the grid, it predicts bounding boxes (B), confidence scores for those boxes (thicker lines indicate higher confidence), and class probabilities (C). These predictions are encoded as a tensor of dimensions S×S×(B×5+C) [52], [53]. The YOLO architectures consist of three distinct components, backbone, neck, and head, as depicted in Figure 46. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 45 Figure 46 - YOLO architecture and how it works [53]. However, this process often results in numerous duplicate predictions. To address this issue, YOLO utilizes the Non-Maximum Suppression (NMS) technique [54]. NMS operates by eliminating redundant bounding boxes with lower probability scores. YOLO achieves this by evaluating the probability scores associated with each option and selecting the one with the highest score as the primary choice. Subsequently, it suppresses bounding boxes that exhibit high Intersection over Union (IoU) with the currently selected high probability bounding box (as illustrated in Figure 47). This process continues until there are no more boxes to be removed, as demonstrated in Figure 48. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 46 Figure 47 - Intersection over Union—B1: Predication Box and B2: Detection Box [55]. Figure 48 - Non-Maximum Suppression Process example [54]. YOLO stands out for its exceptional speed, achieving an impressive rate of 45 frames per second, surpassing the performance of other object detection algorithms. However, it is worth noting that YOLO does have a limitation in terms of detecting exceptionally small objects in images, primarily stemming from the inherent spatial constraints of the algorithm [54], [55]. Data Augmentation Data augmentation in YOLOv5 involves the practice of generating additional training data by applying various transformations to the original images. The primary objective is to enhance the diversity within the training dataset, thereby improving the model's capacity to generalize effectively to new and unseen data [16]. Within YOLOv5, data augmentation encompasses a range of techniques, including random cropping, rotation, flipping, and color jittering, all of which are employed to create novel training images. These transformations are randomly applied to the original images during the training process, expanding the dataset and introducing a broader variety of images for the model to learn from [16]. As illustrated in Figure 49, you can observe an example of augmented training images in YOLOv5. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 47 Figure 49 - Example of augmentation [56]. Through data augmentation, the model gains proficiency in detecting objects under various conditions, encompassing different orientations, scales, and lighting scenarios. This contributes to the enhancement of the model's accuracy and robustness [16]. The data loader in YOLOv5 executes three categories of augmentations [16]: • Basic Augmentations: This category includes random cropping and resizing of images, horizontal flipping, and random color distortion. These techniques aim to enhance the model's generalization capabilities; • Mosaic Augmentations: This category combines four individual images into a single mosaic image. This helps the model learn to detect objects that may appear in proximity within a single image; Chapter 2 – Theorical Concepts ___________________________________________________________________________ 48 • Auto Augmentations: This category employs reinforcement learning to search for the optimal set of augmentations tailored to a specific dataset. This optimization contributes to improved model performance on that particular dataset. In addition to the above, YOLOv5 employs color space adjustments as a form of data augmentation. This involves modifying the color balance, brightness, contrast, and saturation of training dataset images. This adaptation enhances the model's robustness to variations in real-world lighting conditions and color schemes. The data loader in YOLOv5 implements color space adjustments by randomly altering the hue, saturation, and brightness of each image during training. It is important to note that augmentation does not generate entirely new data; rather, it presents existing data in diverse, random configurations each time an image is utilized. Consequently, no two views of an image are identical during the training process. Settings for training augmentation are defined in a hyperparameter file, and hyperparameter evolution can be employed to optimize these values according to specific training requirements (as shown in Figure 50). Figure 50 - Training augmentation defined in a hyperparameter file [56]. Auto Learning Bounding Box Anchors Within the context of YOLOv5, the Auto Learning Bounding Box Anchors (Figure 51) technique is employed to autonomously produce anchor boxes meticulously tailored to the characteristics of the particular dataset in use. Throughout the training process, the model scrutinizes the distribution of object dimensions and configurations within the training dataset, subsequently fine-tuning the anchor box sizes Chapter 2 – Theorical Concepts ___________________________________________________________________________ 49 and aspect ratios to align with these characteristics. This adaptive approach enables the model to more effectively pinpoint objects presenting varying sizes and shapes, ultimately enhancing the precision of object detection [57]. Figure 51 - Learning Bounding Box Anchors [57]. To illustrate, consider an instance of anchors that have been derived from the training data and incorporated into a YOLOv5 configuration file (Figure 52). Figure 52 - A sketch of the object detection task. In this particular instance, it can be observed that three anchor boxes are defined, with each anchor box represented as a list containing six values. These values correspond to the width and height of the anchor boxes at three distinct scales. It is noteworthy that these anchor boxes have been deduced from the training data and are utilized for generating box predictions during the inference phase. In the realm of object detection (see Figure 53), the primary objective is to identify and precisely delineate objects within an image. This differs markedly from image classification, where the primary task involves categorizing the entire image into a single class. In object detection, the challenge lies in the potential existence of multiple instances of identical or dissimilar objects within the image. The overarching goal is to accurately predict both the presence and the precise location of all these objects. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 56 Evaluation Metrics Evaluation metrics in deep learning are quantitative measures used to assess the performance of a deep learning model in specific tasks. Common metrics include accuracy, precision, recall, F1-score, loss, and the confusion matrix. These metrics help in understanding how well the model is performing and comparing different models. The initial evaluation metrics for a deep learning algorithm like YOLOv5 serve the purpose of assessing the performance of the object detection algorithm on a specific dataset. These metrics, commonly employed, are specific to YOLOv5 or other similar algorithms (Faster R-CNN - Region-based Convolutional Neural Network; SSD - Single Shot Multibox Detector; RetinaNet; Mask R-CNN - Mask Region-based Convolutional Neural Network; EfficientDet; YOLOv7; YOLOv8) [61], [62]: 1) Average Precision (AP): Average Precision is a widely used metric for evaluating object detection performance. It measures the accuracy of detections at different confidence levels (thresholds) and calculates the area under the Precision-Recall curve. The higher the AP value, the better the model performance. Average Precision is computed using the Precision-Recall (P-R) curve. The P-R curve is obtained by varying the confidence threshold for positive detections and calculating the Precision and Recall for each threshold. AP is calculated by taking the average of precisions calculated at specific points on the curve (e.g., at recall intervals). AP is as follows: AP= 1 n ∑P (r)∆r r (17) Where: • n is the number of threshold points on the P-R curve; • P(r) is the Precision at the recall point r; • Δr is the difference in recall between two consecutive points on the P-R curve. 2) Mean Average Precision (mAP): mAP is the average of the Average Precisions calculated for various object classes. It is a global metric that provides an overall view of the model performance across all classes. mAP is as follows: mAP= 1 C ∑APt c t=1 (18) Where: • C represents the number of categories in the dataset. The higher the mAP value, the better the model performance; Chapter 2 – Theorical Concepts ___________________________________________________________________________ 57 • [email protected] or mAP@50 or IoU = 0.50 mean Average Precision with an Intersection over Union threshold of 0.5 or 50%; • mAP.5:95 means average mAP over different IoU thresholds, from 0.5 to 0.95. 3) IoU (Intersection over Union): IoU is a metric that measures the overlap between the bounding box predicted by the model and the ground truth bounding box of the object. A high overlap indicates an accurate detection. IoU is as follows: IoU=Intersection Area Union Area (19) 4) Accuracy: Accuracy measures the proportion of correct detections relative to the total detections made by the model. Accuracy is as follows: Accuracy=Number of Correct Detections (Total Number of Detections) (20) Or Accuracy= TP+TN TP+TN+FP+FN (21) Where: • True Positive (TP): the number of samples correctly classified as positive; • False Positive (FP): the number of samples incorrectly classified as positive; • False Negative (FN): the number of samples incorrectly classified as negative; • True Negative (TN): the number of samples correctly classified as negative. 5) Precision: Precision (also known as Positive Predictive Value) measures the proportion of correct detections among the total positive detections made by the model. Precision is as follows: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= 𝑇𝑃 (𝑇𝑃+𝐹𝑃) (22) Chapter 2 – Theorical Concepts ___________________________________________________________________________ 58 6) Recall: Recall (also known as Sensitivity or True Positive Rate) measures the proportion of correct detections among the total true objects present in the images. Recall is as follows: 𝑅𝑒𝑐𝑎𝑙𝑙= 𝑇𝑃 (𝑇𝑃+𝐹𝑁) (23) 7) F1-Score: The F1-Score is the harmonic mean of Precision and Recall. It is useful when one desires a metric that considers both the precision and recall of the model. F1-Score is as follows: F1− Score= 2 x Precision x Recall Precision+Recall (24) 8) Confusion Matrix: The Confusion Matrix (Figure 59) is a table that shows the count of correct and incorrect detections for each class of object. It is a useful tool for evaluating the model's discrimination ability across different classes. These evaluation metrics enable the assessment of the quality and performance of the neural network during both training and testing processes. Figure 59 –Confusion Matrix for Binary Classification [63]. Typically, in the evaluation of detection quality, results are categorized into four groups by comparing the model output to the ground truth. When the model correctly detects an object, it is labeled as a True Positive (TP). Conversely, if the model detects an object that is not present in the image, it is Chapter 2 – Theorical Concepts ___________________________________________________________________________ 59 referred to as a False Positive (FP). When an object in the ground truth goes undetected by the model, it is termed a False Negative (FN). True Negatives (TN), representing correctly undetected objects or background, are not typically considered in the evaluation of object detection. These four categories form the basis of a confusion matrix. The Precision-Recall Curve and mAP (Mean Average Precision) The Precision-Recall Curve and mAP (Mean Average Precision) are important evaluation metrics in machine learning, particularly for tasks like object detection and information retrieval [64]. The Precision-Recall Curve, depicted in Figure 60, provides a graphical representation of the trade-off between precision, and recall for a machine learning model. Precision assesses the accuracy of the model positive predictions, while recall (or sensitivity) evaluates the model ability to capture all the actual positive instances. This curve illustrates how precision and recall values change as the model's decision threshold is adjusted, offering insights into how the model performance can be fine-tuned to meet specific task requirements. The area under the Precision-Recall Curve (AUC-PR) serves as a quantitative measure of the overall model performance, with a higher AUC-PR value indicating superior performance [64]. Figure 60 – An example of the precision-recall curve [64]. mAP (Mean Average Precision), shown in Figure 61, serves as a comprehensive metric for assessing a model prediction in object detection and information retrieval tasks, condensing the overall quality of those predictions into a single numerical value. It is computed by taking the average of the average precision (AP) values for each class or category in a multi-class problem. AP represents the area under the Precision-Recall Curve for a single class and quantifies the model ability to distinguish objects of that class from the background. mAP is particularly valuable when evaluating models in scenarios Chapter 2 – Theorical Concepts ___________________________________________________________________________ 60 involving multiple object classes, as it offers a unified performance metric that accounts for performance across all classes [65]. Figure 61 – An example of the precision-recall curve [65]. In summary, the Precision-Recall Curve illustrates how changes in the classification threshold affect precision and recall. It helps in understanding the model's trade-off between making accurate positive predictions and capturing all relevant instances. mAP, on the other hand, is a summarizing metric that is particularly useful for evaluating models in multi-class classification tasks, such as object detection, by averaging the performance across different classes, providing an overall assessment of the model effectiveness. Both metrics are crucial for evaluating the performance of machine learning models, especially in tasks where precision and recall are vital, such as object detection [65]. k-fold Cross Validation k-fold Cross-Validation is a technique used in machine learning and model evaluation to assess a model performance and its generalization capabilities. It helps to address the problem of overfitting and to obtain a more reliable estimate of how well a model is likely to perform on unseen data. k-fold CrossValidation works as follows [66]. 1. Data Splitting: The dataset is divided into K approximately equal-sized folds or subsets. For example, if k is set to 5, the dataset is divided into five parts; 2. Model Training and Testing: The training and evaluation process is repeated k times. In each iteration, one of the k folds is held out as a validation set, while the remaining k-1 folds are used to train the model. The model is then tested on the validation set; 3. Performance Metrics: For each iteration, a performance metric (e.g., accuracy, mean Chapter 2 – Theorical Concepts ___________________________________________________________________________ 61 squared error) is calculated based on the model's performance on the validation set; 4. Average Performance: After k iterations, one has k performance metric values (one for each fold). The final performance metric is typically calculated as the average of these k values. This provides a more robust estimate of the model's performance compared to a single train-test split. k-fold Cross-Validation offers several advantages in model evaluation. Firstly, it provides a more accurate estimation of a model's performance by subjecting it to testing on different subsets of the data. This approach aids in assessing how well a model generalizes to unseen data, as it undergoes testing on various data partitions, thus enhancing its reliability. Additionally, k-fold Cross-Validation helps mitigate the risk of overfitting, as the model is trained on multiple data subsets and tested on different validation sets, ensuring that the model's performance is not overly tailored to a specific dataset. This method proves especially valuable when dealing with limited datasets, where maximizing data utility is crucial [67]. The choice of k (e.g., 5, 10) depends on the specific dataset and problem. Larger values of k reduce the variance in the performance estimate but can be computationally expensive. Smaller values may introduce higher variance but require less computation. In practice, k-fold Cross-Validation is a valuable tool for assessing and comparing machine learning models and selecting the best-performing one for specific problem [67]. k-fold Cross Validation stands as a robust approach for appraising machine learning models. Its merits lie in mitigating overfitting and underfitting issues, ultimately delivering a more precise gauge of a model's effectiveness on unseen data. This technique is extensively employed in practical scenarios for model validation, comparison, and the selection of an optimal model tailored to a specific problem [68]. A visual representation of the method's implementation can be observed in Figure 62 [69]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 62 Figure 62 – An example of the precision-recall curve [69]. The value k = 10 is frequently chosen for k-fold Cross Validation due to several compelling reasons [70], [71]: 1. Reduction in Result Variance: The use of a larger k value, such as 10, serves to diminish the variance in the results. Consequently, when k-fold Cross Validation is conducted multiple times, the outcomes are more likely to exhibit consistency and accurately reflect the model's true performance; 2. Efficient Data Utilization: In the case of k = 10, roughly 90% of the dataset is allocated for training during each iteration, with only 10% dedicated to validation. This balance is generally considered effective for ensuring that the model learns comprehensively and is rigorously evaluated; 3. Equitable Model Comparisons: The selection of k = 10 fosters equitable comparisons among different models. This is achieved by evaluating all models using the same dataset and adhering to the identical cross-validation methodology. The procedure for k-fold Cross Validation with k = 10 unfolds as follows [72], [73]: 1. The dataset is partitioned into 10 roughly equivalent segments, or folds; 2. The model undergoes 10 rounds of training and evaluation. In each iteration, one-fold is set aside as the validation set, while the remaining 9 folds constitute the training set; 3. In each iteration, performance metrics such as precision, recall, F1-score, among others, are computed to assess the model performance; Chapter 2 – Theorical Concepts ___________________________________________________________________________ 63 4. Upon completion of the 10 iterations, the performance metrics obtained in each round are typically averaged to yield an aggregate representation of the model's performance. This provides a more dependable estimation of how the model will perform on unseen data. 2.5.3 YOLOv5 Algorithm for Object Detection It is the fifth iteration in the YOLO series of algorithms, which have gained popularity for their ability to perform real-time object detection with impressive accuracy. Some key features and characteristics of YOLOv5 are as follows [74]: • Accuracy and Speed: YOLOv5 is renowned for its balance between accuracy and speed in object detection tasks. It achieves impressive real-time performance while maintaining high precision; • Architectural Improvements: YOLOv5 introduces architectural enhancements over its predecessors, such as CSPDarknet53 as the backbone and PANet as the neck, leading to improved feature extraction and aggregation; • Scale Handling: YOLOv5 effectively addresses scale variation, enabling it to detect objects of various sizes within an image, thanks to the feature pyramid and anchor box techniques; • Adaptive Data Augmentation: YOLOv5 employs data augmentation techniques like random cropping, rotation, and color jittering to enhance its ability to generalize to new and unseen data; • Point-Based Sampling: The PA-Net (Point Aggregation Network) Neck in YOLOv5 utilizes pointbased sampling to aggregate features at different scales, contributing to its accuracy; • Model Variants: YOLOv5 comes in different variants (e.g., YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x) with varying model sizes to suit different computational resources and deployment scenarios; • State-of-the-Art Performance: YOLOv5 has achieved state-of-the-art results in terms of accuracy and efficiency in object detection tasks, making it a popular choice for various applications, including autonomous driving, surveillance, and more; • Open-Source: YOLOv5 is open-source, allowing researchers and developers to access its codebase, customize it, and integrate it into their projects; • Community Support: Due to its popularity, YOLOv5 has a growing community of users, contributing to ongoing development, improvements, and the availability of pre-trained models and resources; • Versatility: YOLOv5 can be adapted to a wide range of object detection tasks, including singleobject detection, multi-object detection, and even custom object detection scenarios. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 64 In summary, YOLOv5 is a cutting-edge object detection algorithm known for its speed, accuracy, and versatility, making it a significant asset in computer vision applications. Sub-variants of YOLOv5 YOLOv5 is available in four primary versions: small (s), medium (m), large (l), and extra-large (x), with each version delivering increasingly improved accuracy rates [56]. Additionally, the training time required for each variant varies (Figure 63). Figure 63 – Sub-variants of YOLOv5 [56]. The objective of the Figure 63 is to demonstrate the performance of an object detection model concerning its inference time (X-axis) and overall effectiveness (Y-axis). Initial findings indicate that YOLOv5 excels in achieving exceptional performance relative to other cutting-edge techniques. As illustrated in the Figure 63, all YOLOv5 variants exhibit faster training times when compared to EfficientDet. EfficientDet is a type of object detection model optimized for efficient and accurate object detection, known for its ability to achieve good performance while maintaining low computational demands. Notably, the highest performing YOLOv5 model, YOLOv5x, can process images at a significantly accelerated rate while maintaining a level of accuracy similar to that of the EfficientDet D4 model [56]. It is important to note that YOLOv5 performance improvements are primarily attributed to the training procedures in PyTorch, while the fundamental model architecture remains closely aligned with YOLOv4 [56]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 65 Overview of the YOLOv5s6 Architecture The YOLOv5s6 deep learning detection algorithm has been utilized to identify and characterize yarn hairiness, as will be elaborated in chapter 5. This choice is motivated by its optimized deployment on resource-constrained edge devices, rendering it suitable for manufacturing applications, as well as its unrestricted accessibility for research purposes. Tests were conducted to compare different YOLO versions, and YOLOv5s6 outperformed the others in terms of evaluation metrics, establishing it as the preferred choice. The YOLOv5s6 model comprises three essential components, as illustrated in Figure 64 [75], [76], [77]: • Backbone: YOLOv5 employs a convolutional backbone to extract features from input images. This backbone performs initial convolutions and extracts low-level representations from the input images; • Neck: In certain variants of YOLOv5, such as YOLOv5m, YOLOv5l, and YOLOv5x, a "neck" is incorporated. This neck consists of a sequence of convolutional layers designed to fuse features from various scales. This capability enables the model to capture information at different levels of resolution, enhancing its ability to detect objects of varying sizes; • Head or Detect: The "head" or "Detect" is the final component of the model, where the object detection process occurs. It comprises several convolutional and detection layers responsible for predicting bounding boxes and object classes within the input image. Figure 64 illustrates the network structure for YOLOv5s6, highlighting the interplay between these three key components [76]. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 72 Figure 70 - Comparison of activation functions [81]. In resume, these activation functions play an important role in the training and performance of neural networks, and the choice of which one to use can depend on the specific task and desired results. In YOLOv5s and other YOLO variants, as previously referred, these activation functions are used in the network layers to introduce non-linearity and learning capability. Hyperparameters in YOLOv5s6 Hyperparameters in machine learning models, including deep learning models like YOLOv5, play a crucial role in defining the training and data augmentation settings [82]. When dealing with larger models, which typically have more parameters and complexity, it is common to adjust certain hyperparameters to mitigate overfitting and improve model generalization. Hyperparameters for data augmentations are often defined in a separate hyperparameter file, which is provided as part of the training configuration. In the case of YOLOv5, the hyperparameters that control data augmentations can be found in the 'hyp.scratch.yaml' or 'hyp.scratch-low.yaml' file (or a similar file) located within the data directory. These augmentation hyperparameters include settings for various types of data transformations such as rotation, translation, scaling, flipping, and more. By configuring these hyperparameters, it can be controlled the type and intensity of augmentations applied to the training data, which can help improve the model ability to generalize to different situations. In Figure 71, it is explained how it works the YOLOv5 image data augmentation with a dataset in training phase of a neural network with defined hyperparameters. Chapter 2 – Theorical Concepts ___________________________________________________________________________ 73 Figure 71 – YOLOv5s6 Image Augmentation [82]. It is common practice to define these hyperparameters in a separate file to keep them organized and easily configurable, allowing for experimentation with different augmentation strategies during model training. Adjusting these hyperparameters can have a significant impact on the robustness and performance of the trained model. The hyperparameters configuration for training the YOLOv5 model, are following explained [82]: • lr0: The initial learning rate used by the optimizer during training. It is set to 0.01 for SGD and 0.001 for Adam. SGD stands for "Stochastic Gradient Descent," which is an optimizer used to adjust the weights of a neural network during training. "Adam" refers to another optimizer called "Adam" (Adaptive Moment Estimation), which is also used to optimize the parameters of a neural network. The values mentioned (0.01 for SGD and 0.001 for Adam) are the initial learning rates used by these optimizers to adjust the network's weights during training. Each optimizer may require a different learning rate to work effectively based on the problem and the neural network architecture; • lrf: The final learning rate for the OneCycleLR at the end of training, calculated as the product of the initial learning rate (lr0) and this value (0.01); • momentum: The momentum value used in the SGD optimizer or the beta1 value used in the Adam optimizer; • weight_decay: The weight decay value used by the optimizer to control the penalization of larger weights in the model. In this case, it is set to 0.0005; • warmup_epochs: The number of "warm-up" epochs at the start of training during which the learning rate is gradually increased. It is set to 3.0 epochs; Chapter 2 – Theorical Concepts ___________________________________________________________________________ 74 • warmup_momentum: The initial momentum value during the warm-up period, configured as 0.8; • warmup_bias_lr: The initial learning rate for the bias term during the warm-up period, set to 0.1; • box, cls, cls_pw, obj, obj_pw: These parameters adjust the weights of different components of the YOLOv5 loss function, influencing the impact of different parts of the loss function during training; • iou_t: The training threshold for Intersection over Union (IoU) used to determine whether a detection is considered a true positive during training. In this case, it is set to 0.20; • anchor_t: The threshold that determines whether an anchor is considered active or inactive; • fl_gamma: The gamma value used in the focal loss. In this case, it is set to 0.0. • hsv_h, hsv_s, hsv_v: These parameters control variation in the hue, saturation, and value of the HSV color space during data augmentation; • degrees, translate, scale, shear, perspective: Parameters for adjusting various geometric transformations applied to images during data augmentation, such as rotation, translation, scale, shear, and perspective; • flipud, fliplr: Probabilities for applying vertical and horizontal flipping to images; • mosaic, mixup, copy_paste: Probabilities for applying techniques like mosaic, image mixup, and segment copy-paste during data augmentation. These hyperparameters are crucial for training the YOLOv5 model and its ability to generalize for object detection. Properly tuning these hyperparameters is essential to obtain a high-quality object detection model. Summary of the chapter: This chapter provided a comprehensive foundation of theoretical concepts essential for understanding the subsequent development of a yarn quality analysis system. It encompasses yarn characterization parameters, statistical parameters of yarn, spectral analysis techniques, the principles of image processing and computer vision, and the fundamentals of deep learning. These topics collectively set the stage for the work objectives, enabling a deeper understanding of the methodologies and technologies involved in this interdisciplinary research endeavor. Additionally, this chapter introduces key concepts related to YOLOv5s, which will be further explored and applied in chapter 5 considering their relevance to the work goals. 3. LITERATURE REVIEW Overview Technological evolution has reached important proportions in terms of systems obtaining characteristics that measure yarn quality and the use of artificial intelligence to detect yarn defects. Therefore, this chapter presents some studies that explore image processing/computer vision and artificial intelligence approaches to obtain yarn characteristics. It is intended to analyze what the scientific community has explored, the challenges encountered, and the problems that still need to be addressed in order to undertake the objectives of this work. Next, the most relevant works are presented. At the end, a comparative table and a critical review analysis are presented. 3.1 Use of image processing, computer vision and artificial intelligence in yarn analysis 3.2 Comparison between systems with image processing, computer vision and artificial intelligence techniques 3.3 Critical review analysis Chapter 3 – Literature Review ___________________________________________________________________________ 76 3.1 Use of image processing, computer vision and artificial intelligence in yarn analysis Technological evolution has achieved significant advancements in systems for measuring yarn quality characteristics. Consequently, this subchapter introduces various solutions and initiatives that leverage technology to assess yarn properties. It is important to examine the research undertaken by the scientific community, the obstacles faced, and the outstanding issues that need solutions for enhancing research approaches. Each subchapter highlights the most significant works from 2020 to 2023, providing an assessment of the progress made in this field. Based on the current research, limited documented instances of alternative yarn analysis systems are available for comparison with the proposed developed system, which will be extensively discussed in subsequent chapters. The analysis performed outline the shortcomings and inadequacies found within each system, thereby providing a critical evaluation of their limitations. 3.1.1 System A – Study by Zhisong Li et al. (2020) In the computer vision system developed by Zhisong Li and colleagues in 2020 [83], henceforth referred to as System A, an evaluation of yarn quality was conducted. This assessment focused on acquiring the following parameters: Yarn diameter, Defects and Hairiness. These characteristics were determined using computer vision and image processing techniques. In their proposed system (Figure 72), they defined a diameter image processing unit (DIPU) and selected several sampling points from the moving yarn. The DIPU corresponding to each sampling point was segmented from the captured yarn images. The average DIPU diameter was used to represent the yarn diameter of the test points. When extracting yarn images, the DIPU was categorized into: • Foreground region; • Defined background region; • Unknown region based on gray level projection distribution characteristics. Chapter 3 – Literature Review ___________________________________________________________________________ 77 Figure 72 - Flowchart of the method for detecting yarn parameters and evaluating yarn quality [83]. The unknown region was further processed using the Poisson Matting method [84], which employed a defined connectivity classifier to separate the yarn image from the background image. After the classifier extracted the yarn core, hairiness was differentiated using image subtraction. Furthermore, yarn defects were analyzed using statistical methods to assess yarn quality. The designed system employed a closed box as an image capture platform to shield against stray light disturbance. They used white diffused glass to eliminate reflected light and shadows. For simulating yarn movement, a driver and voltage controller were utilized. The integrated structure of the system is depicted in Figure 73. The yarn image capture system primarily consisted of two components: Figure 73 - System device to detect yarn parameters [83]. Chapter 3 – Literature Review ___________________________________________________________________________ 78 • Imaging system, including a sensor, camera lens, and light source, with a CCD (charge-coupled device) image sensor for collecting the sequence of images from the yarn; • Imaging lens with a large depth of field to ensure clear analysis of hairiness in different positions. Images were captured using a SONY C Zoom lens with a focal length of 12.5 to 75 mm connected to a CCD camera (Basler acA2040-180 km/kc, pixel size 5.5 µm, resolution of 2048 × 2048 pixels, sampling rate of 180 fps) and processed by a personal computer with an image processor. Yarn hairiness refers to the number of fibers projecting out of the yarn core, was identified as an important indicator affecting yarn properties, weaving efficiency, and fabric appearance. In this study, the yarn hairiness index was included as a requirement for yarn quality. The authors explained that the yarn could be divided into a yarn core and hairiness. Using the described method, they extracted the DIPU from the yarn core based on pixel connectivity for each image, thus enabling the determination of hairiness pixels using the image subtraction method (see Figure 74). Figure 74 - a) New foreground image, b) yarn core, c) hairiness [83]. The authors did not propose specific improvements or future work for this project. They simply stated that the proposed method and its corresponding detection system facilitate automation and industrialization in online yarn detection. Several other authors have also worked in the field of yarn image processing but without introducing significant innovations compared to previous research efforts [11], [85], [86]. Chapter 3 – Literature Review ___________________________________________________________________________ 79 3.1.2 System B – Study by Noman Haleem et al. (2021) A system was also developed by Noman Haleem and colleagues in 2021 [87], hereinafter referred to as System B. In their work, the authors emphasized the potential of conducting yarn testing in realtime mode to minimize process latency, provide rapid insights into yarn quality, and enhance the production of high-quality yarn. They pointed out that while capability-based uniformity test equipment and emerging image-based uniformity test systems are widely used in offline settings, there is still a lack of a suitable online system for assessing yarn quality under typical industrial conditions. As a response, the authors of this project proposed an online uniformity test system designed specifically for detecting neps. To achieve this, they harnessed image processing techniques and computer vision. The system they developed (Figure 75) directly captures yarn images on a rotating frame and employs the Viola-Jones object detection algorithm [88] for real-time nep defect detection. The validation of the nep detection algorithms, along with a comparison of the new method to an existing uniformity test equipment in terms of nep count, demonstrated a reasonable level of defect detection accuracy and promising potential for applications in the yarn spinning industry. These results were further validated by comparing them with USTER existing commercial uniformity testing equipment (TESTER). The image collection process (see Figure 75) utilized a Basler 1440-220 μm digital camera from Germany, equipped with a 50 mm lens. Figure 75 - An example of a yarn image with a nep defect obtained using the image acquisition system [87]. The authors leveraged three computer vision models (referred to A, B, and C), all based on the machine learning Viola-Jones algorithm [88]. These models were developed using the OpenCV computer vision library in the Python programming language. They essentially functioned as image classifiers, with variations in the input data used during their training phase. The training data comprised positive and negative images in a 2:1 ratio, and these images were resized by factors of 0.2 and 0.1, respectively, to Chapter 3 – Literature Review ___________________________________________________________________________ 80 optimize the training process. Each model underwent ten training cycles. Each model was trained using a specific type of image data: original images of yarns with neps, synthetic images of neps, or a combination of both. This approach allowed them to explore different aspects of nep detection and optimize the models. After training, each model was validated using a separate dataset containing images of yarns with and without neps. This validation process assessed the models' ability to accurately classify images as either positive (with neps) or negative (without neps) Figure 76 showcases a set of concatenated images of yarns exhibiting the nep defect, acquired using the proposed image acquisition configuration during the yarn spinning process. Additionally, the authors noted that in their analysis of a series of continuous yarn shape images, it was crucial for the field of view to be substantial enough to accommodate the lateral oscillations of the yarn during the spinning process. Figure 76 - An example of a yarn image with a nep defect obtained using the image acquisition system [87]. Figure 77 presents the models confusion matrices which provided insights into the models' performance in terms of true positives, true negatives, false positives, and false negatives. Chapter 3 – Literature Review ___________________________________________________________________________ 81 Figure 77 - Confusion matrices that offer insights into the performance of the model system [87]. For future research directions, the authors suggested the following [87]: • Acquisition of high-quality online yarn images through an imaging system with ultra-low exposure time, combined with external lighting and an appropriate optical setup; • Expansion of the online yarn uniformity measurement system to detect other types of yarn defects, along with necessary additions to the image processing module to create a comprehensive yarn quality testing solution suitable for industrial environments. • Improvement of Training Models: The authors suggested adjusting training strategies to enhance nep detection accuracy. Additionally, increasing the dataset size could improve model performance, while exploring alternative object detection approaches could optimize results. • Expansion of the online uniformity measurement system: The authors believe that in the future a system for identifying additional defects in the yarn in addition to neps could be developed. They also suggest that image processing modules be modified to encompass new defect detection requirements. • Finally, the authors propose an investigation into discrepancies compared to the Uster Tester 3 machine: Conduct further studies to understand and address significant differences between online and traditional testing methods, improving testing methodologies to ensure alignment and accuracy in quality assessment. 3.1.3 System C – Study by Adel El-Geiheini et al. (2020) Concerning the application of artificial intelligence for modeling yarn tensile properties, Adel ElGeiheini and colleagues in 2020 [89], hereinafter referred to as System C, emphasized that the utilization Chapter 3 – Literature Review ___________________________________________________________________________ 88 machine. 10. Analysis Limitations: None of the systems perform spectral analysis on the yarn based on image processing, and only systems B and D use artificial intelligence for yarn defect detection. In the context of deep learning techniques, a thorough literature review revealed no instances of systems or prototypes employing deep learning, specifically the YOLOv5s6 algorithm, for the characterization of yarn hairiness. 3.3 Critical review analysis In the fabric defect detection domain, a multitude of solutions has emerged, capitalizing on enhanced iterations of the YOLOv3, YOLOv4, and YOLOv5 algorithms. These studies consistently exhibit improved metrics, underscoring the advancements in this field [75], [93], [94], [95], [96]. Nevertheless, after conducting a review of the existing literature, no instances were identified where systems or prototypes employed deep learning, specifically the YOLOv5s6 algorithm, for the characterization of yarn hairiness [75], [93], [94], [95], [96]. In the field of yarn analysis, advances driven by image processing and artificial intelligence have revolutionized the assessment of the quality and characteristics of yarns. While prior research has explored the integration of artificial intelligence for fabric analysis, the intricate application of such technology in yarn analysis has remained a relatively uncharted territory. Following this path, the subsequent analysis aims to provide a critical review of the various yarn analysis systems presented in the previous section, highlighting the distinctive features and contributions of each, while also emphasizing existing gaps and what should be done in this regard so that a proposed system can be created that bring all these gaps together, namely: • System A [83]: • Does Not Obtain: Yarn twist orientation, yarn twist step, linear mass, mean deviation U (%), coefficient of variation CV (%), hairiness mean deviation UH (%), hairiness standard deviation sH (%), classification of hairiness types (loop/protruding fibers), spectral analysis. • Limited in Obtaining: Number of cables, number of loose fibers, volume, non-destructive attributes, use of deep learning techniques for yarn defect detection. • System B [87]: • Does Not Obtain: Yarn twist orientation, yarn twist step, linear mass, mean deviation U Chapter 3 – Literature Review ___________________________________________________________________________ 89 (%), coefficient of variation CV (%), hairiness coefficient H, hairiness mean deviation UH (%), hairiness standard deviation sH (%), classification of hairiness types (loop/protruding fibers), spectral analysis, thin and thick places defects. • Limited in Obtaining: Number of cables, number of loose fibers, volume, non-destructive attributes. • System C [89]: • Does Not Obtain: Yarn twist orientation, yarn twist step, mean deviation of mass U (%), hairiness coefficient H, hairiness mean deviation UH (%), hairiness standard deviation sH (%), classification of hairiness types (loop/protruding fibers), Thin, Thick and Nep defects, spectral analysis based in image processing, yarn diameter, linear mass, use of deep learning techniques for yarn defect detection. • Limited in Obtaining: Number of cables, number of loose fibers, volume, non-destructive attributes. • System D [91]: • Does Not Obtain: Yarn twist orientation, yarn twist step, mean deviation of mass U (%), hairiness coefficient H, hairiness mean deviation UH (%), hairiness standard deviation sH (%), classification of hairiness types (loop/protruding fibers), spectral analysis based on image processing, yarn diameter, volume. • Limited in Obtaining: Number of cables, number of loose fibers, volume, non-destructive attributes. The analysis of the characteristics obtained by Systems A, B, C and D elucidates a critical perspective on the existing systems and prototypes for the yarns analysis. Each system presents a mix of strengths and weaknesses, highlighting the complex nature of this field. Although System A present advances in certain parameters, such as mechatronic development and image analysis for defect detection, their limitations in several critical measurements cannot be ignored. Conversely, System B employs innovative approaches like artificial intelligence (AI) and image processing for defect detection, yet it also falls short in essential areas, lacking comprehensive parameters necessary for holistic yarn analysis. Similarly, systems C and D introduce innovative aspects such as AI for defect detection, but they also fail in crucial areas, lacking comprehensive parameters essential for holistic yarn analysis. These gaps collectively indicate the inadequacy of any single system to comprehensively address all aspects of yarn analysis, especially the incorporation of deep learning techniques, image Chapter 3 – Literature Review ___________________________________________________________________________ 90 processing and computer vision for defect detection and obtaining a majority of yarn parameters, which are lacking in most systems. A critical analysis of these systems highlights a notable discrepancy: none alone provide a holistic solution for yarns analysis. The absence of a developed system that incorporates all attributes leads to an incomplete understanding of yarn quality, making comprehensive assessments and quality control in textile production processes difficult. Furthermore, the limitations of each system highlight the gap in the industry search for a comprehensive solution. The absence of spectral analysis based on image processing, the inability to measure yarn twist orientation, and the lack of certain yarn quality parameters continue to be unresolved challenges in all systems. This gap highlights the need for a comprehensive prototype that not only fills these gaps but also addresses the deficiencies present in each system. In the search for an effective and comprehensive solution, it becomes increasingly evident that a prototype capable of assimilating the strengths of Systems A, B, C and D and, at the same time, mitigating their limitations is necessary. Such a prototype should incorporate mechatronic development, incorporate non-destructive attributes, encompass a wide range of yarn parameters for measurement, and leverage advanced technologies such as deep learning for defect detection. This prototype aims to fill the gap in yarn analysis methodologies, enabling a deeper understanding of yarn quality and promoting advancements in textile manufacturing processes. In the context of deep learning techniques, from the authors knowledge literature review revealed no instances of systems or prototypes employing deep learning, specifically the YOLOv5s6 algorithm, for the characterization of yarn hairiness. Following this, the proposed system in this thesis, which responds to all these gaps mentioned above and which justified its creation, is described in the following chapters, and serves as a catalyst for innovation in yarn analysis and establishing the introduction of deep learning for yarn quality assessment with the potential of allowing production optimization. Summary of the chapter: This chapter provided an extensive review of the literature referring to yarn analysis systems, with a specific focus on the utilization of image processing, computer vision, and artificial intelligence techniques. Then a comparative analysis is presented to provide an overview of the strengths and limitations of the systems presented. The chapter concludes with a critical review and analysis of the discussed literature, offering an impartial assessment of the advantages and limitations of different yarn analysis systems. 4. IMAGE PROCESSING AND COMPUTER VISION SYSTEM FOR YARN ANALYSIS Overview This chapter details the image processing techniques utilized in the computer vision system dedicated to yarn analysis. It includes segmentation algorithms, feature extraction methods, and pattern recognition techniques tailored for this purpose. Additionally, the hardware setup for image acquisition is discussed, outlining the specifications of the imaging devices used. Furthermore, the chapter describes the interface software developed to facilitate user interaction with the system. 4.1 Introduction 4.2 Image pre-processing 4.3 Obtaining the yarn core 4.4 Isolation of hairiness 4.5 Isolation of particles 4.6 Image capture software and hardware 4.7 Developed interface Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 92 4.1 Introduction This chapter outlines the image processing techniques employed in the development of algorithms focused on examining yarn characteristics to determine its quality. Key parameters include linear mass, specific volume, defects in the yarn, and its level of hairiness. Additionally, the hardware used to obtain images is presented, and the interface software developed is described. 4.2 Image pre-processing As it will be described in Figure 100 of the flowchart in subchapter 4.7, after image capture the first step employs the conversion of the image to grayscale. This step is necessary even in the case of using a camera that captures images only in grayscale due to the format in which the camera native software stores the images. When these images are subsequently opened in the "Visual Studio" software, there is an imperfect conversion of pixel values, which requires the grayscale transformation. After converting the image to grayscale (Figure 84a), the linearization (thresholding) of the image is initiated. This process is employed to transform the image into a monochromatic format with the aim of distinguishing the background from the yarn. The OTSU linearization method was employed, as depicted in Figure 84b. Figure 84 – a) A yarn in grayscale (left); b) Image after linearization using the OTSU algorithm (right) [97]. In the next step, a Gaussian filter is applied to the previous image (Figure 85). This filter introduces a blurring effect to the image, and its application aims at noise removal. This approach will enable the subsequent image transformations to be more successful in isolating or eliminating desired details from the image. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 93 Figure 85 - a) Gray scale image (left); b) Image with Gaussian filter (right) [97]. 4.3 Obtaining the yarn core To facilitate the analysis of the yarn core exclusively, it was necessary to perform morphological operations to eliminate the yarn hairiness, potential artifacts in the image, and loose fibers. The opening morphological operation was employed, which involves the application of two consecutive morphological operations using the same structural element as a reference. First, an erosion operation is applied to the image in question using a structural element, followed by a dilation operation on the same image using the same structural element. These two consecutive operations are referred to as an "opening." In terms of the structural element, also known as the kernel, a 16 × 16 matrix was utilized, where all its elements have a value of 1 [97]. The results of this operation are depicted in Figure 86. Figure 86 - a) Image prior to the application of the opening morphological operation (left); b) Image subsequent to the implementation of the opening morphological operation (right) [97]. This procedure allows to begin to discern the core of the yarn, which is now clearly separated from all other details in the initial image. However, as can be observed, there are still some remnants of details beyond the core, particularly the starting points of the yarn's fuzz. For this reason, it is necessary Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 94 to eliminate these remnants. To accomplish this, a morphological closing process is applied. This process smoothens the yarn ends, and in certain cases where traces of yarn fuzz are still visible, it removes any remaining yarn traces outside the core. Figure 87a) provides an example of the outcome. Figure 87 - a) Image after the closing process (left); b) Isolated core image (right) [97]. In this image, the isolation of the core can be considered complete. Consequently, it becomes possible to calculate all the desired parameters and characteristics. However, for future operations, as well as for visual and computational consistency, a final inverse "Thresholding" process is advisable. This process serves solely to invert the colors and, consequently, the pixel values. The result of this transformation is illustrated in Figure 87b. With the core completely isolated, it can begin the extraction of parameters from the yarn. The first and most evident parameter is the yarn diameter, which can be determined by counting the white pixels in a column of the image. This process is repeated for all columns, and in the end, the average value is obtained to ensure that it is not influenced by the presence of thick or thin places. In the case of video analysis, there is an important differentiation to highlight. For each image, the yarn diameter is measured in only one column of the image. This is done to reduce measurement redundancy, avoiding measuring the same section multiple times. Additionally, it saves analysis time and computational resources. The diameter measured in this manner is in pixel units, which, on its own, may not be practical unless converted to a real-world unit, in this case, millimeters. To convert from pixels to millimeters, the actual length of the yarn in the image was measured (see Appendix A – Mathematical Calculation of the Real Image Size and Lens Magnification Factor). With the real image size known, it is possible to convert a pixel measurement to millimeters using Equation 28. 𝒅𝒎𝒎 =𝒅𝒑×𝒉 𝟏𝟒𝟒𝟎 (28) Where: • dmm and dp correspond to the diameters in millimeters and in pixels, respectively; Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 95 • h corresponds to the actual height of the image, and 1440 corresponds to the number of pixels present per column. This process can be repeated for determining thick or thin places of the yarn and for detecting neps. Initially, the total number of diameter samples is divided by the desired number of divisions, meaning that the entire analyzed yarn is divided into the sections defined by the user. Next, the average diameter value is calculated for each of these sections, and this value is compared to the overall average diameter. The tolerance value is used as a reference point, from which a section is considered either thick or thin. In contrast, for neps detection, the dimensional tolerance value is constant and set at 200%, meaning that sections with a diameter value above double the average are considered a nep. A higher number of divisions results in greater resolution in detecting dimensional defects in the yarn. All tests aim to achieve a minimum resolution of one sample per every 3 millimeters of yarn, as is used in USTER machines. The second and third parameters to calculate are linear mass and yarn volume, respectively. Referring to their respective equations, it can be seen that the calculation of linear mass and volume depends on the length of the yarn per sample, porosity, material density, and the unknown variable is the diameter. Finally, with all the basic yarn parameters calculated, it is possible to proceed to the statistical results related to mass, including the mean deviation and the coefficient of variation. 4.4 Isolation of hairiness To initiate the process of isolating hairiness, it will return to the end of the common branch (Figure 100), specifically to the image treated with the Gaussian filter (Figure 85b). This image is initially subjected to another Gaussian filter. The reason for this is that the contour detection process that follows is particularly sensitive to image noise, and to best detect the overall silhouette of hairiness, it is necessary to remove as much noise as possible. This process is not part of the common branch because, in tests, it was observed that a heavier Gaussian filter would compromise other results, specifically in particle detection (Figure 86a) and core isolation. After obtaining an image with reduced noise, the Canny algorithm is applied to perform a contour detection process. The resulting image will be a silhouette of the yarn, as shown in Figure 88. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 96 . Figure 88 - Isolated outlines of the yarn. After observation, both the yarn's silhouette and that of its hairiness are delineated. The goal from here is to remove the white pixels in this image, which correspond to the yarn's core, in order to isolate its hairiness. To achieve this, the aim is to obtain an image that contains only the core's pixels, which will be used for subtraction. Fortunately, one of the previously obtained images can serve this purpose. Since the first branch of this methodology was focused on isolating the yarn's core, its final image (Figure 86b) can be used for this purpose. However, this image alone cannot be used directly for the subtraction process, as image subtraction subtracts the value of each pixel pair. In the case of subtracting two white pixels (both with a value of 255) or two black pixels (both with a value of 0), the result would be positive. But subtracting a white pixel from a black pixel (0 – 255) would result in a negative value. For this reason, first it was needed to obtain only the pixels that are common to both the isolated core image and the yarn contours image. It was used a logical "AND" operation, which retains the pixel values that are the same in both images, setting the others to 0, which, in this case, represents a black pixel. An example of the result of this operation can be seen in Figure 89. Figure 89 - Contours of the isolated core. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 97 Analyzing the figure, the resulting image from the logical operation (Figure 89) isolates the core's contour pixels. Using this image then is performed the subtraction operation between the yarn contours image (Figure 88). The resulting image obtains the hairiness isolated (Figure 90). Figure 90 - Contours of the isolated hairiness. With the hairiness isolated it is possible to extract parameters related to the hairiness. To determine the hairiness coefficient (Equation 5), two parameters from the image are needed to know: the yarn's length in the image and the length of the hairiness. The length of the hairiness can be estimated by counting the white pixels in the image and using the established pixel-to-millimeter conversion. However, it is important to note that the images obtained do not contain pixels exclusively from the yarn's hairiness, as the outer core contours are still present. To obtain the actual value of the hairiness length in the image, subtracting twice the yarn length from the pixel count is necessary to eliminate the length measured by the core's contour pixels. The second parameter to determine is the number of loose fibers from the core, in other words, the number of loose hairs strands from the core. To calculate this parameter, the count of white pixels at a specific distance from the core of the yarn is performed in a row of the image. This count is done both above and below the yarn. To perform this operation, it is necessary to first identify the upper and lower boundaries of the yarn core. Referring to Figure 86b, which shows the isolated core, one can analyze column by column to determine the start and end points of the core. With this information it becomes possible to identify the row that approximately contains the core midpoint. This information allows for counting the number of loose fibers located at a certain distance, both above and below the core. It is important to note that since the processed image contains the contours of the hairiness, counting 2 pixels is equivalent to the presence of only one loose end fiber. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 104 2. Specific hairiness data: • Hairiness index; • Number of loose fibers (this parameter is calculated only in image analysis, as it becomes inconsistent in video analysis due to the movement of the yarn). 3. Faults data and statistical parameters: • Thin places; • Thick places; • Neps; • Mean mass deviation (U%); • Coefficient of mass variation (CV%); • Hairiness standard deviation (sH%) (calculated for videos/image sequences only, as a single image does not provide enough data for an exact result). 4. Video and frame processing: • Total video frame; • Processed frames; • Frames with error. Having presented all the program input and output parameters, it will now be described the methodology used to obtain the desired results. While image and video analysis methodologies differ due to the former focusing on single images and the latter involving multiple images, the methodology for analyzing a single image remains quite similar. The primary distinction lies in the repetition of processes for each image within a video during video analysis, with results aggregated at the end. Given the diverse range of data required from a single image, there is no single sequence of processes that provides all the necessary information. To obtain the three different types of images, namely: • an image with an insulated yarn core; • an image with isolated hairs; • an image with isolated winding particles. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 105 A multifaceted approach involving various image processing techniques is employed. The following flowchart (Figure 100) illustrates this multifaceted approach used to generate each of the three image types. Figure 100 – Algorithm Analysis Methodology Flowchart [92]. With this tool it is possible to present information in a legible and intuitive way for the user. Chapter 4 – Image Processing and Computer Vision System for yarn analysis ___________________________________________________________________________ 106 Summary of the chapter: This chapter provided a comprehensive perspective on the elements involved in developing image processing algorithms. These algorithms are designed to evaluate yarn quality by analyzing its attributes, such as linear mass, specific volume, yarn defects, and the level of hairiness. 5. DEEP LEARNING TECHNIQUES FOR YARN HAIRINESS ANALYSIS Overview The following chapter describes the methods employed for the detection and classification of hairiness in yarn using a new improved YOLOv5s6 convolutional neural network (CNN). In this chapter, a comprehensive breakdown of the YOLOv5s6 detection method is provided, elucidating its constituent blocks and the enhancements that have been integrated. A concise overview of the Bot-Transformer module, MHSA (Multi-Head Self-Attention) blocks, and the C2f (coarse-to-fine) module, all integral components of the optimized YOLOv5s6 Hairiness algorithm, is presented. Additionally, the activation functions employed in both the default YOLOv5s6 algorithm, and the variant utilized in the optimized YOLOv5s6-Hairiness algorithm are detailed, along with the rationale behind these selections. 5.1 YOLOv5s6 Network – Proposed Architecture 5.2 Bot-Transformer Module & MHSA Block 5.3 C2f Module 5.4 Hyperparameters 5.5 Activation function GeLU Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 108 5.1 YOLOv5s6 Algorithm – Proposed Architecture Among the various versions of YOLOv5, the YOLOv5s6 architecture was chosen for the study [75], [76], [77]. Several preliminary tests were carried out, and it was observed that this model outperformed other algorithm variants in terms of overall results. The decision to opt for the YOLOv5s6 version instead of YOLOv8 (most recent version) was driven by several factors [102]: 1. Performance superiority: Through preliminary testing, it was evident that the YOLOv5s6 model outperformed other algorithm variants, including YOLOv8, in terms of overall results. 2. Priority on processing speed: Given the emphasis on speed, particularly crucial in applications requiring instant detection and classification of yarn fibers, YOLOv5s6 remarkable frames per second (FPS) rate made it the preferred choice. 3. Utilization of GPU (Graphics Processing Unit) resources: Operating in an environment with GPU access, YOLOv5s6 emerged as the optimal selection due to its optimized performance for GPUs. Leveraging GPU resources significantly accelerated the process of fiber detection in yarn. 4. Balance between accuracy and speed: While YOLOv8 offers marginal higher accuracy, YOLOv5s6 maintains a competitive equilibrium between accuracy and speed. This balance aligns with the requirements of the developed application, necessitating both accuracy and swift response times. 5. User-friendliness: YOLOv5s6 stands out as the more user-friendly option compared to YOLOv8. Built on the PyTorch framework, it offers seamless utilization and deployment for developers, enhancing ease of use during application development and deployment processes. Recognizing the hairiness detection performance of the YOLOv5s6 model, an enhanced version named YOLOv5s6-Hairiness was developed. This improved model incorporates a Bot transformer, a C2f (Coarse-to-Fine) module, and a GeLU (Gaussian Error Linear Unit) activation function, as depicted in Figure 101. Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 109 Figure 101 - Improved architecture based on YOLOv5s6 – YOLOv5s6-Hairiness [76]. The improved YOLOv5s6-Hairiness model incorporates three key modifications when compared to the original/default YOLOv5s6: • In the original YOLOv5s6 architecture, the last C3_1 layer in the Backbone, positioned just before the standard SPPF (spatial pyramid pooling fast) module, has been replaced by the C2f module, which is a feature borrowed from the more advanced YOLOv8 object detection model [103]. This change enhances the algorithm performance and optimization for its specific neural structure; • Within the Neck portion of the architecture, the previous C3_1_F module has been replaced by the Bot-transformer module. This adjustment serves to enhance the model's efficiency in detecting looped fibers within the captured images; • The CBS (Conv + BatchNorm + SiLU) module, which plays a fundamental role in the architecture by performing tasks like downsampling, dimensionality augmentation and reduction, normalization, and applying non-linearities to feature maps, has undergone a transformation. It has been replaced by the CBG (Convolutional, Batch Normalization, GeLU) module, introducing Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 110 a new GeLU activation function instead of the original SiLU (Sigmoid Linear Unit). This block consists of a Convolutional layer, a Batch Normalization layer, and the GeLU activation function (Figure 102). Figure 102 - The CBG module in the optimized YOLOv5s6-Hairiness [104]. This CBG block is utilized in multiple other blocks (Figure 103) and in two distinct bottleneck blocks (Figure 104). Figure 103 - The CBG module employed in various other blocks [104]. Figure 104 - The CBG module is used in two different types of bottleneck blocks [104]. The C3 module plays a vital role in feature extraction, comprising three CBG modules and several stacked Bottleneck blocks. The notation C3_x indicates the presence of x stacked Bottleneck blocks. In Figure 105, the feature map is divided into two branches after entering the C3 module. One branch traverses the CBG and Bottlenecks, while the other solely passes through the CBG. Subsequently, the two branches are concatenated and pass through another CBG module. Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 111 Figure 105 - Two distinct types of CSP blocks (C3) [104]. In each Bottleneck block, two CBG modules are incorporated. The first module conducts a 1x1 convolution, reducing the channel count by half, while the second module performs a 3x3 convolution, doubling the number of channels. The initial reduction in dimensionality aids in enhancing the understanding of feature information by the convolutional kernel, while the subsequent increase in dimensionality facilitates the extraction of more detailed features. To mitigate the vanishing gradient problem, a residual structure is employed, ensuring the addition of the input and output. The "vanishing gradient problem" is a well-recognized challenge in training deep neural networks, particularly those with numerous layers [76], [77], [105]. This issue arises during backpropagation when gradients diminish significantly as they propagate backward through multiple layers. Consequently, this results in slow or no learning in the early layers, rendering the training of deep networks arduous. Several techniques have been developed to address the vanishing gradient problem [76], [77], [105]: 1. Activation Functions: Utilizing activation functions like ReLU promotes the flow of gradients, alleviating the vanishing gradient problem; 2. Batch Normalization: The use of batch normalization normalizes layer inputs, stabilizing gradient flow during training; 3. Skip Connections: The incorporation of skip connections directly connects earlier layers to later layers, facilitating smoother gradient flow and mitigating the vanishing gradient problem. By implementing these techniques, deep learning models can effectively train in deep networks with multiple layers, leading to enhanced convergence and improved performance. The primary function of the C3 module is to capture intricate patterns and details in images as the network's depth increases. At this stage, 3x3 convolutions are widely employed as they enable the neural network to learn richer and more abstract representations of the objects presented in the images. Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 112 When analyzing loop fibers and protruding fibers in yarns using the YOLOv5 model, each block assumes a specific role in enhancing the detection of these hairiness types: 1. Backbone: • Function: The Backbone is tasked with extracting relevant features from yarn images, encompassing low and mid-level information like edges, textures, and patterns in the yarn; • Impact on detection: An efficient backbone captures distinctive features of loop fibers and protruding fibers in yarn. This feature extraction process contributes to more accurate subsequent detection. 2. Neck: • Function: The Neck serves as an intermediate component that amalgamates information at various scales, enabling the model to detect objects of different sizes and contexts; • Impact on detection: Given the varying sizes of loop fibers and protruding fibers in yarn, the Neck facilitates the capture of contextual information across multiple scales. This is particularly advantageous for the accurate detection of smaller or more diffuse hairiness. 3. Head: • Function: The Head is responsible for object detection, generating predictions regarding bounding boxes and object classes within the yarn fibers, • Impact on detection: The Head empowers the model to pinpoint and classify loop fibers and protruding fibers. Bounding box predictions specify the location and size of the hairiness, while class predictions discern whether they are loop fibers or protruding fibers. Each block plays a pivotal role in the detection of loop fibers and protruding fibers, with the Backbone extracting pertinent features, the Neck consolidating information at diverse scales, and the Head executing the final detection. This comprehensive approach provides the requisite information for precise hairiness identification. Furthermore, in the optimized algorithm, the C2f module was introduced, a novel element derived from YOLOv8, to replace the C3_1 module in the Backbone section of the architecture. The C2f module integrates concepts from the C3 module and ELAN (Efficient Layer Aggregation Network) [104], [106]. Figure 106 illustrates the two distinctions between the C2f and C3 modules. Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 113 Figure 106 - Difference between C3_x (top) and C2f module (bottom) [104]. The disparity between the C3 and C2F blocks can be delineated based on their structure, objective, and application [104], [106]: 1) Structure: C3 Block: The C3 block serves as a streamlined iteration of the CSP block utilized in YOLOv4 [104], [106]. It comprises a sequence of 1x1 and 3x3 convolutional layers, followed by stacked Bottleneck modules. A key feature of the C3 block is the utilization of a skip connection to amalgamate the original features with those acquired within the block. C2F Block: The C2F block, denoted as the CSP2 (Cross Stage Partial Network 2) block, extends the original CSP block found in YOLO [104], [106]. It introduces a novel split in the input, creating two distinct pathways (main and secondary). The secondary pathway traverses through additional convolutional Bottleneck modules. Subsequently, the output from these Bottleneck modules is concatenated with the output from the main pathway, leading to the final convolutional layers. 2) Objective: C3 Block: The primary aim of the C3 block is to provide a lighter and more efficient alternative to the CSP block, optimizing computational performance while preserving object detection accuracy. C2F Block: In contrast, the C2F block is specifically designed to enhance the capacity for capturing information at varied resolution levels. By partitioning the input into two pathways and incorporating Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 120 Within the context of the proposed optimized algorithm YOLOv5s6-Hairiness, the placement of the C2F block in the YOLOv5s6 architecture can exert a significant influence on the detection of loop fibers and protruding fibers within yarns. In the YOLOv5s6-Hairiness algorithm, the C2F block was positioned before the SPPF block. The C2F block's function lies in its capacity to capture and fuse information at diverse resolution levels, while the SPPF block is responsible for feature fusion through spatial pyramids, thus augmenting the feature representation without the loss of crucial contextual information. Regarding its impact on the detection of loop fibers and protruding fibers, introducing the C2F block prior to the SPPF block enabled the model to effectively grasp essential contextual and spatial information across different scales. This capability is particularly advantageous in the identification of intricate objects like loop and protruding fibers within yarns. 5.4 Hyperparameters In the specific context of detecting loop fibers and protruding fibers in yarns, significant changes were implemented in high-level hyperparameters. The modified hyperparameters [107] encompassed "anchor_t" and "scale" with the following values: anchor_t: 5.0 • The value anchor_t: 5.0 designates the threshold employed for assigning anchor boxes to objects in the images, set at 5.0; • Anchor boxes serve the purpose of predicting the position and size of objects detected by the model. Anchors whose dimensions are roughly five times larger than the standard anchor dimensions are exclusively considered when anchor_t is set to 5.0; • This adjustment increased the model's capability to select larger and more suitable anchors for the detection of large or unusually sized objects within the images. This was especially relevant in the case of object classes that significantly exceeded the typical anchor dimensions of the YOLOv5s6 model; • The elevation of this parameter helped diminish the occurrence of false positive detections, a valuable characteristic when the training data is prone to noise or complex backgrounds, as is the case with loop fibers; • In the particular context of detecting loop fibers and protruding fibers in yarns, this adaptation was instrumental for accurately capturing structures that span extensive areas and exhibit varied Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 121 sizes. For instance, loop fibers or protruding fibers that extend across substantial regions of the image could be more effectively detected, given that the images had a resolution of 1280x1280 pixels. scale: 2.0 • A scale of 2.0 was employed, which translated to resizing the images to approximately +/-200% to 300% beyond or below the standard image resolution; • The increase in scale resulted in larger images, which proved to be advantageous for the detection of intricate details in loop fibers and protruding fibers within yarns; • It is essential to highlight that resizing images to such an elevated scale substantially augmented the computational demands, necessitating additional memory and processing resources; • These adjustments played a pivotal role in achieving accurate and comprehensive detection. Given that the images were of 1280x1280-pixel resolution, signifying a larger scale, the model's capacity to identify these structures with greater reliability was significantly enhanced. The selection of the specific values for "anchor_t" and "scale" was contingent on the unique characteristics of the yarn dataset and the nature of the loop fibers and protruding fibers that were the target of detection. 5.5 Activation function GeLU The choice of the GeLU activation function was performed after a series of tests and a thorough mathematical analysis, taking into account the following considerations [108]: • Capturing Complex Non-linearities: Loop fibers and protruding fibers exhibit a diverse range of shapes and textures, rendering the detection task intricate. The GeLU function, distinguished by its heightened non-linearity compared to SiLU, is adept at capturing intricate and subtle relationships among the patterns inherent in the fibers and their visual characteristics. This enhancement substantially augments the model's capacity to acquire the capability to discern various types of yarn hairiness; • Reduction of Vanishing Gradients: Deep neural networks can confront the challenge of vanishing gradients, particularly when employing the SiLU activation function, characterized by derivatives that approach zero for large or negative inputs. GeLU, on the other hand, boasts smooth derivatives across all points, effectively mitigating the issue of vanishing gradients and instilling greater stability into the training process; Chapter 5 – Deep learning techniques for yarn hairiness analysis ___________________________________________________________________________ 122 • Improved Training Stability: GeLU, which approximates the cumulative Gaussian distribution function, offers an additional layer of training stability. This stability, in turn, furnishes the neural network with the capacity to fine-tune its weights more precisely, ultimately enhancing its ability to detect loop fibers and protruding fibers; • Superior Performance Metrics: GeLU consistently delivered superior results in metrics when compared to other activation functions, a detailed exposition of which will be presented in chapter 6. Taking these factors into account, GeLU emerged as the most adequate for the activation function in the proposed optimized YOLOv5 algorithm. Summary of the chapter: The chapter presented a comprehensive exploration of crucial components in the enhanced YOLOv5s6 Hairiness algorithm, focusing on the Bot-Transformer Module, MHSA Block, and C2f Module. It explores their integration and operational mechanisms, showcasing how these elements enhance the neural network's capacity in intricate tasks such as object detection and instance segmentation. Additionally, it provides insights into the rationale behind choosing the GeLU activation function, highlighting its benefits in capturing complex patterns, reducing vanishing gradients, ensuring training stability, and delivering superior performance metrics. Overall, this comprehensive analysis contributes significantly to the improved detection and classification of hairiness in yarn using the optimized YOLOv5s6 algorithm. 6. MECHATRONIC PROTOTYPE DEVELOPMENT Overview This chapter presents the design of a specialized prototype tailored to meet industry demands. The development process and key considerations are presented, encompassing mechanical and automation domains. This prototype automatically unwinds and winds yarn while allowing it to move swiftly past a camera lens within predefined tolerances and controlled lighting conditions 6.1 Mechatronic systems design 6.2 Requirements 6.3 Methodology 6.4 Functions of a yarn testing equipment 6.5 Automatic system design 6.6 Controller 6.7 Subsystems and kinematic chains 6.8 Development of the automation control program for the yarn unwinding, conduction, and winding System 6.9 Maintenance and remote access to the mechatronic prototype 6.10 Tests and results 6.11 Comparison between the developed prototype and the other systems cited in state of art 6.12 Critical Review analysis Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 124 6.1 Mechatronic systems design The VDI 2206 standard is a German technical standard that provides guidelines and methodologies for the development of mechatronic systems. It addresses issues related to the development process, from identifying requirements to validating and verifying the final system. The VDI 2206 standard is known for its systematic approach and the "V" model that describes the phases of mechatronic product development. It is used by engineers and industry professionals to ensure efficiency and quality in mechatronic system development [109]. The objective of VDI 2206 is to provide a support methodology for the development of mechatronic systems. The main characteristic of VDI is to be the process, method, and tool for the initial phase of development, focused on system design. The result of the system design is the mechatronic system concept (the solution is established after verification and validation). This VDI guide is a supplement to the VDI 2221 standard and VDI 2422 [109]. According to the VDI 2206 standard, the development of a mechatronic product follows a "V" model. In the model depicted in Figure 106, customer needs are transformed into requirements. Subsequently, these requirements are transformed into specifications, as they define the metric characteristics that the mechatronic product must meet. After this, the next necessary step is System Design [109]. In the development of mechatronic systems, it is essential to follow an appropriate methodology to ensure efficiency and user satisfaction. The VDI 2206 standard [110] offers an important methodological framework that encompasses the multidisciplinary inherent in such systems, integrating electronic, mechanical, and information technology components. At the heart of this methodology is the V-Model, a framework that proposes a sequence of steps for mechatronic systems design. However, it is important to note that not all automation projects strictly fit into a multidisciplinary approach. Following this, the standard related to requirements, their categories, and their significance in shaping systems that align with user and stakeholder needs will be discussed. This exploration aims to establish a robust groundwork for the effective development of automation systems [110]. The VDI 2206 standard provides a methodological framework for the multidisciplinary development of mechatronic systems, particularly those that involve electronic and mechanical components in Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 125 symbiosis with information technology. This methodology revolves around the central element known as the V-model (Figure 109) [110]. Figure 109 – V-Model for the Design of a Mechatronic System - VDI 2206 [110]. The V-Model proposes a sequence of steps for the design of mechatronic systems, which are [110]: 1. Requirements - This is the first stage, consisting of describing the system to be obtained in the form of requirements against which the resulting system will be subsequently evaluated; 2. Conceptual System Design - This stage allows for the creation of a multidisciplinary solution concept that leads to the required characteristics of the system to be obtained. To achieve this, the main function is divided into sub-functions, for which individual solution or operation principles are found. Their performance is evaluated within the context of the system across various domains; Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 126 3. Specific System Design - This step involves developing sections of the global solution related to various areas involved (Mechanical Engineering, Electronics, Computer Science, among other domains); 4. Integration - The results of various components of the global developed solution within each domain are integrated to obtain an overall solution for the system; 5. Property Verification - This stage involves comparing the properties of the developing system with those defined for it and is associated with the concepts of verification and validation. Verification involves formally analyzing whether the system meets all proposed requirements. Validation involves informally checking if the system is suitable, typically done in an informal manner. The property verification process continues throughout the project, although it is formally located during system integration. This can be done virtually, physically, or in a mixed approach; 6. Modeling and Model Analysis - This stage runs parallel to all steps except requirements and involves assessing the system's properties using models or computer-based simulation tools. Another methodology followed for this mechatronic prototype, is based on the rational methodology presented by Nigel Cross, which is similar to but with some improvements over the methodology outlined in the German engineering standard VDI 2221 [110]. The general methodology presented in VDI 2221 consists of seven phases: 1. Clarification of objectives; 2. Function structure; 3. Solution principles and their combinations; 4. Division into modules; 5. Development of the main modules; 6. Development of all modules; 7. Production preparation. However, the "Requirements" stage remains of fundamental importance, as it serves as the foundation of the project. Requirements are the building blocks upon which the automation system will Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 127 be constructed, and their proper definition is essential to ensure that the system meets the needs and expectations of the users. This is where a journey to understand requirements begins, a topic of utmost significance in the development of the automation system for winding and unwinding yarn [111]. The "Requirements" serves as a means to ensure that the system meets the proposed needs and expectations, constituting the primary input for the project. In this regard, the following concepts related to this theme are presented [111]. 6.2 Requirements In the context of developing the mechatronic prototype, this subchapter plays a central role. The objectives set for this research involve the design of a mechatronic device capable of efficiently winding and unwinding the yarn, enabling in-depth analyses of various raw materials. To achieve this challenge, it is crucial to establish a comprehensive understanding of the requirements underlying this mechatronic system. The "Requirements" thoroughly explores the properties that the device must possess to meet the needs and constraints of users, highlighting the distinction between explicit and implicit, functional, and non-functional requirements. These requirements form the foundation upon which the mechatronic prototype will be designed and constructed, ensuring it fulfills its purpose of efficiently winding and unwinding the yarn in perfect alignment with the thesis objectives. Requirements are the properties that a particular system must possess to meet the needs and constraints imposed by the user. Specifically applied to the field of automation/software, they can be divided into two categories: explicit and implicit, and functional and non-functional. Explicit requirements are the elements directly requested by clients, while implicit requirements are those included by the development team based on their technical knowledge, even if they have not been mentioned by the client. Functional requirements are elements associated with the system's functionality, while nonfunctional requirements relate to aspects of its performance. Functional requirements should be coherent and cover all the system's needs, characterizing its response to certain stimuli without being bound to design and implementation aspects in order to maximize the range of solutions to explore. Non-functional requirements, by their nature, do not influence the system's functionalities to be developed and can be classified into eight types: appearance, usability, performance, operability Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 128 (characteristics that enable the system to operate properly in its environment), maintenance and support, security, cultural and political aspects, and legal aspects. 6.3 Methodology Following the goal tree method, the main objectives considered for the project were six [97]: 1. Safety; 2. Simplicity; 3. Reliability; 4. Portability; 5. Versatility; 6. Lower cost. From the objectives, it was then necessary to quantify and define some parameters that the machine must meet to achieve the goals. The boundaries set, according to the method of performance specifications, were as follows in Table 3. Table 3 - Project Performance Specifications Objective Values Reason for choosing this option Analyze various type of yarn materials The machine should be compatible with cotton and polyester; Breaking Force ≥ 1 N; Compatible with variations in breaking tension ≤ |10|%. Versatility: Analyzing various types of yarn materials, diameters, and bobbin dimensions allows the prototype to be versatile, accommodating different materials and sizes commonly used in textile industries. This versatility enables broader applications and compatibility with diverse yarn types, catering to a wide range of production needs. Analyze various type of yarn diameters Compatible with diameters ≤ 2 mm. Compatible with variations in diameter ≤ |10|%. Analyze various bobbin dimensions Compatible with bobbins with an outer base diameter ≤ 160 mm and height ≤ 200 mm. Adherence to Speed ≤ 4.2 m/s with a maximum Accuracy and Precision: Adherence to speed and Chapter 6 – Mechatronic Prototype Development ___________________________________________________________________________ 129 speed tolerances variation of 0.5%. position tolerances ensures that the machine operates with high accuracy and precision. This is crucial in yarn analysis, as precise measurements are necessary for quality control and consistency in production. Adherence to position tolerance Deviation in position ≤ 1 mm. Machine and user safety In case of an emergency, the machine should stop within ≤ 1 second. Safety: Prioritizing machine and user safety is fundamental. Ensuring that the machine stops promptly in case of emergencies, as well as having simple and intuitive setup procedures, contributes to a safe working environment and reduces the risk of accidents or mishaps during operation. Simple, intuitive, and quick setup Setup time ≤ 2 minutes. Durability and Maintenance: Reducing wear of moving components is essential for the durability and longevity of the machine. By minimizing wear and tear, maintenance requirements are reduced, leading to cost savings, and ensuring consistent performance over time. Reduced wear of moving components Optimal operating conditions ≥ 10 km. Quality Control: Avoiding yarn breakage is critical for maintaining the integrity of the yarn being analyzed. It ensures that the process remains uninterrupted and prevents material waste, allowing for accurate analysis without compromising quality. Avoid yarn breakage Avoid ≤ 5 yarn breaks per 10 km. Reduced Mass Mass ≤ 30 kg. Portability and Practicality: Reducing the mass and volume of the machine makes it more portable and space efficient. This aspect is valuable for industries where space is limited or where mobility and flexibility are necessary, enabling easier transportation and installation in different settings. Reduced Volume Volume ≤ 0.275 m3. Lower Cost 1.300,00€ (only hardware and software) This cost advantage allows for more accessible initial investment, making the prototype financially more feasible for testing or development purposes compared to the significantly higher cost of the USTER tester 3. 232 [37] P. K. Sahoo, S. Soltani, and A. K. C. Wong, “A survey of thresholding techniques,” Computer Vision, Graphics and Image Processing , vol. 41, no. 2. 1988. doi: 10.1016/0734189X(88)90022-9. [38] “OpenCV: Miscellaneous Image Transformations.” Accessed: Oct. 16, 2023. [Online]. Available: https://docs.opencv.org/4.x/d7/d1b/group__imgproc__misc.html [39] Muthukrishnan, “Otsu’s method for image thresholding explained and implemented,” Muthukrishnan . 2020. [40] L. Torok, “Método de Otsu,” Instituto de Computação – Universidade Federal Fluminense (UFF). [41] Emanuele. Trucco and Alessandro. Verri, “Introductory techniques for 3-D computer vision,” p. 343, 1998, Accessed: Oct. 16, 2023. [Online]. Available: https://www.researchgate.net/publication/220692096_Introductory_techniques_for_3D_computer_vision [42] A. C. Bovik and S. T. Acton, “Basic Linear Filtering with Application to Image Enhancement,” in The Essential Guide to Image Processing , 2009. doi: 10.1016/B978-0-12-374457-9.00010-X. [43] L. Ding and A. Goshtasby, “On the canny edge detector,” Pattern Recognit , vol. 34, no. 3, 2001, doi: 10.1016/S0031-3203(00)00023-6. [44] S. Anand, T. Thivya, and S. Jeeva, “Edge Detection using Directional Filter Bank,” Int J Appl Inf Syst , vol. 1, no. 4, pp. 21–27, May 2012, doi: 10.5120/ijais12-450162. [45] “OpenCV: Morphological Transformations.” Accessed: Oct. 17, 2023. [Online]. Available: https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html [46] Matlab, “ einforcement Learning for Engineers, Part 1: What Is einforcement Learning? - ouTube,” Matlab ouTube Channel. [47] J. de A. B. Araujo-Filho, A. N. Assunção Júnior, M. A. Gutierrez, and C. . Nomura, “Artificial Intelligence and Cardiac Imaging: We need to talk about this,” ARQUIVOS BRASILEIROS DE CARDIOLOGIA - IMAGEM CARDIOVASCULAR , vol. 32, no. 3, 2019, doi: 10.5935/23188219.20190034. [48] P. Manickam et al. , “Artificial Intelligence (AI) and Internet of Medical Things (IoMT) Assisted Biomedical Systems for Intelligent ealthcare,” Biosensors , vol. 12, no. 8. 2022. doi: 10.3390/bios12080562. [49] “Algoritmos de clustering - Visão geral.” Accessed: Oct. 17, 2023. [Online]. Available: https://isolution.pro/pt/t/machine-learning-with-python/clustering-algorithmsoverview/algoritmos-de-clustering-visao-geral 233 [50] Isha Salian, “Difference Between Supervised, Unsupervised, & einforcement Learning | NVIDIA Blog,” nVidia. [51] A. M. Obeso, J. Benois-Pineau, M. S. García Vázquez, and A. Á. . Acosta, “Visual vs internal attention mechanisms in deep neural networks for image classification and ob ect detection,” Pattern Recognit , vol. 123, 2022, doi: 10.1016/j.patcog.2021.108411. [52] J. edmon, S. Divvala, . Girshick, and A. Farhadi, “ ou only look once: Unified, real-time object detection,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2016. doi: 10.1109/CVPR.2016.91. [53] “C_5. olo V4 explained in full detail-EN - Deep Learning Bible - 4. Object Detection - Eng.” Accessed: Oct. 17, 2023. [Online]. Available: https://wikidocs.net/167833 [54] “Non-maximum Suppression (NMS). A Technique to remove duplicates and… | by Sambasivarao. K | Towards Data Science.” Accessed: Oct. 17, 2023. [Online]. Available: https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c [55] “Ob ect Tracking with DeepSO T and OLO-NAS: A Practitioner’s Guide | Deci.” Accessed: Oct. 17, 2023. [Online]. Available: https://deci.ai/blog/object-tracking-with-deepsort-and-yolo-naspractitioners-guide/ [56] “Git ub - ultralytics/yolov5 at blog.roboflow.com.” Accessed: Oct. 17, 2023. [Online]. Available: https://github.com/ultralytics/yolov5?ref=blog.roboflow.com [57] “What are Anchor Boxes in Ob ect Detection?” Accessed: Oct. 17, 2023. [Online]. Available: https://blog.roboflow.com/what-is-an-anchor-box/ [58] “What is OLOv5? A Guide for Beginners.” Accessed: Oct. 17, 2023. [Online]. Available: https://blog.roboflow.com/yolov5-improvements-and-evaluation/ [59] “A 5-layer dense block with a growth rate of k = 4. Each layer takes all... | Download Scientific Diagram.” Accessed: Oct. 17, 2023. [Online]. Available: https://www.researchgate.net/figure/A5-layer-dense-block-with-a-growth-rate-of-k-4-Each-layer-takes-all-preceding_fig2_306885833 [60] “EfficientDet: Scalable and Efficient Ob ect Detection”, Accessed: Oct. 17, 2023. [Online]. Available: https://github.com/google/automl/tree/ [61] L. Zhu, J. Zhang, and C. Jia, “An Improved OLOv5-based Method for Surface Defect Detection of Steel Plate,” in Proceedings - 2022 Chinese Automation Congress, CAC 2022 , 2022. doi: 10.1109/CAC57257.2022.10055055. 234 [62] J. Zhang, J. Zhang, K. Zhou, . Zhang, . Chen, and X. an, “An Improved OLOv5-Based Underwater Object-Detection Framework,” Sensors , vol. 23, no. 7, 2023, doi: 10.3390/s23073693. [63] “Confusion Matrix for our Multi-Class Machine Learning Model | by Joydwip Mohajon | Towards Data Science.” Accessed: Oct. 17, 2023. [Online]. Available: https://towardsdatascience.com/confusion-matrix-for-your-multi-class-machine-learning-modelff9aa3bf7826 [64] “Mean Average Precision (mAP): Definition, Metrics, and Applications in Computer Vision | Label our Data.” Accessed: Oct. 17, 2023. [Online]. Available: https://labelyourdata.com/articles/mean-average-precision-map [65] “Mean Average Precision (mAP) Explained: Everything ou Need to Know.” Accessed: Oct. 17, 2023. [Online]. Available: https://www.v7labs.com/blog/mean-average-precision [66] O. Cronie, M. Moradi, and C. A. N. Biscio, “A cross-validation-based statistical theory for point processes,” Biometrika , 2023, doi: 10.1093/biomet/asad041. [67] N. Ghavami et al. , “Automatic segmentation of prostate M I using convolutional neural networks: Investigating the impact of network architecture on the accuracy of volume measurement and MRI-ultrasound registration,” Med Image Anal , vol. 58, 2019, doi: 10.1016/j.media.2019.101558. [68] D. Niu, . Liang, . Wang, M. Wang, and W. C. ong, “Icing forecasting of transmission lines with a modified Back Propagation Neural Network-Support Vector Machine-Extreme Learning Machine with Kernel (BPNN-SVM-KELM) based on the variance-covariance weight determination method,” Energies (Basel) , vol. 10, no. 8, 2017, doi: 10.3390/en10081196. [69] A. M. Andrew, “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods,” Kybernetes , vol. 30, no. 1. 2001. doi: 10.1108/k.2001.30.1.103.6. [70] V. . Phung and E. J. hee, “A igh-accuracy model average ensemble of convolutional neural networks for classification of cloud image patches on small datasets,” Applied Sciences (Switzerland) , vol. 9, no. 21, 2019, doi: 10.3390/app9214500. [71] Q. en, M. Li, and S. an, “Tectonic discrimination of olivine in basalt using data mining techniques based on ma or elements: a comparative study from multiple perspectives,” Big Earth Data , vol. 3, no. 1, 2019, doi: 10.1080/20964471.2019.1572452. [72] X. Zhang and C. A. Liu, “Model averaging prediction by K-fold cross-validation,” J Econom , vol. 235, no. 1, 2023, doi: 10.1016/j.jeconom.2022.04.007. 235 [73] E. Sevinç, “An empowered AdaBoost algorithm implementation: A COVID-19 dataset study,” Comput Ind Eng , vol. 165, 2022, doi: 10.1016/j.cie.2021.107912. [74] “ OLO Algorithm for Ob ect Detection Explained [ Examples].” Accessed: Oct. 17, 2023. [Online]. Available: https://www.v7labs.com/blog/yolo-object-detection [75] Z. Guo, . uang, Z. Shan, J. uang, Z. ou, and W. Li, “A digital implantation system for Zdirection yarn of three-dimensional preform based on flexible oriented woven process,” Eng Appl Artif Intell , vol. 116, 2022, doi: 10.1016/j.engappai.2022.105385. [76] A. Z. Sellam et al. , “Deep Learning Solution for Quantification of Fluorescence Particles on a Membrane,” Sensors , vol. 23, no. 4, 2023, doi: 10.3390/s23041794. [77] . u and Z. Zhu, “Sim-YOLOv5s: A method for detecting defects on the end face of lithium battery steel shells,” Advanced Engineering Informatics , vol. 55, 2023, doi: 10.1016/j.aei.2022.101824. [78] . K. Jung and G. S. Choi, “Improved OLOv5: Efficient Ob ect Detection Using Drone Images under Various Conditions,” Applied Sciences (Switzerland) , vol. 12, no. 14, 2022, doi: 10.3390/app12147255. [79] Z. ang, “Activation Function: Cell ecognition Based on oLov5s/m,” Journal of Computer and Communications , vol. 09, no. 12, 2021, doi: 10.4236/jcc.2021.912001. [80] S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural Networks , vol. 107, 2018, doi: 10.1016/j.neunet.2017.12.012. [81] J. Doherty, B. Gardiner, E. Kerr, N. Siddique, and S. S. Manvi, “Comparative Study of Activation Functions and Their Impact on the OLOv5 Ob ect Detection Model,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2022. doi: 10.1007/978-3-031-09282-4_4. [82] “About hyperparameters · ultralytics/yolov5 · Discussion #7319.” Accessed: Oct. 22, 2023. [Online]. Available: https://github.com/ultralytics/yolov5/discussions/7319 [83] Z. Li, P. Zhong, X. Tang, . Chen, S. Su, and T. Zhai, “A New Method to Evaluate arn Appearance Qualities Based on Machine Vision and Image Processing,” IEEE Access , vol. 8, 2020, doi: 10.1109/ACCESS.2020.2972967. [84] J. Sun, J. Jia, C. K. Tang, and . . Shum, “Poisson matting,” in ACM SIGGRAPH 2004 Papers, SIGGRAPH 2004 , 2004. doi: 10.1145/1186562.1015721. 236 [85] . Sun, Z. Li, . Pan, J. Zhou, and W. Gao, “Measurement of long yarn hair based on hairiness segmentation and hairiness tracking,” Journal of the Textile Institute , vol. 108, no. 7, 2017, doi: 10.1080/00405000.2016.1240144. [86] J. Jing, M. uang, P. Li, and X. Ning, “Automatic measurement of yarn hairiness based on the improved M M F segmentation algorithm,” Journal of the Textile Institute , vol. 109, no. 6, 2018, doi: 10.1080/00405000.2017.1368106. [87] N. aleem, M. Bustreo, and A. Del Bue, “A computer vision based online quality control system for textile yarns,” Comput Ind , vol. 133, 2021, doi: 10.1016/j.compind.2021.103550. [88] A. Soleimanipour and G. . Chegini, “A vision-based hybrid approach for identification of Anthurium flower cultivars,” Comput Electron Agric , vol. 174, 2020, doi: 10.1016/j.compag.2020.105460. [89] A. El-Geiheini, S. ElKateb, and M. R. Abd-Elhamied, “ arn Tensile Properties Modeling Using Artificial Intelligence,” Alexandria Engineering Journal , vol. 59, no. 6, 2020, doi: 10.1016/j.aej.2020.07.049. [90] B. Yang et al. , “Parameter identification of proton exchange membrane fuel cell via LevenbergMarquardt backpropagation algorithm,” Int J Hydrogen Energy , vol. 46, no. 44, 2021, doi: 10.1016/j.ijhydene.2021.04.130. [91] M. R. Abd-Elhamied, W. A. Hashima, S. ElKateb, I. Elhawary, and A. El-Geiheini, “Prediction of Cotton arn’s Characteristics by Image Processing and ANN,” Alexandria Engineering Journal , vol. 61, no. 4, 2022, doi: 10.1016/j.aej.2021.08.057. [92] F. Pereira et al. , “Intelligent Computer Vision System for Analysis and Characterization of arn Quality,” Electronics (Switzerland) , vol. 12, no. 1, 2023, doi: 10.3390/electronics12010236. [93] X. Luo, Z. Cheng, Q. Ni, . Tao, and . Shi, “Defect detection algorithm for fabric based on deformable convolutional network,” Textile Research Journal , vol. 93, no. 9–10, 2023, doi: 10.1177/00405175221143742. [94] . Jin and Q. Niu, “Automatic Fabric Defect Detection Based on an Improved OLOv5,” Math Probl Eng , vol. 2021, 2021, doi: 10.1155/2021/7321394. [95] G. Lin, K. Liu, X. Xia, and . an, “An Efficient and Intelligent Detection Method for Fabric Defects based on Improved OLOv5,” Sensors , vol. 23, no. 1, 2023, doi: 10.3390/s23010097. [96] . Zuo and . Du, “Detection method of heterotropic fiber based on improved OLOv5,” MATEC Web of Conferences , vol. 358, 2022, doi: 10.1051/matecconf/202235801008. 237 [97] P. Caldas, F. Sousa, F. Pereira, . Lopes, and J. Machado, “Automatic system for yarn quality analysis by image processing,” Journal of the Brazilian Society of Mechanical Sciences and Engineering , vol. 44, no. 11, 2022, doi: 10.1007/s40430-022-03875-3. [98] “Automação Indústrial | OM ON, Portugal.” Accessed: Nov. 26, 2023. [Online]. Available: https://industrial.omron.pt/pt/home [99] “STC Series (USB3.0 Series) USB3.0 Series/Features | OM ON Industrial Automation India.” Accessed: Nov. 26, 2023. [Online]. Available: https://www.omronap.co.in/products/family/3874/ [100] “Visual Studio: IDE and Code Editor for Software Developers and Teams.” Accessed: Nov. 27, 2023. [Online]. Available: https://visualstudio.microsoft.com/ [101] “OpenCV - Open Computer Vision Library.” Accessed: Nov. 27, 2023. [Online]. Available: https://opencv.org/ [102] E. Casas, L. Ramos, E. Bendek, and F. Rivas-Echeverria, “Assessing the Effectiveness of OLO Architectures for Smoke and Wildfire Detection,” IEEE Access , vol. 11, 2023, doi: 10.1109/ACCESS.2023.3312217. [103] Z. Zhang, “Drone-YOLO: An Efficient Neural Network Method for Target Detection in Drone Images,” Drones , vol. 7, no. 8, 2023, doi: 10.3390/drones7080526. [104] G. u and X. Zhou, “An Improved OLOv5 Crack Detection Method Combined with a Bottleneck Transformer,” Mathematics , vol. 11, no. 10, 2023, doi: 10.3390/math11102377. [105] A. M. oy and J. Bhaduri, “DenseSP -YOLOv5: An automated damage detection model based on DenseNet and Swin-Transformer prediction head-enabled YOLOv5 with attention mechanism,” Advanced Engineering Informatics , vol. 56, 2023, doi: 10.1016/j.aei.2023.102007. [106] A. Srinivas, T. . Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck transformers for visual recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2021. doi: 10.1109/CVPR46437.2021.01625. [107] H.-P.-T. Van and V.-D. oang, “Insulator Detection in Intelligent Monitoring Based on olo Family and Customizing yperparameters,” Journal of Technical Education Science , no. 75A, 2023, doi: 10.54644/jte.75a.2023.1308. [108] B. Singh, S. Patel, A. Vi ayvargiya, and . Kumar, “Analyzing the impact of activation functions on the performance of the data-driven gait model,” Results in Engineering , vol. 18, 2023, doi: 10.1016/j.rineng.2023.101029. 238 [109] I. Graessler and J. entze, “The new V-Model of VDI 2206 and its validation das Neue V-Modell der VDI 2206 und seine Validierung,” At-Automatisierungstechnik , vol. 68, no. 5, 2020, doi: 10.1515/auto-2020-0015. [110] J. Gausemeier and S. Moehringer, “VDI 2206A New Guideline for the Design of Mechatronic Systems,” IFAC Proceedings Volumes , vol. 35, no. 2, pp. 785–790, Dec. 2002, doi: 10.1016/S1474-6670(17)34035-1. [111] C. C. Insaurralde and A. Zoitl, “System requirements in industrial automation: A eview of Modeling Methodologies for Control Software Architectures,” in IEEE International Conference on Industrial Informatics (INDIN) , 2013. doi: 10.1109/INDIN.2013.6622947. [112] “(PDF) indin2012 JMachado ESeabra FINAL mai 2012.” Accessed: Oct. 29, 2023. [Online]. Available: https://www.researchgate.net/publication/258764397_indin2012_JMachado_ESeabra_FINAL _mai_2012 [113] G. Barbieri and D. A. Gutierrez, “A GEMMA-GRAFCET Methodology to enable Digital Twin based on Real-Time Coupling,” in Procedia Computer Science , 2021. doi: 10.1016/j.procs.2021.01.122. [114] E. S. J. Machado, “Tratamento da Paragem de Emergência – Caso de Aplicação,” Guimarães, 2007. [115] . amanathan, “The IEC 61131-3 programming languages features for industrial control systems,” in World Automation Congress Proceedings , 2014. doi: 10.1109/WAC.2014.6936062. [116] “USE ’S MANUAL S SMAC CP Series CP2E CPU Unit Software”. [117] F. Pereira, L. Magalhães, A. A. Santos, A. F. da Silva, K. Antosz, and J. Machado, “Development of an Automated Wooden andle Packaging System with Integrated Counting Technology,” Machines 2024, Vol. 12, Page 122 , vol. 12, no. 2, p. 122, Feb. 2024, doi: 10.3390/MACHINES12020122. [118] F. Pereira and C. Felgueiras, “Learning Automation from emote Labs in igher Education,” in ACM International Conference Proceeding Series , 2020. doi: 10.1145/3434780.3436689. [119] “Ewon Flexy | IIoT Data Gateway | Ewon.” Accessed: Nov. 27, 2023. [Online]. Available: https://www.ewon.biz/products/ewon-flexy [120] F. Pereira and J. Machado, Sistemas de Acesso Remoto a Máquinas e Processos Industriais – Parte I , 2023rd ed., vol. 22. Porto: Publindústria, 2023. 239 [121] J. G. Pinto, J. Monteiro, . Vasconcelos, and F. O. Soares, “A new system for direct measurement of yarn mass with 1mm accuracy,” in Proceedings of the IEEE International Conference on Industrial Technology , 2002. doi: 10.1109/ICIT.2002.1189336. [122] A. Sparavigna, E. Broglia, and S. Lugli, “Beyond capacitive systems with optical measurements for yarn evenness evaluation,” Mechatronics , vol. 14, no. 10, 2004, doi: 10.1016/j.mechatronics.2004.07.002. [123] F. Pereira, L. Pinto, J. Machado, F. Soares, . Vasconcelos, and V. Carvalho, “ arn airiness - Loop & Protruding Fibers Dataset,” vol. 1, 2023, doi: 10.17632/DKV6J6FW6C.1. [124] “ oboflow: Give your software the power to see ob ects in images and video.” Accessed: Oct. 24, 2023. [Online]. Available: https://roboflow.com/ [125] D. Mahto and S. C. adav, “Emotion prediction for textual data using GloVe based eBiCuDNNLSTM model,” Multimed Tools Appl , 2023, doi: 10.1007/s11042-023-16062-w. [126] I. S. Isa, M. S. A. osli, U. K. usof, M. I. F. Maruzuki, and S. N. Sulaiman, “Optimizing the yperparameter Tuning of OLOv5 for Underwater Detection,” IEEE Access , vol. 10, 2022, doi: 10.1109/ACCESS.2022.3174583. [127] “Tips for Best Training esults - Ultralytics OLOv8 Docs.” Accessed: Nov. 27, 2023. [Online]. Available: https://docs.ultralytics.com/yolov5/tutorials/tips_for_best_training_results/#modelselection [128] “Entenda distância focal, ângulo de visão e círculo de imagem – apenas imagens.” Accessed: Oct. 29, 2023. [Online]. Available: https://apenasimagens.com/pt/distancia-focal-circuloimagem/ APPENDIXES A.1 Mathematical Calculation of the Real Image Size and Lens Magnification Factor A.2 Full system controller specification A.3 Higher Hierarchy Grafcet - Coordination of Operating Modes A.4 Predefined screens for HMI installed on the machine body A.5 LabVIEW interface - Local computer A.6 Developed Work Dissemination Appendixes ___________________________________________________________________________ 241 A.1 Mathematical Calculation of the Real Image Size and Lens Magnification Factor Before any measurements can be made on a captured image, it is necessary to precisely know the dimensions of a reference object from which all other desired dimensions can be extrapolated. The best reference object is the pixel, as it is the elemental object of the image and has a simple geometric shape (a square). To obtain the real dimensions of a pixel, the aim is to determine the dimensions of the entire image. Then, it is simply a matter of dividing the width of the image by the number of pixels in a row of the image. To determine the dimensions of the image at the height of the yarn, the real dimensions of the image background were first calculated. Knowing the distance between the image background and the camera, as well as the distance between the camera and the yarn, allows for a comparative calculation to obtain the real length of the yarn in the image. This is possible because the area of the image section increases proportionally with the distance between the section and the camera lens, as demonstrated in Figure A.1-1. Figure A.1-1 - Field of View Visualization [128] To obtain the actual size of the image background, an image was captured with a background containing a known 2-millimeter square. Then, a simple rule of three was applied to find the total length of the image (Y), which is 4.23 millimeters. Knowing that the image dimension is 1440x1080 pixels, the height of the image can be directly calculated as 3.17 millimeters. With the size of the image background calculated, the distance between the image background and the camera (Z) and the distance between the camera and the yarn (B) were measured, resulting in 41.6 millimeters and 12.8 millimeters, respectively. With all these dimensions, a simple rule of three can be applied to calculate the actual length of Appendixes ___________________________________________________________________________ 248 Grafcet C Figure A.2-9 - Grafcet C - Test Mode Appendixes ___________________________________________________________________________ 249 Grafcet of Test Termination Mode Figure A.2-10 - Grafcet of Test Termination Mode – Part 1 Appendixes ___________________________________________________________________________ 250 Figure A.2-11 - Grafcet of Test Termination Mode – Part 2 Appendixes ___________________________________________________________________________ 251 Figure A.2-12 - Grafcet of Test Termination Mode – Part 3 Appendixes ___________________________________________________________________________ 252 Grafcet of Stop Mode Figure A.2-13 - Grafcet of Stop Mode – Part 1 Appendixes ___________________________________________________________________________ 253 Figure A.2-14 - Grafcet of Stop Mode – Part 2 Appendixes ___________________________________________________________________________ 254 Manual Mode Grafcet Grafcet A Figure A.2-15 - Manual Mode Grafcet – Grafcet A – Part 1 Figure A.2-16 - Manual Mode Grafcet – Grafcet A – Part 2 Appendixes ___________________________________________________________________________ 255 Figure A.2-17 - Manual Mode Grafcet – Grafcet A – Part 3 Figure A.2-18 - Manual Mode Grafcet – Grafcet A – Part 4 Grafcet C Figure A.2-19 - Manual Mode Grafcet – Grafcet C Appendixes ___________________________________________________________________________ 256 A.3 Higher Hierarchy Grafcet - Coordination of Operating Modes Figure A.3-1Higher Hierarchy Grafcet - Coordination of Operating Modes Grafcet of Test Mode Grafcet A Figure A.3-2 - Grafcet of Test Mode - Grafcet A – Part 1 Appendixes ___________________________________________________________________________ 257 Figure A.3-3 - Grafcet of Test Mode - Grafcet A – Part 2 Appendixes ___________________________________________________________________________ 264 Figure A.43 - Predefined screens for HMI installed on the machine body – Part 3 Appendixes ___________________________________________________________________________ 265 Figure A.44 - Predefined screens for HMI installed on the machine body – Part 4 Figure A.45 - Predefined screens for HMI installed on the machine body – Part 5 Appendixes ___________________________________________________________________________ 266 A.5 LabVIEW interface - Local computer Figure A.51 - LabVIEW interface - Local computer Appendixes ___________________________________________________________________________ 267 A.6 Developed Work Dissemination All contributions resulting from the developed work are described below: F.1. Journal publications [1] Filipe Pereira, Leandro Pinto, Filomena Soares, Rosa Vasconcelos, José Machado and Vítor Carvalho. A Novel Deep Learning-Based Yarn Hairiness Characterization Methodology Using an Improved YOLOv5 Algorithm. IEEE Access 2024. Submitted [2] Filipe Pereira, Leandro Pinto, Filomena Soares, Rosa Vasconcelos, José Machado and Vitor Carvalho, ONLINE YARN HAIRINESS - LOOP & PROTRUDING FIBERS DATASET, Data in Brief, 2024, 110355, ISSN 2352-3409, https://doi.org/10.1016/j.dib.2024.110355. [3] Filipe Pereira, Alexandre Macedo, Leandro Pinto, Filomena Soares, Rosa Vasconcelos, José Machado and Vitor Carvalho, Intelligent Computer Vision System for Analysis and Characterization of Yarn Quality, Electronics, 2023, 12 , 236. https://doi.org/10.3390/electronics12010236. F.2. Conference publications [1] Filipe Pereira, José Machado, Filomena Soares, Rosa Vasconcelos and Vitor Carvalho, Design of a Yarn Analysis Mechatronic Prototype for Textile Industry, DESIGNCOMMIT 2024, Braga, Portugal, 20-22 May 2024 (In press). [2] Ricardo Pinto, Filipe Pereira, Vitor Carvalho, Filomena Soares and Rosa Vasconcelos, Yarn linear mass determination using image processing: first insights, IECON 2019 - 45th Annual Conference of the IEEE Industrial Electronics Society, Lisbon, Portugal, 2019, pp. 198-203, doi:10.1109/IECON.2019.8926650. [3] Filipe Pereira, Vitor Carvalho, Rosa Vasconcelos, Filomena Soares, A Review in the use of Artificial Intelligence in Textile Industry, Innovations in Mechatronics Engineering – ICIENG, 2021, Lecture Notes in Mechanical Engineering, Springer, https://doi.org/10.1007/978-3-030-79168-1_34. Appendixes ___________________________________________________________________________ 268 F.3. Book publications [1] Filipe Pereira, Vitor Carvalho, Filomena Soares, Rosa Vasconcelos, José Machado, 6 - Computer vision techniques for detecting yarn defects, Textile Institute Book Series, Applications of Computer Vision in Fashion and Textiles, Woodhead Publishing, 2018, Pages 123-145, ISBN 9780081012178, https://doi.org/10.1016/B978-0-08-101217-8.00006-3. F.4. Dataset publication [1] Filipe Pereira, Leandro Pinto, José Machado, Filomena Soares, Rosa Vasconcelos and Vitor Carvalho, Yarn Hairiness - Loop & Protruding Fibers Dataset, Mendeley Data, 2023, V1, https://data.mendeley.com/datasets/dkv6j6fw6c/1