Exploring the impact of Deep Learning in Masked Facial Recognition

Exploring the impact of Deep Learning in Masked Facial Recognition By Muhammad Uzair Mughal (Registration No: 00000359599) Department of Electrical Engineering School of Electrical Engineering and Computer Science National University of Sciences and Technology (NUST) Islamabad, Pakistan (2024) Exploring the impact of Deep Learning in Masked Facial Recognition By Muhammad Uzair Mughal (Registration No: 00000359599) A thesis submitted to the National University of Sciences and Technology, Islamabad, in partial fulfillment of the requirements for the degree of Masters in Electrical Engineering Supervisor: Dr. Arbab Latif School of Electrical Engineering and Computer Science National University of Sciences and Technology (NUST) Islamabad, Pakistan (2024) DEDICATION This is dedicated to my parents for their endless efforts to help me accomplish my goals ACKNOWLEDGEMENTS I extend my gratitude to the Almighty ALLAH for granting me strength and endurance to complete my journey and formalities for my degree. This is indeed a new beginning for my lifelong learning in the ever so vast field of Machine Learning and Pattern Recognition. I would also like to acknowledge the efforts put in by my mother through all walks of my life, enabling me to be able to continue my education. I owe an immeasurable debt to my parents for their continued support, love and the sacrifices they made throughout. I am deeply thankful to my supervisor Dr. Arbab Latif for being patient and available to support me whenever I have asked for. His guidance in between the progress of this thesis helped me a lot in succeeding and eventually fulfilling the requirements. I would also like to thank my friends and colleagues for their unwavering support and kindness to motivate me through tough and hard times. Lastly, I appreciate the facilities provided to me by SEECS and NUST to excel in my career and the field of passion. It was because of SEECS and the courses offered to us that made me interested in Machine Learning and Deep Learning which paved way to involve in the rapidly evolving technologies revolving AI. Contents LIST OF TABLES V LIST OF FIGURES VII LIST of Symbols and Abbreviations VIII ABSTRACT X 1 Introduction 1 1.1 Problem Definition ............................. 3 1.2 Motivation .................................. 4 1.3 Proposed Solution .............................. 5 1.4 Research Methodology ........................... 6 1.5 Thesis Organization ............................. 7 2 Background Study 8 2.1 Facial Recognition vs Masked Facial recognition ............. 9 2.2 Survey of Architectures and Algorithms .................. 10 2.3 Face Mask utilization approach ...................... 11 2.3.1 Disguised Face Recognition using AlexNet + SVM ........ 11 2.3.1.1 Limitations of the architecture ............. 12 2.3.2 DeepMaskNet for Masked Facial recognition ........... 13 2.3.2.1 MDMFR Dataset ..................... 15 2.3.2.2 Limitations of the architecture ............. 16 2.3.3 PCA based Masked Facial Recognition .............. 16 2.3.3.1 Limitations of the architecture ............. 18 2.4 Face restoration approach for MFR via Generative Models ....... 19 2.4.1 GAN based Face Reconstruction for Masked-Face ........ 19 II 2.4.1.1 CycleGAN architecture for MFR ............ 20 2.4.1.2 Dataset and Evaluation Metric ............. 22 2.4.1.3 Limitations of the architecture ............. 22 2.4.2 Privacy Preserving Face Recognition using Masked Autoencoders 23 2.4.2.1 Datasets .......................... 24 2.4.2.2 Limitations of the architecture ............. 25 2.4.3 Masked Face Recognition with Generative Data Augmentation . 25 2.4.3.1 Proposed Dataset ..................... 26 2.5 MFR using Attention based models .................... 27 2.5.1 Cropping and attention based MFR ................ 27 2.5.1.1 Intra-channel attention .................. 27 2.5.1.2 Spatial attention ..................... 28 2.5.1.3 Datasets .......................... 29 2.5.1.4 Limitations of the architecture ............. 29 2.5.2 MFR using Deep Metric Learning ................. 30 2.5.2.1 MFR in static images .................. 30 2.5.2.2 MFR in video stream and video files .......... 31 2.5.2.3 Datasets .......................... 32 2.5.2.4 Limitations of the architecture ............. 33 2.6 Diverse Approaches for Masked Face Recognition ............ 34 2.6.1 Siamese Networks .......................... 34 2.6.1.1 Advantages of Siamese Networks ............ 36 2.6.1.2 Loss Functions ...................... 36 2.6.1.3 Limitations of the architecture ............. 38 2.6.2 Ensemble Models .......................... 39 2.6.2.1 Dataset .......................... 40 2.6.2.2 Limitations of the architecture ............. 40 2.7 Datasets in MFR .............................. 41 2.8 Post-processing for Model Deployment .................. 42 2.8.1 Model Quantization ......................... 43 2.8.1.1 Linear Quantization ................... 44 2.8.1.2 Post-training quantization vs Quantization Aware training .......................... 45 2.8.2 Model Pruning ........................... 46 2.8.2.1 Pruning Aware Training vs post-training model pruning 47 2.8.3 Knowledge Distillation ....................... 48 2.8.3.1 Adversarial Knowledge Distillation ........... 50 2.8.3.2 Multi-teacher Distillation ................ 50 3 Methods and Methodology 52 3.1 Overview of the MFR pipeline ....................... 52 3.2 Face Verification stage ........................... 54 3.2.1 Choice of the Face Verification block ............... 57 3.3 Inference time based architecture selector ................. 59 3.4 Denoising Diffusion Implicit Models .................... 59 3.4.1 Choice of the Generative Model .................. 61 3.4.1.1 DDIM vs Conventional Generative models ....... 61 3.4.1.2 DDIM vs Other variants ................. 62 3.4.2 Analysis of the Diffusion Implicit Models ............. 63 3.4.2.1 Forward Diffusion Process ................ 63 3.4.2.2 Reverse Diffusion Process ................ 65 3.4.2.3 Loss Function ....................... 67 3.4.2.4 Conditioned Image generation using DDIM . . . . . . 70 3.5 Choice of the backbone model ....................... 70 3.6 Attention based model ........................... 71 3.7 Matching the Feature Embedding from the database ........... 72 3.8 Hardware Acceleration ........................... 72 3.8.1 Jetson Xavier AGX ......................... 73 3.8.2 Time profilers ............................ 74 4 Results and Discussions 75 4.1 Face Localization stage ........................... 75 4.2 Generative Model stage ........................... 78 4.3 Attention-based model ........................... 81 5 Conclusion and Future Research 83 5.1 Future Research ............................... 84 List of Tables 2.1 Description of the AlexNet architecture [18] ............... 12 2.2 Gender based classification for MDMFR ................. 15 2.3 Classification of the masked images in MDMFR ............. 15 2.4 Execution time for different input image resolutions ........... 32 2.5 Summary of the common MFR benchmarking datasets ......... 41 2.6 Floating-Point Datatype Characteristics ................. 43 2.7 Bitwise allocation for floating point variants ............... 43 2.8 Comparison between Post-training Quantization and Quantizationaware Training ............................... 45 2.9 Comparison of Post-Model Pruning and Model Pruning During Training 47 3.1 Comparison of Face Detection Methods for Masked FR ......... 57 3.2 Key Features of Jetson Xavier AGX .................... 74 V List of Figures 1.1 A demonstration of various occlusion scenarios .............. 2 1.2 Outline of the proposed model architecture ................ 6 2.1 Masked version of the data sample from CelebA dataset ......... 9 2.2 MFR pipeline with robustness to background noise ........... 11 2.3 Architecture of the DeepMaskNet model ................. 14 2.4 PCA based Masked Facial Recognition Model .............. 17 2.5 CycleGAN architecture for Face restoration ............... 20 2.6 FaceMAE architecture for masked face reconstruction .......... 24 2.7 Statistics of the MFSR Dataset ...................... 26 2.8 Detailed architecture of the CBAM .................... 28 2.9 Modified convolution block in ResNet-50 using CBAM block ...... 29 2.10 Detailed architecture of FaceMaskNet-21 ................. 31 2.11 Weights and Bias sharing in Siamese Networks .............. 35 2.12 Architecture of the One-shot Siamese Network .............. 35 2.13 A demonstration of margin in the Triplet Loss Function ......... 38 2.14 Architecture of the Ensemble model .................... 39 2.15 Exponential Growth of the model size over time ............. 42 2.16 Mapping of FP32 to 8-bit integer ..................... 44 2.17 A post-training demonstration of model parameter pruning ....... 46 2.18 Post training model pruning vs Pruning aware model training ..... 48 2.19 Knowledge Distillation from teacher to the student network ....... 49 2.20 Multi-teacher knowledge Distillation ................... 51 3.1 Outline of the proposed model architecture ................ 53 3.2 Standard Convolution vs Deformable Convolutional Neural Network . . 55 3.3 A subset of images in the WIDER Face dataset ............. 56 3.4 Performance comparison of MTCNN vs RetinaFace on Masked Faces in the wild ................................... 58 VI might require a different approach to covered areas than when focusing solely on accuracy. However, it’s still important to examine and possibly test different methods for how the algorithm deals with both covered and uncovered parts of the face. This will form the basis of the analysis of the previously presented algorithms extending to our own proposed models and approaches to solve the problem. 1.1 Problem Definition Much of the work have been carried out on facial recognition before the outbreak of COVID-19. In most cases the focus was to ensure facial recognition with a clear face image which later transformed into a pose estimation problem under various scenarios including lighting changes, side facial view and blurred images. The main problem in this case was the discrepancy among the training data distribution and the test data. Many benchmark datasets have been published in this regard to ensure an exhaustive and a diverse representation of the facial images. Furthermore, techniques such as Data Augmentation and some other standard computer vision operations help to ensure the same data distribution for the training and the test dataset. Following this approach for data preprocessing and using the appropriate architecture, researchers have developed many robust facial recognition systems. They tend to perform well on possible pose estimation and lighting changes to the extent that they exceed the human level performance, which serves as a proxy for Bayesian error. Few of the renowned state-of-the-art models for unobscured facial recognition are FaceNet, Deep Face, DeepID, InsightFace, and MTCNN [5,6,7,8,9]. An additional contribution made in some of these models is their robustness towards the possible adversarial attack commonly known as "Face Spoofing". It helps in ensuring that the model is not fooled by an image of the face rather than the actual face itself. It is common particularly in sensitive matters such as surveillance and attendance systems where the chances of individuals trying to cheat the system are significantly higher. Although these Deep Learning models were remarkably exceptional in recognizing visible faces, they tend to struggle when they have been validated on the masked faces. This particular domain gained increased attention following the COVID-19 pandemic, and its utility can be expanded to include surveillance purposes, as well as in regions where certain female individuals are culturally mandated to wear a hijab. Expanding on this idea a model used in such applications needs to be further optimized both in terms of the architecture and the timing constraints for any good use. The work carried out in this thesis is regarding the timing optimization of the masked facial recognition while ensuring robustness in terms of the accuracy and the confidence of the model in both masked and unmasked cases. It can be considered as an elevation of the previously trained face recognition models but with a much wider scope of applications. To validate the best possible architecture’s performance for a real-time 3 application, our model has been deployed on the CUDA cores of an AI accelerated embedded hardware that is Jetson Xavier AGX [10]. It features CUDA and Tensor cores that allow embarrassingly parallel tasks such as a typical Neural Network to execute much more efficiently compared to a standard CPU operation. Moreover, the device itself is compact and robust enough to be considered for deployment in remote areas as an independent node or perhaps an integral part of a bigger IoT network. 1.2 Motivation The problem of face recognition has been researched in the past decade to the extent that it was thought to be completely solved. Many state of the art algorithms for face recognition and recognition have been developed with a focus on pose estimation and changes in the background including external lighting effects. The performance of state-of-the-art face recognition model such as VGGFace dropped when tested on the masked dataset by 40% that is from 82.1% on unmasked faces to 40.4% on masked dataset [4]. The same trend of decline is seen with respect to other models including FaceNet, DeepFace and DeepID. The motivation for this research can therefore be summarized as follows. •To improve the conventional facial recognition models in order to be able to adapt not only the varying pose estimation scenarios but also to cater occlusion. •The shallow masked facial recognition models such as the one cited in [11] that employs AlexNet as the backbone with the SVM classifier comes with the drawback that they are designed for closed class classification problems. These models tend to perform well in scenarios including attendance systems when there are fixed number of students or in the security related problems with access to limited number of individuals. The problem of open-class classification was not addressed which plays a crucial role in applications such as surveillance and automated ticketing systems. •MFR finds its application in the problems that requires real-time operation. No paper or research material was found to the best of our knowledge that had utilized any model post-processing techniques or optimizations to be able to deploy the model on the hardware. No or negligible discussions have been found in the papers with attention regarding the timing efficiency of the model. This can potentially be a serious bottleneck when trying to use such models in practice. •Some of the recently developed MFR papers have utilized generative models such as Auto encoders in [12] and custom made GAN in [13]. These models tend to approach the problem by treating the masked portion of the face as corrupted 4 pixels and therefore they reconstruct the face semantically to assume "no noise" facial image. One of the problems with these models is their time consumption and too much resource allocation is required to restore the image which is often not suited for real-time operation and certainly not to be able to deploy on a node where computational resources are also limited. •Another ethical aspect to take into consideration are those areas where it is forbidden to wear hijab for Muslim women as it hinders in their identity recognition. This problem can be solved if only for once their face can be recorded in the database and later they can be verified at security checkpoints without having to reveal their masks. 1.3 Proposed Solution To solve the problems briefly highlighted in the previous sections, we have reviewed both the generative models and the attention-based mechanisms. In the case of Generative models, we have used the novel approach of utilizing Diffusion Models. Unlike GANs, the image generation process is tractable, which paves the way for semantically correlated images. It is an absolute necessity for the masked facial recognition problem as the generated images are required not only to be accurate but also related to the facial features of the same person. Among the diffusion models themselves, more emphasis has been given to the DDIM than to their probabilistic counterparts. DDIM offers a better sampler, and we can always sample the image for different sampling paths with having the same Markov chain for the forward diffusion process. This is because, unlike DDPM, the focus of the DDIM is more on the denoising part of the image rather than actually modeling the probability distribution of the masked faces. On the other hand, the attention-based mechanism approach ensures better performance of our model by adaptively paying attention to the hidden and unoccluded parts of the face. The idea is similar to the optimization techniques used in famous optimizer such as RMSProp, Adam and some commonly known algorithms including L1 and L2 regularization. Focusing more on the visible facial areas while still acknowledging the obscured parts is crucial to retain information from the corrupted pixels in prior methods, without assimilating unnecessary patterns from the masked regions of the face. The algorithms are then validated on the mentioned Embedded AI accelerator board for the real-time computational complexity and the time consumption. The results are compiled and presented in this thesis as we move along. The chosen architecture is suited for real-time applications and is comparable with the known state-of-the-art algorithms in terms of accuracy and the time efficiency. A blue print of the model architecture has been shown below. 5 Figure 1.2: Outline of the proposed model architecture The proposed architecture works in stages to extract independent features for better performance. The first stage is the face verification process, which extracts the facial coordinates from external background noise. The mask detection stage follows it, a semantic segmentation for the face mask. The last stage is face recognition, which can be achieved by the DDIM or the attention-based mechanisms depending on the trigger signal. This signal is externally supplied at the time of inference depending upon the device to which it is subjected to operate. The analysis and the results for the best possible architecture based on the trade-off for accuracy and computational resources have been shared in the later parts of this thesis. 1.4 Research Methodology The adopted research methodologies aim to follow objectives: •To explore and research extensively on state-of-the-art facial and masked facial recognition model architectures. •In-depth analysis of the pros and cons of the architectures with respect to both the accuracy and the time efficiency. 6 •Exploring the Architecture of Jetson Xavier AGX for Streamlined Model Deployment Post-Training. •To develop a masked facial recognition pipeline that seamlessly identifies faces regardless of masks or background noise, rivaling the robustness of state-of-theart models. Furthermore, our focus lies on optimizing inference time to ensure swift real-time deployment. •Exploring the model optimization technique including model pruning, quantization, knowledge distillation, sparse and dynamic inference for better timing efficiency while maintaining the drop in accuracy within bounds •Deployment and validation of the MFR model on the hardware, post-optimization. The utilization of the CUDA and Tensor Cores needs to be optimized for the eloquent number of threads, blocks and proper allocation of the grids inside the GPU. 1.5 Thesis Organization The thesis in the coming chapters has been organized as follows: •In Chapter 02, all the available literature on the facial and masked facial recognition architectures has been compiled with a special focus on the time of inference and the accuracy of the model on state-of-the-art datasets. •Chapter 03 is dedicated to the extensive analysis and details of the implementation with and without the hardware. In this section, a comprehensive discussion has been carried out on the best possible architecture and the potential tradeoffs between accuracy and the efficient time of inference for real-time application. It includes all the details of the proposed pipeline, from the cleaning of the dataset to the model post-processing and deployment on the hardware. •Chapter 04 is dedicated to the discussion regarding the obtained results. •Chapter 05 serves as both the conclusion, summarizing the findings and implications of the study, and as a platform for outlining future research directions and potential advancements in the field. 7 Chapter 2 Background Study In computer vision, recognizing faces is a standout application. It works like our natural ability to tell people apart by their faces. Unlike natural language processing and time series forecasting, computer vision is an area where the accuracy of a model can often be fine-tuned to the point where it’s highly reliable for making decisions. It is usually because of the strong ability of humans to be good at image recognition. Therefore, it acts as the Bayes’ Optimal error and guides us nicely as to what factors can lead to better model output. It’s crucial to determine the optimal error for a specific problem. Without this estimate, it becomes challenging to assess whether we’re encountering issues like overfitting, underfitting, or a mismatch in probability distribution between the training and test datasets. Moving forward this information will be super helpful in optimizing our proposed model architecture for the benchmark datasets. The human level performance that can potentially serves as the Bayes’ optimal error is the recognition accuracy of an expert in the field [14]. For instance, in the case of masked facial recognition, humans tend to perform well to recognize faces that are partially occluded. It is worthwhile to note that for a human to recognize a person, it might also take into account other factors such as Gait analysis, speech recognition and psychological behavior. The ceiling on the optimal error should therefore be less than the average human recognition for a person. A good reference for the performance optimization is to compare the newly trained model with the previously known state-of-the-art models. The pursuit to automate facial recognition is not a new one. Many researchers have published their work in the past few decades even before the formal advent of Deep Learning and Neural Networks in practice. The improvements have been made in the early days to the algorithms so much that many of the recent algorithms utilize a hybrid model having both the Neural Networks followed by the classical algorithms for classification. In the coming section a detailed discussion will be carried out on the previously design architectures including their pros and cons. This extensive review of the literature on facial recognition will shape our approach of designing our own 8 algorithm that is both robust and efficient in terms of the inference time. 2.1 Facial Recognition vs Masked Facial recognition It is important to note the difference before analyzing the architectures for masked facial recognition. Before COVID-19, identity recognition primarily centered around facial recognition, requiring a clear, unobstructed view of the entire face. In MFR however, almost half of the face is hidden behind some object that can be a face mask, scarf, hat, or synthetic image inpainting. The differences can therefore be summarized as under. •Facial recognition with the face being fully visible is much easier as the architecture can be designed to extract features from the raw image followed by the classifier algorithm. It can be projected as the multi-class classification problem with the typical CNN + fully connected layers architecture. Other classifiers such as SVM or tree based classification schemes may as well be utilized depending upon the dataset. •The image below demonstrates the masked and the unmasked image of the same individual, in this case taken from CelebA dataset [15]. Figure 2.1: Masked version of the data sample from CelebA dataset 9 The masked version of the image has been created utilizing the dlib based features for capturing the necessary features and the face tilt. The implementation is carried out the with the help of an API known as "MaskTheFace" [16]. The highlighted mask represents the set of corrupted pixels that alters the overall data distribution, leading to a decrease in the model’s accuracy. The performance of the trained model on unmasked faces will therefore be sub optimal on masked faces due to the change in the data distribution of the training and the test data. •Apart from using the same model, there is a limitation of using the same architecture on masked face dataset. This is because once the face has been cropped out of the background noise, the image is only the relevant features. Therefore, the recognition could have been done easily by utilizing the embedding-based models for the open-end classification problem and CNN for the closed-end ones. In MFR the corrupted pixels either needs to be ignored, restored, or given lower weights by employing the attention-based mechanisms. •MFR might also pose an additional challenge due to the lack of the vast dataset compared to the unmasked faces. Although, the masked faces can be generated synthetically the distribution of the synthetic masked faces might differ significantly in comparison to the actual face masks. Based on the arguments stated above, MFR will be dealt with entirely differently from normal facial recognition, and it also states why the conventional facial recognition models underperform when tested on the masked dataset. The algorithms discussed in this case will mostly be for masked facial recognition, with any exceptions in the case of an idea borrowed from the facial recognition counterpart. 2.2 Survey of Architectures and Algorithms This section will explore the algorithms and architectures that have been documented for facial recognition in masked scenarios. Particular focus and attention will be given to the timing efficiency of the algorithms if documented. Their advantages and disadvantages will also be examined in challenging scenarios, such as pose variations, lighting conditions changes, and background noise. For ease of analysis, all the literature on this particular topic can be divided based on how they treat the corrupted pixels occluded by the face mask. Based on this, we can have three(3) different types of architectures, stated below: •Ignores the face mask and uses it as a helpful feature as in the case of visible faces. 10 •Restores the occluded faces by utilizing generative models. •Attention-based approaches partially utilize masked pixels and focus only on the visible part of the face. All three (3) approaches will be discussed subsequently, stating their pros and cons, which will form the basis of our improved architecture. 2.3 Face Mask utilization approach The common theme in these architectures is the utilization of face mask as the normal facial pixel to extract features from the raw image. The main difference in all these architectures is the use of feature extraction layers and their classifiers for closed-end classification problem. Some of the eminent algorithms have been discussed as under. 2.3.1 Disguised Face Recognition using AlexNet + SVM One of the common approaches for feature extraction is the use of transfer learning. The backbone model used in [11] was AlexNet that have been trained on the ImageNet dataset for a closed classification problem, having a total of 1000 different objects [17]. The architecture can be summarized as shown in the image below [11]. Figure 2.2: MFR pipeline with robustness to background noise 11 The pipeline suggested in the research provides a mechanism of removing the background noise by employing a face detection algorithm based on haar-cascade classifier. After cropping the face, the image is processed as a set of raw features devoid of noise, which are then extracted using AlexNet. Specifically, only the convolutional layers of AlexNet are employed for feature extraction, omitting the softmax layer due to discrepancies in the number of facial entities between the dataset used and the ImageNet dataset for training AlexNet. Following the extraction of embeddings by the AlexNet model, a multiclass SVM classifier is employed. AlexNet can be act as an effective tool for feature extraction from raw images. The architecture for for the backbone model used in this pipeline is a combination of the convolutional layers, maxpooling and the dropout layer. The detailed architecture for the model has been tabulated below. Table 2.1. Description of the AlexNet architecture [18] Layer # Filters/ Filter Size Stride Pad Size of Activation Neurons Feature Map Function Input - 227 ×227 ×3 Conv 1 96 11 ×11 4 Same 55 ×55 ×96 ReLU Pool 1 3×3227 ×27 ×96 Conv 2 256 5×51 Same 27 ×27 ×256 ReLU Pool 2 3×3213 ×13 ×256 Conv 3 384 3×31 Same 13 ×13 ×384 ReLU Conv 4 384 3×31 Same 13 ×13 ×384 ReLU Conv 5 256 3×31 Same 13 ×13 ×256 ReLU Pool 3 3×326×6×256 Dropout 1 rate = 0.5 6×6×256 While the neural network architecture may appear simplistic, its effectiveness is bolstered by the extensive training performed on the expansive ImageNet dataset. As a result, the model is adept at extracting sturdy features from images. Another important aspect of using the AlexNet is the extensive number of filters used in a layer. Overall, it has around 62.3 Million learnable parameters. Last but not the least, SVM was used for classification which can help in both the linear and non-linear classification problem. 2.3.1.1 Limitations of the architecture The limitations in the discussed pipeline are stated below. •The pipeline was initially crafted specifically for the closed-case classification issue. To accurately classify the identity of a new individual, rigorous retraining 12 2.4 Face restoration approach for MFR via Generative Models It has been seen in the previous category of approaches that the face mask was considered an essential part of the face itself. Therefore, the feature extraction layers were utilized on the entire occluded region. It is a significant assumption in Masked Facial Recognition since the mask or the occlusion holds no information for the person wearing it. A model that is robust to the external noise and performs significantly well on the diverse dataset remains a challenge face mask utilization approach. An alternative to the MFR is the use of generative models to restore a person’s facial pixels. It relies on the fact that the unoccluded faces have been recognized sometimes better than the human-level performance [21,22]. Therefore, if the face can be adequately restored, we will have the face recognized, closing the gap between the accuracy of the model and Bayes’ Optimal error [23,24,25]. Generative models, since the advent of GAN, have been the centre of attraction for generating images from noise based on the prompt. In MFR, these models can be utilized to restore the occluded portion of the face. The daunting challenge in these architectures is to ensure that the generated face of the individual is semantically related to the original person. Furthermore, the choice of the generative model contributes significantly to the robustness of the MFR pipeline. From GANs to diffusion models, MFR has explored numerous possibilities, with some still requiring refinement and further exploration for improvement. All of the notables literature have been presented in this section with inspiration into our own proposed model. 2.4.1 GAN based Face Reconstruction for Masked-Face One of the foremost choices among generative models is Generative Adversarial Networks, first presented by Ian Goodfellow et al. in 2014 [26].The architecture of these models is based on the zero-sum game that relies on two independent neural networks, commonly referred to as the Generator and the Discriminator. The Generator is responsible for generating images from isotropic Gaussian noise, while the Discriminator acts as a binary classifier, distinguishing between the generated images labeled as either ’Real’ or ’Fake’. The adversarial training of GAN allows the generation of images from essentially nothing. In [27], a novel approach for masked face recognition has been presented, utilizing GAN for reconstructing the face followed by state-of-the-art models for non-occluded facial recognition. There have been many variants of the standard GAN model, one of such variants used in this paper is commonly known as "CycleGAN". CycleGAN is most commonly known for the Image-to-Image translation task, such as in this case where the masked face is being tried to transform into the non-occluded face. In this case, the architecture consists of 2 generators and a discriminator for 19 each. For the sake of clarity the architecture of CycleGAN and its working have been explained below. 2.4.1.1 CycleGAN architecture for MFR The fundamental backbone of this approach was the use of CycleGAN for Image-toImage translation particularly used to transform masked image into the non-occluded face images. Unlike the standard GAN, CycleGAN utilizes a pair of generators and a discriminator against each. The architecture of has been shown below. Figure 2.5: CycleGAN architecture for Face restoration The role of Generators within GANs can be validated by their effort to mimic the distribution of data transitioning between masked and unmasked states, and vice versa. This data distribution is exemplified as a∼pdata(a),and b∼pdata(b),where the generator endeavors to map the provided input image. Simultaneously, the discriminator assesses each generated image to categorize it as "Real" or "Fake". In the case of masked face recognition, though, there must be a few considerations regarding the overall architecture, as stated below. •In the original paper [27], the authors utilized a dlib-based feature extractor to map the mask onto the dedicated regions of the face using MaskTheFace API [16]. This is crucial in ensuring that we have a reliable means of replicating the data distribution of the masked faces given a non-occluded image. It serves as the ground truth for the Discriminator that is evaluating the Generator responsible for learning the masked face data distribution. •The primary objective of the GAN upon its inception was to generate images capable of deceiving the discriminator into perceiving them as authentic. However, 20 in this particular context, the image not only needs to be genuine but also needs to maintain semantic relevance to the individual it represents. This necessitates establishing correlation between the visible portion of the face and the masked area. Consequently, this influences the selection of the loss function Based on the arguments the CycleGAN used in the paper included each generator with the inclusion of the cycle consistency loss function along with the adversarial loss of the generators. The adversarial loss function of the generator is responsible for generating real images. The loss function used in the paper is the conventional Minimax loss function stated by the following equation λGAN(G, DB, A, B) = Eb∼pdata(b)[log DB(b)] + Ea∼pdata(a)[log(1 −DB(G(a)))] (2.1) where: •G(a)operates on the data distribution Ato generate B. •DBis the discriminator operating on both the generated images G(a)and the original images A. The Generator aims to minimize the stated loss function while the Discriminator tries to maximize it. The adversarial training is continued unless the Discriminator’s classification accuracy is 50% which is as good as a coin toss. To ensure the semantic correlation among the reconstructed face with the intended person, the cycle consistency loss has been stated below. Lcyc(G, F ) = Ea∼pdata(a)[∥F(G(a)) −a∥1] + Eb∼pdata(b)[∥G(F(b)) −b∥1](2.2) The stated loss function is the L1-norm distance between the generated image distribution and the ground truth. The actual loss function used in the paper is in fact the sum equation (2.1) and (2.2), stated as follows. L=λGAN(G, DB, A, B) + LλGAN(F, DA, B, A) + Lcyc(G, F)(2.3) The architecture for the GAN used in the paper is Deep Convolution Generative Adversarial Nets with the initial convolutional layers followed by the 64-channel convolutional ResNET block. The Discriminator used for the pipeline included 70x70 patch GAN to evaluate the performance on the subgrid of the image rather than the entire image itself or on the pixel level. It is important to note that this approach is similar to the batch processing of the dataset in a Machine Learning model, where it is used to accelerate the training of the network. Furthermore, just like the case of Batch Gradient Descent, it also helps in the backpropagation of the algorithm by removing the noise of updating the weights after each iteration. 21 2.4.1.2 Dataset and Evaluation Metric The paper employs a custom dataset comprising 8131 unpaired facial images, each with a resolution of 256x256 pixels. This dataset includes both masked and unmasked samples, and the model undergoes training for a total of 150 epochs. The lambda value utilized is set to 10. Additionally, a distinct dataset containing 2118 masked faces is utilized for model evaluation. Consistent with the methodology outlined in the original CycleGAN paper [29], both perceptual and quantitative measures are employed to assess image quality. For the quantitative measure part, the Structural Similarity Index (SSIM) has been utilized. This metric compares images based on their luminance, contrast, and structural information. Luminance represents the mean intensity of the images, while contrast corresponds to their standard deviation. Structural similarity quantifies the correlation between the structures present in the images. The SSIM index is computed using the following equations: L(x, y) = 2·µx·µy+C1 µ2 x+µ2 y+C1 (2.4) C(x, y) = 2·σx·σy+C2 σ2 x+σ2 y+C2 (2.5) S(x, y) = σxy +C3 σx·σy+C3 (2.6) SSIM(x, y) = L(x, y)·C(x, y)·S(x, y)(2.7) 2.4.1.3 Limitations of the architecture The limitations of the architecture have been shared as follows. •The architecture introduced a method for image-to-image translation with masked images; however, it lacked evaluation of the complete pipeline, specifically using a state-of-the-art face recognition model to verify recognition accuracy. •Generative models are known for their time-consuming nature, raising concerns about real-time deployment. The paper’s omission of any reference to time efficiency validation leaves implementation for real-time applications questionable. •While the paper emphasized perceptual evaluation of generated images, better perceptual accuracy does not necessarily guarantee efficient performance with face recognition models, as recognition of individuals may vary. 22 •The image generation process remains opaque, with the model essentially operating as a black box. Despite its awareness of the masking process, its ability to unmask images equally well is assumed. •Adversarial training is inherently unstable, with GANs prone to issues like Mode Collapse [30], where the Generator outputs images with minor variations, failing to capture the full diversity of the data distribution. Additionally, as the Generator improves, its output becomes less informative as classification accuracy approaches 50%. 2.4.2 Privacy Preserving Face Recognition using Masked Autoencoders Autoencoders indeed represent another class of generative models that have been widely employed in image-to-image translation [31]. Their versatility extends across various domains, encompassing tasks such as image denoising, synthesis, color enhancement, and beyond. They are commonly used to learn the data distribution of a given dataset using an Encoder-Decoder architecture. The encoder functions to map the input image onto a latent space with reduced dimensionality. Subsequently, a straightforward decoder employing an upsampling architecture reconstructs the original image. A loss function is applied to ensure the fidelity of the generated image to the input. A modified approach in [32] uses Masked Autoencoders for masked face recognition. The primary task for deploying generative models is the same as before. It is used to generate unmasked facial images using masked ones. An additional layer of feature in this paper is the preservation of privacy as the model can generate the image of a person that may reveal his/her identity causing a huge problem. The Masked auto encoders utilize an encoder of the conventional ViT [33] that utilizes the image tokens extracted from the non-overlapping 2D grids of the image. The idea of the encoder is to take in the masked patches of the image with their corresponding positional embedding. The masked patches of the image are ruled out of the encoder’s input to ensure over growing length of the encoder itself. This is essential to ensure that the Encoder is more deep rather than in length for better learning of the features and dimensionality reduction. On the decoder side of the architecture the masked patches are then reused with their unique embedding to ensure the images are put back together and that the reconstruction is reliable. The trained model in the paper is termed as FaceMAE with the detailed architecture shown as follows [32]. 23 Figure 2.6: FaceMAE architecture for masked face reconstruction The pipeline has been shown for both the training and deployment stages. The individual blocks utilized mainly feature MAE, which has been trained on the masked image, in this case, the masked facial photos. The reconstructed image is then compared with the original one to ensure image fidelity. One of the most common choices for the loss function is Mean Square Error to compare the patches or the pixels themselves. However, it fails to meet the semantically comparing data distribution requirement. To solve this issue, the architecture uses a separate model to generate the embedding of the original and the reconstructed images and then compare them using Euclidean distance. This method has been called Instance Relation Matching (IRM) in the paper. This is mentioned by the pre-trained backbone model in the architecture shown above. Last but not least, the model in the deployment stage uses a different dataset with the same trained MAE without the pre-trained backbone. As explained earlier, the idea behind the generative models in MFR is to utilize state-of-the-art face recognition models by ensuring the reliable generation of the image. The same idea has been used in the deployment stage, including the ArcFace loss function that tries to maximize the geodesic distance among the feature embedding [34]. The details of the pre-trained backbone model in the training stage of the model is ResNet50. Still, they can be achieved either with the feature extraction layer of the models such as ResNet, FaceNet or MobileNet [35,36,37]. One of the added advantages of utilizing MAE is that it is a self-supervised algorithm and does not have extra labels for the training, which ensures the flexibility of the datasets that can be used. 2.4.2.1 Datasets The datasets used for the training are WebFace260M and CASIA-Webface [38,39]. The prior dataset contains a total of 260 Million images, and only 10% of the images have been used for the training of FaceMAE in the stated pipeline. 24 2.4.2.2 Limitations of the architecture The limitations of the pipeline presented in the paper are as follows. •The paper discussed the use of masked faces for face reconstruction. Still, the dataset utilized for the training consisted of visible faces, and the masking of the face was completely based on losing some of the positional embedding patches. •The utilization of the lost patches from an image can be thought of as an imageimpainting problem. The model was neither tested on the real masks on an image nor considered any synthetic masking of the training dataset with visible faces. •There is no consideration to the background noise removal. In the case of deploying the model for surveillance systems, the captured image often contains the image with only a single patch containing the face itself. The currently proposed architecture would include a single positional embedding and might be ignored, or even if considered, it will be considered as a single element. •The comparison in the results portion have not been made with the state-ofthe-art MFR models to compare the accuracy and overall latency in a real-time network. •The model information in terms of the hardware have been present such as the utilization of the modern A100 GPU but the inference time of the model have neither been reported nor discussed to reduce. 2.4.3 Masked Face Recognition with Generative Data Augmentation Another approach within generative models involves generating datasets using data augmentation techniques [40]. This approach presents a scenario for image-to-image translation that can produce masked faces from visible face datasets, thereby expanding the available dataset. This is achieved by employing the CycleGAN architecture with multiple Generators and Discriminators, as proposed in the paper and termed IAMGAN, to preserve identity during image masking. Data augmentation is a crucial technique utilized to reduce variance in models and mitigate overfitting. Typically, there exists a trade-off between the bias and variance of a model, as they are conjugate variables. Data augmentation offers a significant advantage by reducing variance without increasing bias when the dataset size increases. Consequently, it serves as an effective method for improving model accuracy. This paper presented an approach similar to the semantic segmentation of the mask on a person’s face, and therefore the same CycleGAN can be utilized to further reverse the masking of the face if needed. 25 The loss function to ensure semantically intact images is the adversarial loss to ensure the images being real which is followed by the cycle consistency loss for the images to semantically intact to each other. The loss function can be stated as follows L=λGAN(G, DB, A, B) + LλGAN(F, DA, B, A) + Lcyc(G, F)(2.8) The contribution of the paper can be seen in terms of the dataset that they have proposed using their trained model weights. The proposed dataset is known as the Masked Face Segmentation and Recognition dataset. The statistics of the dataset have been shown below [40]. Figure 2.7: Statistics of the MFSR Dataset 2.4.3.1 Proposed Dataset The MFSR dataset proposed in the paper consists of two different subsets listed as follows. •Facial Mask Segmentation subset: This part of the dataset contained around 9,742 well annotated images collected from the internet. This subset is responsible for the mask segmentation that is presented for the task of instance segmentation of the mask in the facial images. It also considers various diversity in the dataset including various pose, age variation and ethnicity. It is represented as the MFSRSEG dataset in the image of statistics shown above. •MFR Benchmark Dataset: This dataset is for the Masked Facial Recognition containing a total of 11,615 images belonging to 1,004 identities. Out of the these unique individuals 704 belong to the real-world whereas the other 300 have been obtained from the internet. The images have been filtered out to ensure it contains only the images where the face is visible and there are no contradictions in terms of the file’s reliability. The proposed model architecture was not presented in terms of its performance with the dedicated MFR pipeline, however the paper does highlight and the product of the model that is the dataset has been released for custom training. 26 2.5 MFR using Attention based models Up to this point, the discussed approaches have involved incorporating the face mask as an integral feature, treating it as a standard component, and employing generative models primarily centered around the CycleGAN architecture. One notable drawback of the generative model approach was its reliance on GAN, which inherently poses challenges due to its intractability and the instability encountered during model training, as elaborated upon with the specific issues mentioned earlier. Another approach recently adopted by the researchers is the utilization of attention-based mechanism that allow more focus on the non-occluded part in a masked facial image. These approaches are typically more time-efficient compared to their generative counterparts. With careful design, the pipeline can produce superior results, leveraging the efficiency of these methods. 2.5.1 Cropping and attention based MFR One of the advantages of attention based modules is the utilization of the non-masked regions to be extracted from the overall masked image. This novel approach has been adopted to extract and use only the visible portion of the masked faces exclusively for MFR [41]. The cropping algorithm has been merged with the attention-based mechanism as to reduce the feature map and the input pixels to work with. Cropping, as recommended by the author, has been performed manually to ensure that all nonmasked regions are accurately extracted from the original image. On the attention part, there are two different mechanisms mentioned in the proposed architecture by the authors. These are: •Intra-channel attention mechanism •Spatial attention mechanism These attention layers have been used inside each of the ResNet blocks to allow attention to the desired part while extracting feature maps. The details for their implementation are as follows. 2.5.1.1 Intra-channel attention The idea behind the attention network is to capture possible correlations among the input data image. One such dependency that can possible exist among the various channels of an RGB image. The channel attention is carried by the help of two different descriptors, obtained by the average and maxpooling respectively. The channel attention module depends only on the RGB image, so the descriptors must belong to 27 the vector space such as Gavg ∈RC×1×1and Gmax ∈RC×1×1, where Gavg represents average pooling and Gmax represents max pooling. These descriptors are then fed into the Multi Layer Perceptron network which unifies them into a single embedding such that Mc∈RC×1×1. The inclusion of MLP allows the model to have the non-linearity due to the presence of the activation function. The activation function chosen in the paper for the sake of implementation is Sigmoid, given by the equation stated below. σ(x) = 1 1 + e−x(2.9) 2.5.1.2 Spatial attention The spatial attention in a Neural Network helps localize the object coordinates in an image. From the implementation point of view, the spatial attention layer is similar to the intra-channel attention. As in the previously discussed attention mechanism, feature descriptors are obtained using the average pooling and Maxpooling layers. The subtle difference in spatial attention is that instead of using the MLP, there is a convolutional filter kernel convolved with the descriptors, followed by the Sigmoid activation function. We have the exact implementation, except that it has utilized CNN-type convolution instead of the fully connected MLP. The detailed architecture for CBAM is as shown below [41]. Figure 2.8: Detailed architecture of the CBAM 28 Figure 2.11: Weights and Bias sharing in Siamese Networks The block diagram illustrated above for a Neural Network represents another option for object detection. As with any other case, the suitability of this approach depends heavily on the nature of the problem and the complexity of objectives to be achieved, such as time efficiency or model accuracy. Different architectures may be preferred based on specific requirements and constraints of the task at hand. One of the recent implementations of the network for One-shot learning has been presented in [54]. The architecture is shown below. Figure 2.12: Architecture of the One-shot Siamese Network It can be seen from the NN shown above that it consists of the basic CNN followed by the necessary non-linear layers required to allow the model to attain pattern recognition for complex datasets. The network in [54] was a generic representation of many possibilities for different tasks including Facial recognition or even Masked Facial Recognition. One of the key advantages of Siamese Networks is their flexible architecture, which enables the modeling and solution of various problems. By adding a final layer followed by a sigmoid activation function, the network can be transformed into a binary classification problem. Similarly, by using softmax activation, it can address multi-class classification problems. Furthermore, the same trained network can be leveraged to address open-class problems, such as Masked Facial Recognition 35 (MFR) in many cases. This adaptability makes Siamese Networks a versatile choice for a range of tasks. 2.6.1.1 Advantages of Siamese Networks The advantages of the Siamese Networks have been listed below. •Class Imbalance: An added advantage of these networks is their robustness to class imbalance in the dataset. The architecture of the network has been designed to ensure that even in case of limited dataset for a particular class the learning can be carried out effectively. •One-shot learning: Siamese networks are inherently one-show learners much like YOLO. This makes them one of the faster ways to detect and recognize objects such as faces in a typical face recognition problem. •Semantic Similarity: Unlike the common classification networks, Siamese networks work on the principle of extracting feature embedding from the input image. The output is the feature vector rather than the output class number. The feature vectors can be further stored in a database as both the useful and compressed representation of the original image. 2.6.1.2 Loss Functions Siamese Networks operate by comparing the feature embedding of the input data with the corresponding label. The label in this case is also an embedding for the known entity. The dimensionality of the acquired feature vector depends on the number of neurons in the output layer. It should be the same as that of the ones stored in the database for comparison’s sake. The common common choices for the loss function in case of Siamese Networks have been discussed as under. •Contrastive Loss: This loss function is based on the distance metric among the feature embedding [55]. It assumes that the model is comparing a pair of images in order to ensure that the images belonging to the same person are pushed closer and the unrelated images are separated in the feature space. The distance metric most commonly used in the process is the Euclidean Distance as shown below. Dw(X1, X2) = ||Gw(X1)−Gw(X2)||2(2.10) The above equation computes the distance among the feature vectors, but it does not include any penalty terms in case the model predicts the output wrong. The pairwise implementation of the Contrastive Loss function has been shown below. 36 L(W, Y, X1, X2) = 1 2[(1 −Y)·D2 w+Y·max(0, m −Dw)2](2.11) The loss function includes the term ’m’ which indicates the margin. This ensures that there is a certain distance among the dissimilar feature vectors in a feature space. In the above loss function the value of ’Y’ indicates the label for each sample image. If both the images belong to the same person then the value of Y = 0, which means that the loss function is reduced to the following equation. L(W, Y, X1, X2) = 1 2D2 w(2.12) The above equation is essentially the half of the Euclidean Distance stated in equation 2.10. It means that the penalty on the Neural Network is only dependent on the distance among the feature vectors of the similar images. However, in case of dissimilar images the loss function is reduced to the following equation. L(W, Y, X1, X2) = 1 2[Y·max(0, m −Dw)2](2.13) The margin introduced in the above equation ensures that there is at least ’m’ units of distance among the unrelated feature embedding. It is not a hyper parameter but the threshold that can be set to have the desired results. For instance, in case of sensitive application such as surveillance system the threshold can be set to high ensuring that the predictive model has to be absolutely sure to be able to classify the person as "wanted". In other less sensitive applications such as ticketing system in the park the threshold can be adjust to a slightly lower value to ensure faster training process. The total loss function for all the training examples can be computed by the cumulative sum over the entire set of example images. L(W) = P X i=1 L(W, (Y, X1, X2)∗)(2.14) •Triplet Loss: The Triplet loss is based on the triplet of images rather than the pair of them [56]. The images are labelled as the Anchor, the Positive, and the Negative example. The idea behind this loss function is to improve the contrastive loss function by increasing the distance between the anchor and the negative example while simultaneously decreasing the Euclidean distance between the anchor and the positive example. The main objective of the loss function is to ensure that the distance of the feature embedding inside a feature space is such that the anchor and the positive example are closer than the anchor and the 37 negative example image. Therefore, the mathematical relation for the objective function can be stated as follows. L(a, p, n) = max{d(a, p)−d(a, n) + α, 0}(2.15) In the above equation, d(a,p) is the Euclidean distance among the positive and anchor image, whereas d(a.n) is the distance among the anchor image and the negative example image. ’α’ is the margin term that sets out a threshold as to how much the positive and the negative example images must be set apart from the anchor image. The application of the margin and how it helps in the separability of the embedding in the feature space can be demonstrated by the image shown below [57]. Figure 2.13: A demonstration of margin in the Triplet Loss Function From the above image, it can be demonstrated that the training of the Siamese Networks with respect to the Triplet Loss function is carried out in a way that sets the negative examples further apart from the anchor compared to the positive example by at least the amount of the magnitude of the margin term αincluded in the loss function as shown in equation 2.15. Just as in the case of the contrastive loss function, the term can be varied during the network training, depending upon the application for which the model was trained. 2.6.1.3 Limitations of the architecture Although Siamese Networks offer an excellent pipeline for ensuring the mapping of the masked and the unmasked face images, it needs to include the following limitations in its implementation. •Extensive Dataset: The dataset must be prepared, and the availability of the paired datasets must be ensured in order to train these networks. This is because 38 both the contrastive and the triplet loss functions depend on the image pairs, if not the triplets, for training the model. For the sake of MFR, the image for both the masked and the unmasked face image must be present in order to train the model. •Difficult training: The problem with the Triplet Loss function is explicitly ensuring that the dataset contains hard enough triplet examples for the network to be effectively trained. In the case of a relatively straightforward set of examples, it would be easier for the model to distinguish among the examples, and the learning would not be effective as the model might not be able to perform well when deployed. This accounts for the dataset to be prepared cautiously, ensuring that the examples are not randomly picked but prepared to provide better model performance. •Longer Training: The training of the Siamese Networks is a computationally complex task and therefore it demands for the powerful GPU and longer training times. •Sensitivity to Imbalance Dataset: The requirement of the loss function utilized in Siamese Networks demands the availability of the data pairs that need to be mapped against each other. In case of an imbalanced dataset with some examples missing the image pairs, the network will not be able to pair the single images. Therefore, the training for the particular example image will be halted. 2.6.2 Ensemble Models Considering the pros and cons of the techniques discussed in the previous section, a common approach for improvement would be to utilize the ensemble models. One such example presented in [58] is the utilization of Transformer models followed by the convolution and fully connected networks. The architecture for the model has been presented below. Figure 2.14: Architecture of the Ensemble model 39 The architecture has been presented to process each part of the image separately. The transformer network for the image encoding has been inspired by Dosovitskiy et al [59]. The idea for the transformer network is to ensure that the input image is divided into patches so that the individual patches can be processed independently. The division of the images into the patches distorts the spatial information of the image and therefore the positional embedding have been inserted into the network for proper ordering. Once the patches have been developed, the smaller images have been passed into the rest of the network for feature extraction and attention based learning. The networks has been designed in the way that the patched images have been processed using the Batch Normalization layer, CNN , and finally it feeds into the fully connected layer. The first block into the network has a different Multi-Headed Attention layer inspired from the transformer. Once the attention have been established, the features can be extracted as normal. The last layer of the network contains the fully connected layer with the softmax activation function that computes the probability of the person’s identity given the total number of person in the original database. The usage of softmax ensures that the architecture for the model is in fact well suited for the closed-class problem. The utilization of the positional embedding at the start is utilized later on in the pipeline for the proper ordering of the patches into the complete image. 2.6.2.1 Dataset The dataset utilized in this paper is the well-known benchmark known as the LFW dataset. The overall dataset used in the training of the model included around 25,000 images with a total of 5,749 subjects. The accuracy achieved for the proposed ensemble model was less compared to the other models. The authors attribute the lesser accuracy towards the nature of the LFW dataset where 4,047 subjects out of the total 5,749 contains only a single image. 2.6.2.2 Limitations of the architecture The drawbacks of the proposed architecture have been briefed as follows. •The model under performed when tested on the LFW dataset depicting the underfitting caused by the training of the architecture •The paper presented no grounds towards the inference time or real-time deployment of the pipeline •The architecture could have been improved with the utilization of pre-trained transformers instead of training the model on limited dataset. 40 2.7 Datasets in MFR It is only possible to train a state-of-the-art Deep Learning model with a refined and cleaned dataset. All of the better models have a common feature: they exhibit the utilization of a better set of data for the training. This section presents the most commonly used datasets in Masked Face Recognition. Although the datasets have been mentioned alongside the discussed approaches and the paper, this section will explore holistically the available datasets for the convenience of the reader. The datasets have been compiled as shown in the table below. Table 2.5. Summary of the common MFR benchmarking datasets Dataset Size (Images) Identities Types of Masks RMFRD [60] 95,000 525 Real-world SMFRD [60] 500,000 10,000 Synthetic MFSR [40] 11,615 1,004 Real-world / Synthetic LFW-SM [37] 13,233 5,749 Synthetic MFR2 [37] 269 53 Synthetic MFV [61] 400 200 Synthetic MFI [61] 4,916 669 Synthetic MFD [62] 990 45 Synthetic MFW-mini [63] 3,000 300 Synthetic CelebA [64] >200 K 10,177 Synthetic CelebA-HQ [65] >30 K 307 Synthetic MS1MV2-Masked [66] 5.8 M 85,000 Synthetic CFP-FP 7,000 500 Synthetic CFP-FF 7,000 500 Synthetic CASIA NIR-VIS 2.0 [67] 17,580 725 Synthetic Oulu-CASIA NIR-VIS [68] 7,680 80 Synthetic BUAA-VisNir [69] 2,700 150 Synthetic CASIA-FaceV5 [70] 2,500 500 Synthetic VGG-Face2m[71] 3.3 M 9,131 Synthetic Webface [72] 500 K 10,000 Synthetic AR [73] 4,000 126 Synthetic Extend Yela B [74] 16,128 28 Synthetic AgeDB [75] 16,488 568 Real-world A total of 23 benchmark datasets have been reported and published as open-source. These datasets are important as all of the papers utilise the derivatives of these datasets for the evaluation and performance verification of the trained models. The proposed 41 model in this thesis has also been evaluated based on the derivatives of the mentioned dataset. 2.8 Post-processing for Model Deployment There needs to be more than the training of the model to ensure a reliable deployment for real-time applications. The idea of the post-processing of an already trained model has been borrowed from the recent open-source deep-learning architectures that utilize billions of parameters [76]. With the exponential increase in the number of parameters, there needs to be equivalently suitable hardware in terms of the GPU for loading the model and processing it correctly. One such study has been shown below [76]. Figure 2.15: Exponential Growth of the model size over time It can be visualized from the graph above that the model size risen exponentially to the point when it is almost impossible for the large models to be accessed on the available hardware. The concept can also be utilized for the case of MFR application in order to deploy the model on an AI accelerated board such as Jetson Xavier AGX for efficient implementation on the CUDA cores. Not only does it help in the faster inference time of the model, but it also ensures that the accuracy of the model is retained within acceptable limits. The AI-based hardware is generally designed for the 42 installation as an independent node to be utilized in a greater mesh network. This accounts for limited storage memory and the constraints on power consumption asking for the model to be optimized for practical use. The model optimization techniques have been discussed in the upcoming section. 2.8.1 Model Quantization The training of the model consists of tuning the weights and biases in order to achieve the desired result. These quantities are expressed in a matrix consisting of numbers that are usually floating point, most commonly FP32 standard. FP32 standard ensures that each individual weight and bias requires a total of 4 bytes each for storage. This size also translates in the computational complexity of the model as these matrices are utilized in the computation of the desired activations. A possible way to solve this problem is to quantize the model weights into other data types that are more supportive of the intended hardware. It ensure less computational time for the algorithm with the same computational complexity.The data types for the storage of model weights and biases have been compared as follows. Table 2.6. Floating-Point Datatype Characteristics Datatype Precision Maximum FP32 Best ∼1038 FP16 Better ∼104 BF16 Good ∼1038 From the above table it can be concluded that the default data type most commonly used by the Deep Learning models offers the best precision with a high maximum value. In the default configuration a single weight parameter requires a space of 4 bytes. For a large model that has billions of parameters the memory of the device can be easily exhausted leading to the real-time application being crashed in the production cycle. The details of the number of bits assignment for the different floating points have been stated below. Table 2.7. Bitwise allocation for floating point variants Format Sign Exponent Fraction Min (approx.) Max (approx.) FP16 1 5 10 −6.10 ×10−56.55 ×104 FP32 1 8 23 −3.40 ×1038 3.40 ×1038 BF16 1 8 7 −3.39 ×1038 3.39 ×1038 The exponent is the range of the floating point; the greater the number of bits for the exponent, the more outstanding the minimum to maximum range for the number. 43 Based on the stated table, it can be deduced that the model compression can be achieved by down-changing the data type of the model from FP32 to either BF16 or FP16. The quantization achieved is this manner is naive-based approach as it does not ensure that the accuracy of the model is conserved while attempting to compress the model. To ensure little to no drop in the model’s accuracy, the linear quantization technique is most commonly used in SOTA machine learning models [77]. Using the linear quantization technique, the FP32 standard for the weights and biases can be reduced to the integer data type for 8 bits each. The quantization have been explained as shown below. 2.8.1.1 Linear Quantization In linear quantization, the mapping among the 32-bit floating point and the 8-bit integer have been mapped using a linear transformation. The mapping of the data types in the case of model quantization from FP32 to 8-bit integers have been shown below [78]. Figure 2.16: Mapping of FP32 to 8-bit integer 44 learning a unified representation of all these models. The interaction of the student models with a bunch of cumbersome models have been shown in the figure below [90]. Figure 2.20: Multi-teacher knowledge Distillation The idea of incorporating post-training optimization technique can indeed result in the overall computational efficiency of the model while maintaining the same accuracy. In the use-case of this thesis it is preliminary as the model is intended for deployment on the hardware in order to ensure reliable performance in real-time application. 51 Chapter 3 Methods and Methodology This thesis presents a novel approach for real-time Masked Facial Recognition. The critical analysis from the literature review of the concerned topic has led to the adaptation of the positive impacts of the pipeline above and improvements to their adverse effects. Since the algorithm emphasizes the real-time implementation of the MFR pipeline, particular focus has been given to the model optimization techniques and the CUDA core optimization on the dedicated hardware. As discussed in the section below, a mismatch in any optimization strategies may lead to adverse performances even when compared to an ordinary CPU [95,96]. Starting from the developed pipeline, this chapter is focused on the insights for the model developed, process management, bottleneck analysis and finally, the implementation results on the Jetson Xavier AGX. The entire pipeline has been discussed in the block-wise scenario, where each single block is responsible for learning a specific task that eventually contributes to the more significant cause of Masked Facial Recognition. A detailed analysis of the drawbacks of all the previous architecture, such as background noise, lack of face verification and a robust pipeline for a reliable face generation that is also semantically correlated with the visible part of the face. The hardware and real-time model deployment with an external camera demand specific analytical changes to the architecture discussed in the following section. 3.1 Overview of the MFR pipeline Masked Facial Recognition is a complex task requiring the development of a complete pipeline rather than a simple Neural Network as in the case of regression or classification. The designed pipeline is a tree-like tree-like, depending on the need for the analysis or the user’s requirement. An ensemble model-based approach allows for the tradeoff between the model’s accuracy and the timing efficiency of the model. An analysis has been made for the attention-based and generative models to ensure optimal 52 performance during deployment. For the sake of convenience and further discussion, the overall sketch of the developed pipeline have been reiterated as shown below. Figure 3.1: Outline of the proposed model architecture The individual block stages for the pipelines designed have been listed as follows: •Face Verification stage. •A decision stage to account for the tradeoff between the model’s accuracy and the computational complexity. •A generative model-based pipeline based on the conditional Denoising Diffusion Implicit Models to reconstruct the masked face. This is followed by the backbone model to map the reconstructed face onto the feature space. •An attention-based model scheme for the extraction of the feature embedding based on the attention-based model followed by the loss function that optimizes the geodesic distance •The availability of a database that can accessed in real-time with a linear time complexity or less. 53 Masked Facial Recognition is a complex task requiring the development of a complete pipeline rather than a simple Neural Network as in the case of regression or classification. The designed pipeline is a tree-like tree-like, depending on the need for the analysis or the user’s requirement. An ensemble model-based approach allows for the tradeoff between the model’s accuracy and the timing efficiency of the model. An analysis has been made for the attention-based and generative models to ensure optimal performance during deployment. 3.2 Face Verification stage One of the foremost and essential elements of the Face Recognition or Masked Face recognition model is using the face verification step. It enables us to remove the background noise and focus more on extracting the features of the relevant part of the image. In real-time, a picture might contain more than one face, and therefore, the face verification steps need to be robust, and negligence in developing this type of model can lead to a severe bottleneck. This is because when an MFR model is deployed in real-time, the camera will capture some extra background noise, and running a feature extractor model right away might hinder the model’s output [97]. The face verification stage in the pipeline above is the same for the entire ensemble model, including the generative and attention-based models. Face verification aims to accurately localize any N number of faces in an image, whether masked or unmasked, from the complete image captured by the camera in real time. In the previous architectures, a number of face verification models, including the MTCNN, SSH, PCN, and Tiny Face detector [9,98,99,100]. All these face detectors, in general, and MTCNN in particular, have been used widely as state-of-the-art face detection algorithms. However, all these algorithms were developed in the pre-COVID era and used to detect faces when presented in an unoccluded form precisely. Considering the problem of Masked Facial Recognition, the choice of face localization or verification step features the famous RetinaFace model. The initially proposed architecture of RetinaFace comes with many more features than simple face localization. It has been widely used for 3D face construction, given 2D images of the face, face location and bounding box regression [101]. For the use case in our pipeline, only the face localization part of the feature has been given due importance. The face detection part in RetinaFace, much to the benefit and advantage of our application, has been carried out in a single shot, so the algorithm is efficient and reliable enough to work with real-time deployment. The architecture for the face localization step consists of the following three (3) blocks. •Feature Pyramid Network: This network stage is responsible for applying the Convolutional Neural Network with varying filter sizes by constructing the 54 five (5) pyramid features of resized images. This is one of the critical steps for ensuring that all the faces in a photo of any given size are detectable in an image. The feature pyramids have been extracted using the ResNet model, which has been trained to extract the features from images [35]. •Context Head Module: This module is one of the significant blocks in ensuring reliable face detection in masked and unmasked faces. It utilizes the Deformable Convolution Network to ensure that the features have been extracted from different locations rather than a set of localized grids inside an image. It helps and acts as the attention mechanism as the information is routed in the entire image space rather than being confined in the grid size of the small patch of the images that is equal to the filter size. The deformed convolutional approach have been depicted as shown in the figure below [102]. Figure 3.2: Standard Convolution vs Deformable Convolutional Neural Network It can be verified that the distortion in the selection of pixels for the computation is unstructured in the case of DCN leading to the flow of information across the image. It is important as the masked faces will require more information form the unobscured parts of the faces. •Classification step: This part of the architecture is responsible for the binary classification of whether the face has been detected. Based on the IoU metric, the algorithm determines the region of union among the detected faces by the bounding face regression and the ground truth. The decision criterion chosen in the paper is as follows: 55 f(IoU) =        positive if IoU ≥0.7 negative if IoU ≤0.5 ignored otherwise These configurations allow the model to accurately determine the coordinates of the face even in the wild. Another important aspect of the model is that have been trained on the WIDER Face dataset [103]. It contains a wide range of faces for facial detection under challenging circumstances including occlusion, pose variations, and lighting changes. A subset of images from this dataset have been shared as follows. Figure 3.3: A subset of images in the WIDER Face dataset The images in the WIDER face dataset can be to have a diverse group of images from a single face localization example to multiple faces including the faces with occlusion. RetinaFace utilizes the multi-task loss function. For the face localization we have the following. L=Lcls(pi, p) + λ1Lbox(ti, t) + λ2Lpts(li, l) + λ3Lpixel (3.1) In the above equation we have four (4) terms included in the objective function that have been explained as follows. –Lcls(pi, p): This term represents the softmax classification loss for the network. It is a binary loss that determines whether there is the face of a person present in the image or not. –λ1Lbox(ti, t): It represents the bounding box regression loss for the face detection. This part of the algorithm is important as it involves the precise localization of the images forming an integral computational block in the overall MFR pipeline. 56 –λ2Lpts(li, l): To ensure the localization of the face even in the case of occlusion and pose estimation the algorithm utilizes 5 different facial landmark features including eyes, jawlines on both sides of the face, eye brows, nose and lips. –λ3Lpixel: It is the pixel wise loss with respect to the ground truth. This loss is responsible to ensure that the predicted face along with the bounding box is well-within the threshold of tolerance. 3.2.1 Choice of the Face Verification block The choice of RetinaFace for the face verification step has been inspired by the results of other well-known facial detectors, including MTCNN, tiny face detectors, and SSH, as reported in earlier studies. Furthermore, it has been noticed in practice that only the RetinaFace includes facial landmark-based detection and the modern Deep Learning-based approach, which allowed the hybrid model to be utilized efficiently in our case of masked facial recognition. A small recollection of the comparison of other detectors with the RetinaFace have been presented in the table below. Table 3.1. Comparison of Face Detection Methods for Masked FR Architecture Description Speed vs. Acc Masked FR Potential Tiny Face Detector Lightweight model optimized for small faces. Very fast, lower accuracy on complex scenes. Limited. Mainly focuses on small faces. SSH Context-aware detector using multiple scales. Good balance of speed and accuracy. Moderate. Can handle partial occlusion but not designed for it. RetinaFace Robust to occlusion and pose variation. Moderately fast, high accuracy. Good. Explicitly designed for various occlusions. MTCNN Landmark-based, cascaded approach. Slower, but very accurate for unoccluded faces. Poor. Highly reliant on facial landmarks, disrupted by masks. Most of the detectors have been designed explicitly for facial recognition, with more emphasis on lighting and pose variations. The more classical approaches, 57 including HOG-based features, concluded that they were less efficient than the modern detectors involving machine learning models, which have surpassed their statistical counterparts. Recently, one of the most widely used face detection algorithm have been MTCNN and RetinaFace, but the architecture of the RetinaFace as disscused earlier focuses more on the face using the context head module and is also a single shot algorithm so it supports the face extraction process in MFR applications. Both the implementations have been carried out on the faces in the wild; a small subset of the overall test have been as shown in the figure below, as conducted in our experiment. The subset contains all the aspects including the pose variation, diverse ethnicity and multiple faces located at distance with background noise. Figure 3.4: Performance comparison of MTCNN vs RetinaFace on Masked Faces in the wild It can be observed that the application of the RetinaFace model, even on the lowest of the possible pixels and resolution of the image, allows multiple detection of the faces when masked compared to MTCNN as verified in the first 2 images from the input test dataset. Furthermore, the time consumed by MTCNN was 174 milliseconds for the image of the most significant resolution, 318 x 159. In contrast, 58 RetinaFace took around 20 milliseconds, making it faster at the inference time due to the single-shot nature. 3.3 Inference time based architecture selector The choice of pipeline at the time of deployment was made based on the inference time. The Python script used for the deployment allows the user with extra tags to select the accuracy vs tradeoff. The decision could have also been made independently based on the application. For instance, in the case of a sensitive application with fewer stakes on the inference time, one can focus more on the Diffusion model-based pipeline and even combine the results of both the diffusion model and the attention-based mechanism. However, the face detection is joint to both pipelines; therefore, only the pre-processed images are fed into the model. In the case of multiple faces, the images are fed into batches, and the optimization algorithm with the mini-batch processing has been implemented—mini-batch optimizers, including the Gradient Descent and the Adam optimizer, allow faster model parameter convergence. Therefore, the input image fed into the model has the feature dimension ηX= (pH, pW,3, m)where ’m’ is the number of face images extracted from the input image depending on the number of detected faces. 3.4 Denoising Diffusion Implicit Models Another model employed in the pipeline is the Denoising Diffusion Implicit Model (DDIM), specifically designed to generate unoccluded face images from masked images. DDIM stands as an advancement over previously published research on the probabilistic version of the diffusion model. This research indicates that diffusion probabilistic models surpass conventional Generative Adversarial Models by introducing tractability to the generation process [104]. Unlike conventional Artificial Neural Networks or Convolution models, DDIM solely utilizes Neural Networks for the reverse diffusion process, often referred to as the sampling process. The architecture of DDIM encompasses the following key aspects: – Forward Diffusion process: This process entails constructing a Markov chain process that involves the gradual noising of the image from an initial state x0∼q(x0)to a final state xT∼q(xT), where q(x)represents the probability distribution. The noise is added in incremental steps based on the noise scheduler βT, which serves as a hyperparameter that can be adjusted to facilitate proper learning of the model. At each step, the noise is 59 sampled from a standard normal distribution and then scaled by the mean and variance, as specified by the following closed form. q(xt|xt−1)∼ N xt;p1−βtxt−1, βtI(3.2) The above equation states that any given latent space variable q(xt)can be attained by the previous state only, hence making the chain causal and memory less. The number of total noising steps ’T’ is also a hyperparameter that can be tuned for optimal performance based on the needs of the application. – Reverse Diffusion process: The reverse diffusion process involves a Neural Network tasked with learning to generate the noiseless image sampled from the same distribution from which x0was originally extracted. Unlike aiming to learn the exact distribution of input data samples, this process focuses on extracting a sample from the same distribution. The noising and denoising processes are interconnected through an embedding mechanism similar to those utilized in Vision Transformers. These embeddings enable the model to keep track of the denoising step, determining how much noise should be predicted at each stage. The goal of the reverse diffusion process is to learn the mean and variance of the image since the input to the Neural Network is an isotropic Gaussian image. This is the key difference when compared to GANs as it does not look to produce an image from noise which would have made the process intractable [105]. The reverse diffusion process can be expressed in closed form as depicted below. p(xt−1|xt) = N(xt−1;µθ(x, t),Σθ(x, t)) (3.3) To illustrate more on the idea of the Denoising Diffusion Implicit Models, the detailed architecture have been presented as shown below [106]. Figure 3.5: A demonstration of the Denoising Diffusion Implicit Models From the above shown architecture, it can be verified that the generation of the image from the isotropic Gaussian image have been generated using general 60 Wout =Win −F+ 2P S+ 1, Hout =Hin −F+ 2P S+ 1(3.13) where Pis the padding on the image, Fis the filter kernel size and S is the stride of the filter kernel. This layer learns the parameters for the filter kernel weights having the size 3x3. The Max pooling layer have not parameters of learning and they are only responsible for reducing the image size by extracting the most dominant features in the local image space. – Decoder block: These are the set of blocks shown in the right hand side of the architecture depicted in figure 3.7. The decoder blocks are connected internally utilizing the up-sampling layers that reverses the spatial reduction of the image by the encoder. One of the foremost task of the decoder block is to ensure that the spatial dimension of the image have been kept reserved and that the output image have the exact same size as the input image to the model. This assumption have been made since the forward diffusion process was responsible for the noising of the image while maintaining the same size for the image. 3.4.2.3 Loss Function The loss function for the previously seen generative models include the penalty if the generated image was fake. An addition to this adversarial loss function is the cyclic loss to ensure that the images generated by both the generators are semantically related by generating an embedding for each. Ideally, the loss function for DDIM should be the negative log-likelihood of the generated image that is −log(pθ[x0]) but since the generation of the image is dependent on all the individual latent variables it is not possible to keep track of all the transitions. A better approach for the loss function in this case is to use the Variational Lower bound. This is to ensure that the addition of the divergence of the original and the generated distribution of the images increase the log-likelihood of the image. The relation for this inequality have been shared below. −log pθ[x0]) ≤ −log pθ[x0]) + DKL hq(x1:T|x0)|pθ(x1:T|x0)i(3.14) It is imperative to further modify the equation since the computation of the negative log-likelihood is required in the variational lower bound as well. This can be achieved by replacing the KL divergence in equation (3.14). KL divergence is a measure of the similarity of the probability distribution, in this case among 67 the generated and the original data distribution. The relation for KL-divergence has been given below. DKL p||q=Zx p(x) log p(x) q(x)dx (3.15) The modified loss function yields the following results. −log pθ[x0]) ≤ −log pθ[x0]) + log q(x1:T|x0) pθ(x1:T|x0)(3.16) Using the conditional probability rule to the term in the denominator and some simplification steps, the equation can be further further reduced to a more compact form. −log pθ[x0]) ≤ −log pθ[x0]) + log pθ(x0)q(x1:T|x0) pθ(x0:T)(3.17) Finally, the intractable negative-likelihood term of the model can be vanished by utilizing the properties of logarithms −log pθ[x0]) ≤ −log pθ[x0]) + log pθ[x0]) + log q(x1:T|x0) pθ(x0:T)(3.18) The loss function is finally reduced to the following form. −log pθ[x0]) ≤+ log q(x1:T|x0) pθ(x0:T)(3.19) The perimeter to compute on the right side of the inequality is finally deprived of the intractable terms. Speaking from the implementation point of view, the loss function contains the forward diffusion process in the numerator and the reverse diffusion process in the denominator. Computing the entire forward and reverse diffusion process requires the latent variables to be computed at all time stamps. This is again one of the tedious tasks that is not possible computationally in realtime with the limited constraints of the GPU provided by Jetson Xavier, and it requires much more than O[n] space complexity. The equation is further reduced by expanding to the known forward and reverse diffusion equations stated in (3.2) and (3.3), respectively. log q(x1:T|x0) pθ(x0:T)= log QT t=1 q(xt|xt−1) p(xT)QT t=1 pθ(xt−1|xt)!(3.20) 68 Further simplification is necessary to ensure better image generation This has been carried out as follows. log q(x1:T|x0) pθ(x0:T)=−log p(xT) + log "QT t=1 q(xt|xt−1) QT t=1 pθ(xt−1|xt)#(3.21) log q(x1:T|x0) pθ(x0:T)=−log p(xT) + T X t=1 log q(xt|xt−1) pθ(xt−1|xt)(3.22) log q(x1:T|x0) pθ(x0:T)=−log p(xT) + T X t=2 log q(xt|xt−1) pθ(xt−1|xt)+log q(x1|x0) pθ(x0|x1)(3.23) It is important to note that the forward diffusion process q(xt|xt−1)can be further expressed using the Bayes rule with the maximum likelihood estimation. q(xt|xt−1) = q(xt−1|xt)q(xt) q(xt−1)(3.24) The forward diffusion process’s maximum likelihood conditions the image’s generation xt−1on xt. In the context of the DDIM, this transition is from a highvariance image to a slightly lower-variance image. It can be challenging for the model to learn the reverse diffusion process as the model needs to be aware of the direction in which it should move. Therefore, the conditional probability is further conditioned on x0and xt. Substituting the values and simplifying the equation yields the following loss function. L= ϵ−ϵθ(xt, t)  2(3.25) The mean of the predicted noise can be computed using the following relation. ˜µt=1 √αtxt−βtϵ √1−¯αt(3.26) For improvement, the model can be trained with another sampler that will try to work on the variance of the Gaussian noise for better denoising of the noise. The analysis above shows that DDIM training is made possible without adversarial training. Still, the challenge lies in conditioning the model for generated unmasked faces from the masked ones, ensuring the semantic relation among the image so that the generated face belongs to the same person. This problem is addressed by utilizing the adversarial training of the model using DDIM as the generator along with the patch discriminator. 69 3.4.2.4 Conditioned Image generation using DDIM The generation of the image can be conditioned by utilizing the GAN-like structure using the patch-based discriminator to allow for the semantic relation among the unmasked and masked images. The masked face image is fed as input to the DDIM model, the generator in this architecture. The output image is then fed into the patch discriminator and the unmasked image. The discriminator generates an embedding with the distance closer if generated, and the original image is semantically coherent; otherwise, they have been pulled apart. The addition of adversarial training allows for the conditioned and semantically generated images but at the expense of the stability of the training. The model’s training is made stable, but adding noise to the discriminator ensures that the training of both the generator and the discriminator happens in sync. For implementation, a pre-trained DDIM has been utilized from the famous Hugging face framework commonly referred to as the "diffusers". Using the pre-trained models allows for the transfer learning of the model along with our dataset. The model has been tested on the state-of-the-art extended Yale B dataset for synthetic masks and MAFA dataset for real face mask based performance comparison [74][109]. The same dataset has also been used for the attention-based model discussed in the coming section. The architecture of the patch discriminator is to process the images using a CNN + Dense layer architecture. The output is a 256-d feature embedding. Much like the case of GAN, the discriminator has only been utilized in fine-tuning the model, while the inference of the model deploys only a based generator. 3.5 Choice of the backbone model The backbone model utilized in the generative and attention-based branches was selected based on the requirements of the final deployment of the hardware. This choice of deployment demands the model to be both lightweight and robust to balance the accuracy and efficient inference time. Therefore, the backbone model for transfer learning has been chosen as MobileNetV3 [108]. The architecture of the MobileNet mode has been designed with particular consideration for deployment on mobile devices, such as the embedded hardware utilized in this case, Jetson Xavier AGX. Implementing the MobileNet as a backbone model is also much easier since it is available as part of the TensorFlow API. The weights of the model imported were frozen to ensure the previous layer’s learning was carried along for our custom dataset. The overhead on top of the pre-trained model was the simple ANN for the final 256-d feature embedding. This feature embedding is then utilized for the final comparison with the database to recognize the person 70 in the frame. In the case of multiple faces, the comparison can be scaled to more than a single instance. 3.6 Attention based model The attention-based model branch is responsible for generating faster inference time when deployed on the hardware. This is because these models do not invest any extra time in the generation of an image. The working of the model have been inspired by the ResNet and U-NET models [35,107]. The idea of operation of these models is to add skip connections in order to be to route global information all across the image. A series of residual blocks have been utilized with the input image being added as part of the skip connection to help the model patch the relevant information. A single resiudal block have been shared as follows. Figure 3.8: A single residual block The convolution, batch normalization, and activation layers follow the input layer. In most of the blocks utilized in the proposed network, intra-channel attention has been used by the Max pooling and Average pooling layers to extract the essential and dominant features of the image. The model is well-suited for the open class classification problems since the surveillance and the attendance systems, including the parking ticketing system, can sometimes be re-trained to recognize a new as unknown for generating a ticket to a new person. This has been accomplished by generating a 256-dimensional vector embedding. The geodesic distance between the ground truth and the classified output is optimized using 71 the ArcFace loss function [34]. Using geodesic distance instead of the Euclidean distance allows the optimization of the angle among the feature embedding, which proves to be beneficial in the case of face recognition as the data distribution is complex and multi-dimensional. This is superior to using the softmax based loss function since the separation among the unlike feature embedding is maximized. 3.7 Matching the Feature Embedding from the database A database of all the faces has been maintained. This is not a relational database such as Structured Query Language but rather a local to the host memory block. It allows the point-to-point matching of the generated embedding with the already stored one. The feature embedding labeled with the person’s identity has been stored inside a hash-map that allows linear lookup time that is O[n] time complexity. A point-to-point comparison was made utilizing the Euclidean distance. This is because the semantic correlation among the features has already been carried out, and these features are well within the threshold of the feature embedding of the person being stored in the database. The person is recognized as "unknown" if there is no match. The model can deny access in such cases for applications such as surveillance systems and access to sensitive areas. In contrast, an entry to the database may be made for mildly sensitive applications like parking ticketing systems. Ensuring the person recognizes a new entity is simpler than re-training the model. It is based on the database entries. A few diverse images must be converted into the feature embedding using the same architecture. Any new image added to the database will, therefore, be converged to the geodesic distance within the threshold of the feature embedding. The system can, however, be made more robust in case of the fine-tuning of the model as the new entry is available for the system to recognize. 3.8 Hardware Acceleration The advent of new GPU-accelerated hardware in the form of a computer or an Embedded AI board is essential for the model’s faster training and inference time. The proposed architecture has been trained on the T4 cloud-based GPU offered by Google Colaboratory. The model is then optimized using the quantization and the model pruning techniques. Knowledge distillation is also possible for 72 optimizing a Neural Network, but for the case of generative models, it was preferred to deploy the model as it is. It is more prudent to optimize conventional models using knowledge Distillation. The quantization and pruning have been carried out using TensorFlow, as it forms the brains of the project in terms of programming and practical implementation. 3.8.1 Jetson Xavier AGX The Jetson Xavier AGX is a powerful edge computing platform designed specifically for AI applications. Powered by NVIDIA’s Volta GPU architecture and equipped with multiple processing units, including CPUs, GPUs, and Deep Learning Accelerators (DLAs), it offers unparalleled performance for running complex AI algorithms at the edge. The hardware can be seen in the image below. Figure 3.9: Hardware accelerator used for model deployment; Jetson Xavier AGX It features 5120 CUDA and 640 tensor cores for model deployment and acceleration [110]. Furthermore, a small power requirement of 20 Watts can be quickly supplied if deployed remotely on a node. The hardware works on an Ubuntubased operating system with quite a few profiling options, including NV prof and Nsight systems. On top of these profiling options, we can also utilize GPU Timers and clock64, which are accurate to the precision of even nanoseconds. Another possible option for profiling and yielding the upper bound of the timing requirements is the ’time’ module from Python / C++ may also be utilized, but it uses the CPU and, therefore, the data transfers among the synchronous calls of data 73 transfer and operation performance on GPU may not be profiled efficiently. The key features offered by the accelerator have been shared in the following table. Table 3.2. Key Features of Jetson Xavier AGX Feature Specification GPU Architecture NVIDIA Volta CUDA Cores 5120 Tensor Cores 640 CPU 8-core ARM v8.2 64-bit CPU GPU Memory 32 GB 256-bit LPDDR4x Deep Learning Accelerators (DLAs) 2 Deep Learning Performance 32 TOPS (INT8) Video Encode Up to 4K60 Video Decode Up to 4K60 Camera 16 CSI lanes supporting up to 6 cameras (36 via virtual channels) Connectivity Gigabit Ethernet, 10/100/1000BASE-T Ethernet Peripheral Connectivity PCIe, USB 3.1, SATA, UART, I2C, SPI, GPIO Operating Temperature -25°C to 80°C Power Consumption 10W - 30W Overall, this hardware is one of the best for deep learning based model deployment as it leverages the power of CUDA and Tensor cores along with the minimal power consumption. It can also support camera up to 60 frames per second making it an ideal candidate for real-time application. 3.8.2 Time profilers The time profiling on the hardware has been carried out using the following tools: –nvprof –Nsight systems –GPU Timer –Clock64 All these profilers are accurate up to even nano seconds for precise time profiling. 74 Chapter 4 Results and Discussions The results for each of the section have been shared followed by the time profiling of these models in real-time post deployment. The deployment of these models have been carried out on Jetson Xavier AGX. 4.1 Face Localization stage The results obtained from the lace localization stage have been presented in the previous chapter as part of the performance comparison among the MTCNN and RetinaFace models. This is the first and foremost stage for Masked Facial recognition, as the pipeline depends on this step for background removal and a source of face verification to work only on the Masked Face rather than the extra noise. The results obtained using the Face verification step can be shown as follows. The images were taken from the Extended Yale B dataset, and the synthetic mask was applied using MasktheFace API as part of the data preprocessing step [16,44]. Figure 4.1: Face Verification using Retina Face detector 75 The face verification performance, as presented in the last chapter, could distinguish the faces from the wild to capture the faces even when they are masked. It is a crucial step as the input to both the DDIM and the attention-based model requires the images to contain only the concerned part of the image, the masked face alone. With the validation of the algorithm proven on the Masked Faces, it is essential to establish the performance evaluation of the model on persons of different castes and regions. Although the chosen datasets for the training of the pipeline, including the MAFA dataset and the Extended Yale B, are the benchmark datasets and they contain all the possible diverse examples, the convenience of the reader must depict a small subset of those images and the performance of the retina Face model. A couple of these images have been shared below. Figure 4.2: RetinaFace performance on a subject of younger age bracket It can be verified that there is no effect on the performance of the masked faces; in fact, it is performing equally well on the masked and unmasked faces. The performance boost on the masked faces of the algorithm is its ability to extract facial landmarks by applying the attention-based mechanism. It can be observed that although the facial features have been covered by the mask in the image, the algorithm diverts its attention to the visible dominant features, which are the eyes and still detects the face completely fine. The image above has a resolution of 871 x 850 pixels, and the model takes 3.6908 seconds to extract the face. In parallel processing using CUDA and GPU, the processing time can be reduced to 10x-50x using a shared memory approach. The shared memory allocation is typical of a given block execution, and all the threads inside the blocks share the resources. 76 Chapter 5 Conclusion and Future Research Facial Recognition has been a center of attention in Computer Vision applications for decades. Due primarily to the COVID restrictions in the past, there has been a spike in research and industrial interest in masked facial Recognition. This allows several benefits, including: –A more robust facial detection and recognition system allowing less contact and a smooth work environment –Ideal for surveillance systems –Ticketing and attendance systems in parks and many more Numerous approaches for MFR have been analyzed in this thesis, from simple feature-based face detection to employing generative models and leading up to the attention-based models. The limitations of all these state-of-the-art models have been documented to observe and improve them in the proposed architecture and implementation. A novel approach based on the adversarial training of the DDIM has been proposed, allowing better image synthesis that is semantically coherent with the visible part of the face image. An ensemble model-based approach involving the attention-based model alongside the generative model has been used to allow the user the flexibility of accuracy vs. speed trade-off. For realtime model deployment, an embedded AI hardware accelerator that is Jetson Xavier AGX has been used, allowing the use of CUDA cores for profiling and measuring the time of inference in real time. In the previous architecture, the models proposed could have been more robust for MFR or those developed for the MFR in real-time, which reported a time as high as a few minutes that becomes very slow for real-time operation and decision making. A park queue will be added if the model is that slow. The 83 proposed ensemble model approach provides the implementation of 10-20 seconds for the generative model-based approach and 550-650 milliseconds for the attention-based approach. The time of inference varies depending on the state of the GPU, such as the temperature, the number of operations in the pipeline, and the overhead caused by the operating system. The recorded time is faster than the papers with reported inference time. The architecture also considers the possible background noise and tries to eliminate it by using the face localization stage. The choice of each of these stages has been thoroughly discussed and optimized for implementation. Last but not least, the models have been optimized using post-training quantization and pruning. 5.1 Future Research While this thesis represents a significant contribution to its field, it must acknowledge that it does not claim infallibility. By recognizing these shortcomings, we pave the way for future research to build upon this foundation and further advance our understanding. The scope of future research in this domain of MFR has been listed below. – Privacy Concerns: The datasets used for this thesis included the MAFA and Extended Yale B dataset. The DDIM has been trained on the dataset, providing samples from the distribution of these datasets alone while maintaining diversity. This can potentially lead to a privacy concern if a person’s face can be rendered at any given time and potentially be used without their permission. This needs to be addressed to ensure the encryption and confidentiality of terms in the pipeline. – Conditioning the Diffusion models: The DDIM has been conditioned using adversarial training. Although unstable, adversarial training has been utilized with the addition of noise in the patch discriminator, allowing for better results. If the model can be conditioned without having to train adversarially, it would significantly improve the architecture, decreasing the training time. – More computational resources: The computational resources was limited in this case. Several papers have reported using four stacked V100 GPUs, which is an insane computation power. Training the DDIM on a larger dataset with these powerful GPUs can further improve the pipeline. – Output fusion: The branched pipeline implementation, in this case, was pure because of research-based since the model was intended to be deployed on the hardware. In the case of an actual application developed, the output 84 can be fused, or the choice of the model to be used should reside with the quality of the input image and the application of choice. The output embedding from both the branches can be fused, or the brain of the node itself can make the decision based on the input image. – Improvements in the hardware: Machine learning has seen a tremendous spike in the last decade; manufacturers are keeping up with hardware accelerators to ensure the development phase is caught up with the research phase. In the future, with more powerful hardware devices with better power consumption, the pipeline can be improved to yield better results. 85 Bibliography [1] P. B. Balla and K. T. Jadhao, "IoT Based Facial Recognition Security System," 2018 International Conference on Smart City and Emerging Technology (ICSCET), Mumbai, India, 2018, pp. 1-4, doi: 10.1109/ICSCET.2018.8537344. [2] Ullah, N., Javed, A., Ghazanfar, M. A., Alsufyani, A., Bourouis, S. (2021). A novel DeepMaskNet model for face mask detection and masked facial recognition. Journal of King Saud University - Computer and Information Sciences. Retrieved from www.sciencedirect.com. [3] J. Deng, J. Guo, X. An, Z. Zhu and S. Zafeiriou, "Masked Face Recognition Challenge: The InsightFace Track Report," 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, 2021, pp. 1437-1444, doi: 10.1109/ICCVW54120.2021.00165. [4] M. A. Khan et al., "Performance evaluation of state-of-the-art 2D face recognition algorithms on real and synthetic masked face datasets," Journal of Advanced Research in Applied Sciences and Engineering Technology, vol. 30, no. 2, pp. 225–242, 2023, doi: 10.37934/araset.30.2.225242. [5] Schroff, F., Kalenichenko, D., Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 815–823), doi: https://doi.org/10.48550/arXiv.1503.03832 [6] Y. Taigman, M. Yang, M. Ranzato and L. Wolf, "DeepFace: Closing the Gap to Human-Level Performance in Face Verification," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014, pp. 1701-1708, doi: 10.1109/CVPR.2014.220. [7] Sun, Y., Wang, X., Tang, X. (2014). Deep Learning Face Representation by Joint Identification-Verification. Advances in Neural Information Processing Systems (NIPS). [8] Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In Proceed86 ings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 462-471). [9] Zhang, K., Zhang, Z., Li, Z., Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10), 1499-1503. [10] NVIDA, "Jetson AGX Xavier Developer Kit – User Guide", nvidia.com, December 17, 2019, Available at https://developer.download.nvidia.com/embedded/L4T/ [11] Abhila A.G and Sreeletha S.H. "A Deep Learning Method for Identifying Disguised Faces Using AlexNet and Multiclass SVM." International Research Journal of Engineering and Technology (IRJET) 05, no. 07 (July 2018). Available at www.irjet.net [12] Lee, Seungho. "A Two-Stage Deep Generative Model for Masked Face Synthesis." Sensors 22, no. 20 (2022): 7903. Available at https://doi.org/10.3390/s22207903. [13] Geng, Mengyue, Peixi Peng, Yangru Huang, and Yonghong Tian. "Masked Face Recognition with Generative Data Augmentation and Domain Constrained Ranking." In Proceedings of the 28th ACM International Conference on Multimedia (MM ’20), October 2020, pp. 2246-2254. Available at https://doi.org/10.1145/3394171.3413723. [14] Rushiti, B. (2021). Human-Level Performance and Bayesian Optimal Error. Medium. Retrieved from https://bardhrushiti.medium.com/human-levelperformance-and-bayesian-optimal-error-fadf4f55cd48 In-text citation: (Rushiti, 2021) [15] Liu, Z., Luo, P., Wang, X., Tang, X. (2015). Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015. [16] Anwar, A., Raychowdhury, A. (2020). Masked Face Recognition for Secure Authentication. arXiv preprint arXiv:2008.11104. [17] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 211-252. [18] Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 25 (pp. 1097–1105). Curran Associates, Inc. 87 [19] Ackerman, S., Farchi, E., Raz, O., Zalmanovici, M., Dube, P. (2022). Detection of Data Drift and Outliers Affecting Machine Learning Model Performance Over Time. arXiv preprint arXiv:2012.09258v3 [stat.AP]. [20] Ejaz, M. S., Sifatullah, M. (2019). Implementation of Principal Component Analysis on Masked and Non-masked Face Recognition. In 1st International Conference on Advances in Science, Engineering and Robotics Technology 2019 (ICASERT 2019). Natore, Bangladesh: Bangladesh Army University of Engineering and Technology. [21] Y. Sun, Y. Chen, X. Wang, and X. Tang, “Deep learning face representation by joint identification-verification,” in NeurIPS, 2014 [22] J. Deng, Y. Zhou, and S. Zafeiriou, “Marginal loss for deep face recognition,” in CVPR Workshop, 2017 [23] B.-N. Kang, Y. Kim, B. Jun, and D. Kim, “Attentional feature-pair relation networks for accurate face recognition,” in ICCV, 2019 [24] Y. Kim, W. Park, M.-C. Roh, and J. Shin, “Groupface: Learning latent groups and constructing group-based representations for face recognition,” in CVPR, 2020 [25] Y. Kim, W. Park, and J. Shin, “Broadface: Looking at tens of thousands of people at once for face recognition,” in ECCV, 2020 [26] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680). [27] Farahanipad, F., Rezaei, M., Nasr, M., Kamangar, F., Athitsos, V. (2022). GAN-based Face Reconstruction for Masked-Face. In PETRA ’22: The 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece. DOI: https://doi.org/10.1145/3529190.3534774 [28] Esri. (March 10, 2024). How CycleGAN works. Retrieved from https://developers.arcgis.com/python/guide/how-cyclegan-works [29] J. Zhu, T. Park, P. Isola, A. A. Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. [30] Y. Kossale, M. Airaj and A. Darouichi, "Mode Collapse in Generative Adversarial Networks: An Overview," 2022 8th International Conference on Optimization and Applications (ICOA), Genoa, Italy, 2022, pp. 1-6, doi: 10.1109/ICOA55659.2022.9934291. [31] Bank, D., Koenigstein, N., Giryes, R. (2021). Autoencoders. arXiv preprint arXiv:2003.05991v2. 88 [32] Wang, K., Zhao, B., Peng, X., Zhu, Z., Deng, J., Wang, X., Bilen, H., You, Y. (2022). FaceMAE: Privacy-Preserving Face Recognition via Masked Autoencoders. arXiv preprint arXiv:2205.11090v1. [33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." arXiv preprint arXiv:2010.11929, 2020. [34] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690-4699, 2019. [35] K. He, X. Zhang, S. Ren, and J. Sun. "Deep Residual Learning for Image Recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. [36] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications." arXiv preprint arXiv:1704.04861, 2017. [37] F. Schroff, D. Kalenichenko, and J. Philbin. "FaceNet: A Unified Embedding for Face Recognition and Clustering." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815-823, 2015. [38] Y. Sun, X. Wang, and X. Tang. "Deep Learning Face Representation from Predicting 10,000 Classes." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1891-1898, 2014. [39] J. Deng, Y. Zhou, J. Zhu, P. Hu, and S. Zafeiriou. "WebFace260M: A Benchmark Unfiltered Dataset for Face Recognition in the Wild." arXiv preprint arXiv:2107.00772, 2021. [40] Mengyue Geng, Peixi Peng, Yangru Huang, and Yonghong Tian. 2020. Masked Face Recognition with Generative Data Augmentation and Domain Constrained Ranking. In Proceedings of the 28th ACM International Conference on Multimedia (MM ’20), October 12–16, 2020, Seattle, WA, USA. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3394171.3413723 [41] Yande Li, Kun Guo, Yonggang Lu, and Li Liu. "Cropping and Attentionbased Approach for Masked Face Recognition." Published online: 1 February 2021. Accepted: 26 November 2020. ©The Author(s), under exclusive license to Springer Science+Business Media, LLC part of Springer Nature 2021. 89 [42] Wang Z, Wang G, Huang B, Xiong Z, Hong Q, Wu H, Yi P, Jiang K, Wang N, Pei Y et al (2020) Masked face recognition dataset and application. arXiv:2003.09093 [43] Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv:1411.7923 [44] Georghiades AS, Belhumeur PN, Kriegman DJ (2001) From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6):643–660 [45] Martinez AM, Benavente R Ar face database, 2000 [46] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. "Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks." IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499-1503, 2016. [47] Rucha Golwalkar and Ninad Mehendale. "Masked-face recognition using deep metric learning and FaceMaskNet-21." Applied Intelligence, vol. 52, pp. 13268–13279, 2022. DOI: 10.1007/s10489-021-03150-3. Published online: 25 February 2022. Accepted: 24 December 2021. ©The Author(s), under exclusive license to Springer Science+Business Media, LLC, part of Springer Nature 2022. [48] Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. "Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments." University of Massachusetts, Amherst, Technical Report 07-49, October 2007. [49] Sergey Ioffe and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Proceedings of the 32nd International Conference on Machine Learning (ICML), vol. 37, pp. 448-456, 2015. [50] "Understanding Facial Identification: Face Verification vs. Face Recognition." Shuftipro Blog, https://shuftipro.com/blog/understanding-facialidentification-face-verification-vs-face-recognition/. Available on March 23,2024. [51] Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. "YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information." arXiv:2402.13616v2 [cs.CV], 29 Feb 2024. [52] Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., ... Sackinger, E. (1994). Signature verification using a siamese time delay neural network." Advances in neural information processing systems, 7, 737-744 90 [53] Vasilev, I. (2019). Advanced Deep Learning with Python. Packt Publishing. ISBN-10: 178995617X, ISBN-13: 978-1789956177. [54] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov. "Siamese Neural Networks for One-shot Image Recognition." ICML Deep Learning Workshop, vol. 2, 2015. [55] Chopra, S., Hadsell, R., LeCun, Y. (2005). Learning a Similarity Metric Discriminatively, with Application to Face Verification. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005. IEEE [56] Elad Hoffer and Nir Ailon. "Deep Metric Learning Using Triplet Network." arXiv:1412.6622v4 [cs.LG], 4 Dec 2018. [57] Sarıgöz, Y. "Triplet Loss — Advanced Intro: What are the advantages of Triplet Loss over Contrastive loss, and how to efficiently implement it?" Towards Data Science, Mar 25, 2022. Available [January 10, 2024]. URL: https://towardsdatascience.com/triplet-loss-advanced-intro49a07b7d8905. [58] Al-Sinan, M. R., Al-Mefarej, M. S., Al-Fadhli, A. A., Luqman, H. (2022). Ensemble learning using transformers and convolutional networks for masked face recognition. arXiv, 2210.04816. https://arxiv.org/abs/2210.04816 [59] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020 [60] Wang, Z.; Wang, G.; Huang, B.; Xiong, Z.; Hong, Q.; Wu, H.; Yi, P.; Jiang, K.; Wang, N.; Pei, Y.; et al. Masked face recognition dataset and application. arXiv 2020, arXiv:2003.09093 [61] Ding, F.; Peng, P.; Huang, Y.; Geng, M.; Tian, Y. Masked face recognition with latent part detection. In Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA, 12–16 October 2020; pp. 2281–2289. [62] Ejaz, M.S.; Islam, M.R. Masked Face Recognition Using Convolutional Neural Network. In Proceedings of the 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh, 24–25 December 2019; pp. 1–6. [63] . Mandal, B.; Okeukwu, A.; Theis, Y. Masked Face Recognition using ResNet-50. arXiv 2021, arXiv:2104.08997. 91 [64] Din, N.U.; Javed, K.; Bae, S.; Yi, J. A novel GAN-based network for unmasking of masked face. IEEE Access 2020, 8, 44276–44287. [CrossRef] [65] Karras, T.; Aila, T.; Laine, S.; Lehtinen, J. Progressive growing of gans for improved quality, stability, and variation. arXiv 2017, arXiv:1710.10196 [66] Boutros, F.; Damer, N.; Kirchbuchner, F.; Kuijper, A. Unmasking Face Embeddings by Self-restrained Triplet Loss for Accurate Masked Face Recognition. arXiv 2021, arXiv:2103.01716 [67] Li, S.; Yi, D.; Lei, Z.; Liao, S. The casia nir-vis 2.0 face database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA, 23–28 June 2013; pp. 348–353 [68] Chen, J.; Yi, D.; Yang, J.; Zhao, G.; Li, S.Z.; Pietikainen, M. Learning mappings for face synthesis from near infrared to visual light images. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 156–163. [69] Huang, D.; Sun, J.; Wang, Y. The BUAA-VisNir Face Database Instructions; Tech. Rep. IRIP-TR-12-FR-001; School of Computer Science and Engineering, Beihang University: Beijing, China, 2012; p. 3 [70] Deng, H.; Feng, Z.; Qian, G.; Lv, X.; Li, H.; Li, G. MFCosface: A maskedface recognition algorithm based on large margin cosine loss. Appl. Sci. 2021, 11, 7310. [CrossRef] [71] Deng, H.; Feng, Z.; Qian, G.; Lv, X.; Li, H.; Li, G. MFCosface: A maskedface recognition algorithm based on large margin cosine loss. Appl. Sci. 2021, 11, 7310. [CrossRef] [72] Yi, D.; Lei, Z.; Liao, S.; Li, S.Z. Learning face representation from scratch. arXiv 2014, arXiv:1411.7923 [73] Martinez, A.; Benavente, R. The AR Face Database: CVC Technical Report, 24. 1998. Available online: https://www2.ece.ohiostate.edu/ aleix/ARdatabase.html (accessed on 5 October 2021). [74] Georghiades, A.S.; Belhumeur, P.N.; Kriegman, D.J. From few to many: Illumination Cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 2001, 23, 643–660 [75] Moschoglou, S.; Papaioannou, A.; Sagonas, C.; Deng, J.; Kotsia, I.; Zafeiriou, S. Agedb: The first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 21–26 July 2017; pp. 51–59 92