AttenGait: Gait recognition with attention and rich modalities

AttenGait: gait recognition with attention and rich modalities Francisco M. Castroa,∗, Rub´ en Delgado-Esca˜ noa, Ruber Hern´ andez-Garc´ ıab, Manuel J. Mar´ ın-Jim´ enezc, Nicol´ as Guila aDepartment of Computer Architecture, University of M´ alaga, Spain bFaculty of Engineering Sciences, Universidad Cat´ olica del Maule, Chile cDepartment of Computing and Numerical Analysis, University of C´ ordoba, Spain Abstract Current gait recognition systems employ different types of manual attention mechanisms, like horizontal cropping of the input data to guide the training process and extract useful gait signatures for people identification. Typically, these techniques are applied using silhouettes as input, which limits the learning capabilities of the models. Thus, due to the limited information provided by silhouettes, state-of-the-art gait recognition approaches must use very simple and manually designed mechanisms, in contrast to approaches proposed for other topics such as action recognition. To tackle this problem, we propose AttenGait, a novel model for gait recognition equipped with trainable attention mechanisms that automatically discover interesting areas of the input data. AttenGait can be used with any kind of informative modalities, such as optical flow, obtaining state-of-the-art results thanks to the richer information contained in those modalities. We evaluate AttenGait on two public datasets for gait recognition: CASIAB and GREW; improving the previous state-of-the-art results on them, obtain- ∗Corresponding author Preprint submitted to Pattern Recognition ing 95.8% and 70.7% average accuracy, respectively. Code will be available at https://github.com/fmcp/attengait. Keywords: Gait, optical flow, deep learning, attention, biometrics. 1. Introduction Gait refers to the walking style of a person. It can be considered a biometric feature that, like fingerprints or iris, can be used to identify people by the way they walk. In contrast to fingerprintor iris-based recognition approaches, gait recognition is considered non-invasive since it is performed at a distance and does not require the cooperation of the subject that has to be identified. Therefore, great effort has been put into gait recognition [1] in the last years. Although there exist approaches using different input modalities, such as optical flow [2], inertial sensors [3], gray images [4], or even multiple modalities at the same time [5], most of the recent methods rely on silhouettes [6,7,8,9]. This decision goes against the intuition of other computer vision problems that try to use the richest possible inputs [10,11]. The gait recognition community supporting the use of silhouettes justify them for privacy reasons and in order to force models to learn from gait trait and not from textures (i.e. visual appearance) obtained from clothing, for example. Recent gait recognition approaches [6,7,9] based on silhouettes propose neural models with a reduced number of parameters but producing a huge number of activations to represent the same input in many different ways. Thus, are these models really learning to identify people by focusing on the subtle patterns of the gait? or just learning to produce many repetitive features? In our humble opinion, this is a way to get around the enormous limitation imposed by silhouettes since data only contains binary values and, in addition, their quality highly depends on the silhouette extractor, as we show in Sec. 4. Nevertheless, in a real-life scenario 2 where the background changes over time, this decision gives rise to the following research questions: what would be the performance of silhouette-based models if the silhouette extractor failed? or even thinking further, could another modality be used? Let us imagine a real-life environment, which is far from the ideal laboratory conditions seen in common gait recognition datasets [12,13,14], where we need to identify people by their way of walking. Will a model be penalized for achieving high accuracy using non-silhouette modalities, and if so, what might be the motivation behind this decision? One could think about privacy since the dataset is distributed with raw videos where subjects can be easily identified or the information codified in model weights could be employed to recover subjectsensitive data [15]. Another problem of raw videos is the domain shift from gait recognition to person re-identification since models could focus on unrelated details (i.e. faces, textures, clothes, etc.) to the gait. While we acknowledge these limitations, another kind of data such as optical flow can be used since it guarantees subject anonymity by removing appearance details but having data much richer than silhouettes. To answer all those questions, this paper proposes AttenGait, an attention network for gait recognition that can use rich input data such as optical flow. In addition, we provide thorough experimentation to justify using other modalities instead of the traditional silhouettes and, hopefully, inspire the gait recognition community to think about using different input data. In contrast to previous works that rely just on traditional 2D/3D CNN models [6,7,9], AttenGait is inspired by Transformers [16,11] and their attention mechanism to focus along time on essential regions of the body from a gait recognition perspective, as shown in Fig. 1. Therefore, the model should be able to abstract from items, such as bags or cloth3 Gait Signature h w AttenGait h w LL Figure 1: Overview. AttenGait is a novel architecture that receives a sequence of optical flow maps derived from a walking person to generate a gait signature, considering the importance of the different regions of the input data. L,w, and hrepresent the number of frames, width, and height of the input sample, respectively. ing, and pay more attention to other elements, such as arms or legs swinging during walking. To achieve this goal, we propose three different attention blocks as shown in Fig. 2: an Attention conv to discover essential areas without considering the temporal information, a Spatial Attention HPP takes horizontal crops of the input data that are weighted according to their spatial importance, and a Temporal Attention HPP that locates important horizontal regions according to their temporal information. As pointed out in previous works on video transformers [11], these kinds of models require a massive amount of training data to achieve good performance. In our case, since all public datasets are relatively small compared to image or action recognition datasets, we have to escape from silhouettes and use richer input data such as optical flow. This way, our model receives more information with a wider range of values instead of just binary images, which allows us to train it better. The main contributions of this paper can be summarized in: (i) a novel gait recognition model equipped with three novel trainable attention mechanisms that 4 automatically learn to focus on essential regions of the input data; (ii) a flexible model in terms of input data that is able to handle different visual modalities; (iii) an experimental study to measure the impact of the different modalities on the final accuracy of the model; and, (iv) state-of-the-art results for CASIA-B and GREW using optical flow as input data. The rest of the paper is organized as follows. We start by reviewing related work in Sec. 2. Then, Sec. 3explains the proposed approach. Sec. 4contains the experiments and results. Finally, we present the conclusions in Sec. 5. 2. Related work Gait recognition. The most commonly used approaches in the literature involve silhouette-based methods [6,7,8,9,17]. GEI [18] represents one of the most straightforward approaches since the whole video sequence is summarized into a single frame containing spatio-temporal information. However, it is sensitive to variations in real-world scenarios. GaitSet [6] uses a random stack of silhouettes where each frame is handled independently to extract features that are finally combined and fed into a Horizontal Pyramid Pooling (HPP) module that extracts non-learned features from 62 splits from the same input. A different approach called GaitPart is presented in [7], where a novel part-based model extracts features from horizontal splits of intermediate convolutional activations. Similarly to [6], each frame is handled independently to build features that are combined and fed into an HPP module. GLN [8] concatenates intermediate convolutional activations to build a huge descriptor that reinforces the discrimination capabilities of the model. To reduce the dimensionality of the output, the authors of this approach propose a compression module attached to the model’s end. Finally, 5 GaitGL [9], which is the current state-of-the-art approach, borrows the idea of split convolutions from [7] and applies it to 3D convolutions together with a simplified version of the HPP proposed in [6]. However, silhouette-based approaches can be subject to drastic variations, such as changes in body shape, clothing, illumination, dynamic backgrounds, or other factors. Other works rely on alternative sensors like accelerometer [3,19] or wave-sensors [20]. Skeleton-based systems have also been employed as an alternative to silhouetteand sensor-based systems thanks to new gait datasets that provide skeleton information [21,14]. However, the results obtained are still far from the other visual modalities like silhouettes. For example, Liao et al. [22] propose a CNN model to deal with 3D joint information. Li et al. [23] go a step forward and propose a joint model that computes silhouette and skeleton information together with the identity of the subjects using that derived information from RGB input data. Teepe et al. [24] decompose the skeleton movement into three components: joints, bones and velocities, which are fed into a multibranch CNN to predict the identity of the subjects. And, Wang et al. [25] propose a transformer-based approach with a new attention block using skeletons as input. Recently, some approaches [26,27] have been proposed using a 3D mesh representation, which can be seen as a combination of silhouettes with 3D pose information to produce a richer representation. This way, it includes binary values and information about the different body parts and their spatial location. Optical flow maps have also been proposed to represent gait [28]. This representation is computationally easy to calculate, although its precision depends on the speed of the object, a non-critical fact in gait recognition. A logical extension of all previous approaches that focus on a single kind of input data is the use of multimodal models. This kind of methods use data from different sources or 6 sensors. For example, Kumar et al. [29] use data from multiple inertial sensors to obtain a 3D-skeleton representation accompanied by video images. Castro et al. [5] builds a CNN model that jointly uses optical flow, depth, and gray images to improve the global gait accuracy of the model. UGaitNet [30] proposes a robust multimodal approach against missing modalities. This way, when an input modality is missed due to external circumstances, the model can produce robust results with the rest of modalities. Liang et al. [31] propose a two-staged multimodal approach that obtains silhouettes from RGB information to feed them to a gait recognition model that identifies the subjects. GaitCopy [4] uses a teacher-student methodology to train a student model that uses gray images to mimic the behavior of a master model trained with optical flow. Attention mechanisms. Attention mechanisms [32] began to gain importance with the advent of vision transformers (ViT) [16], which is a rising topic in the computer vision field since the first adaptation from language processing to image processing. After that first implementation, several approaches have proposed improvements to the original ViT [16], e.g., using a teacher-student scheme to help the training process [33], a better tokenization scheme to produce more robust crops from the input data [34], using a hierarchical scheme of tokens to obtain richer local information [35,36], or adding convolutions to the attention mechanism [37,10] to the best of both worlds. However, in the field of gait recognition, only a few works use attention mechanisms or transformers. For example, [38] uses a simple attention mechanism by manually splitting the input samples into horizontal crops. A further step is proposed in [39], where horizontal and vertical crops are used to improve the learning capabilities of the model. In [40], its authors propose a simple multi7 modal approach by attaching two transformers blocks with self-attention at the end of a traditional CNN architecture. A basic gait recognition approach applying a ViT transformer is presented in [41]. However, its results are not comparable with other approaches since a different data split is used in their experiments. A more elaborated approach is presented in [42], where authors use an initial set of traditional convolutions and HPP to produce a set of gait features fed into a ViT-inspired transformer. Cui et al. [43] proposes a multimodal approach using two well-known encoders for silhouettes [9] and pose information [44] together with two attention modules that fuse both modalities. Fan et al. [45] propose a novel model combining convolutional blocks with transformers blocks at the end. In this work, we propose a novel approach, AttenGait, inspired by transformers and their cropping and attention mechanisms. Our attention mechanism weights feature regions according to their importance for the gait recognition task. Moreover, unlike previous state-of-the-art approaches, AttenGait uses richer information, like optical flow, to learn better gait signatures than binary silhouettes. 3. Proposed model We propose AttenGait which is a network for gait recognition using a trainable attention mechanism different from traditional self-attention [32]. Our model employs a cropping and attention mechanism to build robust gait signatures using rich data from modalities such as optical flow in contrast to traditional silhouettes. This way, these modalities allow to design and apply a more complex model than traditional architectures designed for silhouettes. In this section, we first motivate AttenGait, then we describe its architecture (Sec. 3.1) and, finally, we comment on the training and testing procedures (Sec. 3.2 and Sec. 3.3). 8 Lx16x16x128 7936 15872 Gait Signature 1984 Lx64x64xC Main Branch AC7AC6AC5AC4AC3AC2AC1C1 Common Branch F C 1 F C 2 7936 L Spatial Attention HPP Temporal Attention HPP 2D-C1 2D-C2 2D-C3 2D-C4 2D-C5 AM 3D-C1 3D-C2 3D-C3 3D-C4 3D-C5 AM Figure 2: AttenGait: gait recognition model equipped with trainable attention mechanisms to focus on important regions of the input data. Sequences of L consecutive frames of 64x64xC pixels are fed into a main branch to extract general spatio-temporal features. Then, two branches extract temporal features and spatio-temporal features. After concatenating each branch features (Loperation), a common branch builds the final spatio-temporal gait signatures. ‘ACx’ refers to Attention Convs, ‘2D-Cx’ and ‘3D-Cx’ refers to 2D and 3D convolutions, respectively, and ‘AM’ refers to the Attention Mask. Finally, ‘FCx’ refers to dense layers. Activations shape is included after each block. Motivation. In the last years, state-of-the-art gait recognition approaches [6,7,9] that obtain the best results for different datasets use silhouettes as input and produce very long gait signatures (from 4k to 15k dimensions). Although some approximations [8,46] have been proposed to solve these problems, their results are worse than the state-of-the-art methods. Therefore, we introduce AttenGait, an artificial neural network equipped with cropping and attention mechanisms that can generate shorter and more discriminative gait signatures from richer modalities (e.g. optical flow or gray images, although not limited to them). 9 the test samples and classify them using a simple Nearest Neighbour (NN) classifier. Thus, we follow the typical protocol of gallery and probe sets used in other works [7,9]. During test time, we use the whole sequence as input instead of taking subsequences of Lframes like during the training step. 4. Experiments In this section, we present experimental results with AttenGait. First, we present the datasets and metrics used in our experiments (Sec. 4.1). After that, we provide some implementation details used in our approach (Sec. 4.2). Then, we report the experimental results of AttenGait using different modalities, and later, we compare it to the state-of-the-art (Sec. 4.4), together with an ablation study on the model (Sec. 4.5). 4.1. Datasets and metrics We run our experiments on two of the largest public datasets for gait recognition: CASIA-B [12] and GREW [14]. Datasets that do not release the RGB images or the optical flow, as OU-MVLP [13], cannot be used in our experimental evaluation. We hope that the authors release them in the near future. CASIA-B [12]. In this dataset, 124 subjects walk in an indoor environment while they are recorded from 11 viewpoints (i.e. from 0◦to 180◦in steps of 18◦) with a video resolution of 320 ×240 pixels and 24 fps. Three walking conditions are considered: normal walking (NM), carrying a bag (BG), and wearing a coat (CL). In our experiments, we follow the experimental protocol defined in [18]. GREW [14]. This is the first large-scale dataset for gait recognition in the wild, composed of videos that contain hundreds of cameras and thousands of hours of 16 streams in open environments. The dataset includes 26,345 subjects and 128,671 sequences, 14,185,478 human boxes and a distractor set with 233,857 sequences. Moreover, it provides several modalities: silhouettes, Gait Energy Images (GEIs), optical flow, and 2D and 3D poses. In this case, we follow the experimental protocol explained in [14], and we provide the results obtained from Codalab 1. Metrics. To evaluate the system performance, we use the standard Rank-1 (R1) accuracy, i.e. the percentage of correctly classified videos: R1 = #correct/#total. 4.2. Implementation details. Input data Following the recommendations of previous state-of-the-art works [7, 9], during training, we set the maximum temporal length of the input samples to L= 30 frames to reduce the memory requirements, while at test time, we use all frames. To reduce the computation, we resize the input data to 85 ×64 pixels, keeping the original aspect ratio. The unnecessary background is removed by cropping the frames to 64 ×64, keeping the total height. Therefore, the shape of our input data is 64 ×64 ×C×Lframes, where Cis the number of channels of the input data and Lrefers to the number of frames of the sample, being C= 2 for optical flow (xand ycomponents), C= 1 for silhouettes and gray images, and C= 3 for optical flow represented as RGB images, which is the optical flow included in GREW dataset. In this paper, we use all visual modalities included in each dataset. This way, we use optical flow computed with Farneback [47], gray images, and silhouettes for CASIA-B. For GREW, we use optical flow represented as RGB images and 1GREW test results: https://codalab.lisn.upsaclay.fr/competitions/ 3409#results 17 silhouettes. Therefore, AttenGait was tested on two datasets, three data modalities, and two different methods for obtaining the same modality (i.e. optical flow), demonstrating its robustness. Architecture. AttenGait is inspired by the well-known architecture called GaitSet, proposed in [6] but extended to make all layers trainable. Moreover, we use attention mechanisms inspired by visual transformers [16]. As shown in Fig. 2, our model is composed of a main branch with an initial convolutional layer (C1) followed by seven Attention convs (AC1, ..., AC7) with SiLU activation function, attention parameters kw= 3 and kh= 3, and a Max Pooling of 2×2after AC3 and AC5. The details about each convolutional layer are defined as follows. C1: conv layer of 5×5and 32 filters; AC1, AC2: 3×3and 32 filters; AC3, AC4: 3×3and 64 filters; AC5, AC6, AC7: 3×3and 128 filters. For GREW dataset, we include two additional Attention convs after AC2, AC4, and AC6 using the same setup as its previous layer. The Spatial Attention HPP comprises five 2D convolutions (2D-C1, ..., 2D-C5) with kernel shape d×16 with SiLU. The Temporal Attention HPP is composed of five 3D convolutions (3D-C1, ..., 3D-C5) with kernel shape d×d×16 with SiLU. For both cases dtakes the following values: [1,2,4,8,16]. In this way, we build a trainable HPP based on convolutions with different kernel heights. Finally, to deal with different temporal lengths during test time, the exceeding temporal information is compressed using the sum of the mean and max values across the temporal axis as shown in Eq. 5. G(·) = mean(·) + max(·),(5) where G(·)are the summarized activations. The last common branch is composed of two dense layers with 128 and 32 units per pyramidal partition, respectively. The final gait signature used to identify 18 the subjects has 1984 dimensions. Hyperparameters. We implement AttenGait using Keras and TensorFlow 2.12. During training, we started from scratch using Adam with a minibatch of 64 samples for CASIA-B and 48 samples for GREW. To build our batch samples, we follow the p×kwidely used strategy for Triplet loss, with p= 8 classes and k= 8 samples per class for CASIA-B. For GREW, we use p= 12 and k= 4. The learning rate starts at 0.001 and is reduced by 0.1during the final fine-tuning (see below) using only Triplet loss. To avoid overfitting, we apply L2 regularization in all layers and dropout of 10% in the attention masks. The model is trained during 8000 epochs + 1000 fine-tuning epochs for CASIA-B and 2000 epochs + 500 epochs for GREW. For training CASIA-B, we use 2 NVIDIA A100 for eight hours and 8 NVIDIA A100 for seven days for GREW. To predict the identities of the subjects, we use a NN classifier with the gait signatures obtained from the last dense layer of the model. 4.3. Modalities comparison In this experiment, we measure the impact of the different modalities on the final accuracy of AttenGait. Based on average accuracy, silhouettes obtain the lowest result, followed by gray/RGB images and optical flow. Tab. 1summarizes the results for gray, RGB, silhouettes, and optical flow (OF) modalities on CASIAB. These results make sense since silhouettes contain much less information than the other modalities. Focusing on gray and RGB images, both modalities achieve similar performance, although RGB encodes more information. In our opinion, RGB is not necessary for gait recognition since models need to get away from appearance details like clothes, body shape, hairstyle, etc., which are the details where colors play an important role. Therefore, using gray-scale images is more 19 Modality NM BG CL Avg RGB 98.8 98.2 88.0 95.0 Gray 98.8 98.3 88.1 95.1 Silhouettes 95.1 90.5 71.3 85.6 OF 98.8 97.7 91.0 95.8 Table 1: Modality study on CASIA-B. Comparison between AttenGait models trained with different modalities. Each row represents a different modality. Each column represents a different scenario of CASIA-B. It is reported the Rank-1 accuracy (%) excluding identical-view cases. The best results are marked in bold. Modality Rank-1 Rank-5 Rank-10 Rank-20 Silhouettes 50.2 67.3 73.4 77.9 OF 70.7 82.9 86.9 89.2 Table 2: Modality study on GREW. Comparison between AttenGait models trained with different modalities. Each row represents a different modality. Each column represents a different Rank-nidentification rate (%). suitable and faster. Comparing optical flow and gray images, in that case, the most significant difference happens in the CL scenario where raw pixels can penalize the performance of the model due to the considerable differences in the visual appearance of the subjects due to changes in the clothes. This way, optical flow is less penalized by those changes and can achieve better results than raw pixels. The results of the GREW dataset (Tab. 2) show the same behaviour identified in CASIA-B. Note that in this case, only optical flow and silhouettes are available. As expected, optical flow achieves the best results for all Rank-n, showing the benefits of using better modalities with richer information. Silhouettes obtain lower results than the other modalities but better than well-known silhouette-based state-of-the-art approaches like GaitSet [6] or GaitPart [7] (shown in Tab. 4). 20 4.4. State-of-the-art comparison To put AttenGait results in context, we compare them with existing methods on two widely used datasets: CASIA-B and GREW. Note that our AttenGait results are obtained using optical flow as input. CASIA-B. Case Method 0 18 36 54 72 90 108 126 144 162 180 Avg NM GaitGL [9] 96.0 98.3 99.0 97.9 96.9 95.4 97.0 98.9 99.3 98.8 94.0 97.4 CSTL [48] 97.2 99.0 99.2 98.1 96.2 95.5 97.7 98.7 99.2 98.9 96.5 97.8 TransGait [40] 97.3 99.6 99.7 99.0 97.1 95.4 97.4 99.1 99.6 98.9 95.8 98.1 GaitStrip [39] 96.0 98.4 98.8 97.9 96.6 95.3 97.5 98.9 99.1 99.0 96.3 97.6 GaitEdge [31] 97.2 99.1 99.2 98.3 97.3 95.5 97.1 99.4 99.3 98.5 96.4 97.9 GaitGCI [49] 97.3 98.6 99.2 98.2 97.3 95.7 97.1 99.2 99.0 99.1 96.8 97.9 GaitCoTr [42] 97.3 98.8 99.3 98.3 97.3 97.2 98.3 98.9 99.5 98.8 92.7 97.9 DANet [50] 96.4 99.1 99.2 98.2 96.6 95.5 97.6 99.4 99.5 99.3 96.9 98.0 MMGaitFormer [43] 98.1 98.6 99.0 98.1 98.4 97.8 98.1 99.0 99.2 99.1 97.3 98.4 AttenGait-OF (ours) 98.4 98.2 98.6 98.1 98.7 99.0 98.9 99.2 99.3 99.3 99.3 98.8 AttenGaitOF+Gray (ours) 99.2 99.2 99.1 99.2 98.8 99.0 99.2 99.4 99.3 99.2 98.9 99.1 AttenGaitOF+Sil (ours) 99.1 99.1 99.2 99.2 98.6 98.7 99.1 99.3 99.1 99.4 99.4 99.1 BG GaitGL [9] 92.6 96.6 96.8 95.5 93.5 89.3 92.2 96.5 98.2 96.9 91.5 94.5 CSTL [48] 91.7 96.5 97.0 95.4 90.9 88.0 91.5 95.8 97.0 95.5 90.3 93.6 TransGait [40] 94.0 97.1 96.5 96.0 93.5 91.5 93.6 95.9 97.2 97.1 91.6 94.9 GaitStrip [39] 92.8 96.6 97.2 96.5 95.2 90.5 93.5 97.5 98.3 97.6 91.4 95.2 GaitEdge [31] 95.3 97.4 98.4 97.6 94.3 90.6 93.1 97.8 99.1 98.0 95.0 96.1 GaitGCI [49] 93.2 96.8 97.6 96.2 93.9 90.5 93.7 96.8 98.3 97.2 91.7 95.0 GaitCoTr [42] 87.7 98.4 97.8 96.0 94.7 92.1 93.7 96.5 97.7 97.5 92.5 95.0 DANet [50] 95.0 97.3 98.3 97.4 94.7 91.0 93.9 97.4 98.2 97.6 94.2 95.9 MMGaitFormer [43] 97.1 95.9 97.1 95.7 96.1 95.2 95.2 97.1 97.3 96.1 93.5 96.0 AttenGait-OF (ours) 96.9 97.9 98.3 98.2 97.2 95.4 97.2 98.7 98.8 98.6 97.4 97.7 AttenGaitOF+Gray (ours) 99.2 99.0 98.9 99.0 98.4 98.4 98.8 99.4 99.4 99.3 98.9 99.0 AttenGaitOF+Sil (ours) 98.1 98.5 99.1 98.4 96.6 96.0 97.9 99.0 99.2 99.1 99.2 98.3 CL GaitGL [9] 76.6 90.0 90.3 87.1 84.5 79.0 84.1 87.0 87.3 84.4 69.5 83.6 CSTL [48] 78.1 89.4 91.6 86.6 82.1 79.9 81.8 86.3 88.7 86.6 75.3 84.2 TransGait [40] 80.1 89.3 91.0 89.1 84.7 83.3 85.6 87.5 88.2 88.8 76.6 85.8 GaitStrip [39] 79.9 92.3 93.4 89.2 86.0 80.0 86.0 88.5 91.7 87.5 73.5 86.2 GaitEdge [31] 84.3 92.8 94.3 92.2 84.6 83.0 83.0 87.5 87.4 85.9 75.0 86.4 GaitGCI [49] 81.1 91.3 93.2 90.4 85.7 80.6 87.1 88.3 89.3 87.3 75.5 86.4 GaitCoTr [42] 84.4 93.0 95.1 92.9 87.9 85.7 88.1 90.6 92.4 89.8 84.0 89.4 DANet [50] 82.8 94.8 96.9 94.3 89.0 83.9 87.9 92.3 95.1 92.0 80.3 89.9 MMGaitFormer [43] 93.9 98.0 96.9 96.0 93.7 91.6 93.5 96.4 96.5 95.7 90.2 94.8 AttenGait-OF (ours) 91.1 95.3 96.0 95.3 89.9 88.4 89.5 91.3 88.8 89.4 86.0 91.0 AttenGaitOF+Gray (ours) 96.0 97.0 97.7 97.3 94.6 92.9 93.0 94.8 91.1 91.4 88.6 94.0 AttenGaitOF+Sil (ours) 92.8 95.5 95.8 94.2 91.6 88.8 91.1 92.8 91.2 92.0 87.4 92.1 Table 3: State of the art on CASIA-B. Rank-1 identification rate (%) excluding identicalview cases. The best single-modality results are marked in bold. The best multimodal results are underlined. Methods in italics use multiple modalities. According to the results shown in Tab. 3, AttenGait obtains the best results for NM,BG and CL scenarios, achieving an average Rank-1 accuracy of 95.8% among 21 NM,BG and CL cases, while previous single modality state-of-the-art approaches like GaitGCI [49], GaitCoTr [42], and DANet [50] achieved 93.1%,94.1%, and 94.6%, respectively. Comparing the results with the multimodal approach MMGaitFormer [43], we achieve comparable results even though we use a single modality, and it only improves the CL scenario thanks to the multimodal setup. To perform a fair comparison, we propose a simple multimodal approach using a late fusion that performs a weighted sum [5] of each single modality model. Note that, we compute the L2 distance for each probe sample to all gallery samples, and then, we compute the inverse of those distances to obtain the probability distributions that are used for the late fusion. After thorough cross-validation, we have selected the weights [0.6,0.4] for both combinations of modalities (OF+Gray and OF+Silhouettes). With both fusions, our results improve, and on average, we obtain the best multimodal results with 97.4% and 96.5% for OF+Gray and OF+Sil, respectively, while MMGaitFormer obtained 96.4%. Focusing on the different viewpoints, AttenGait obtains very stable results among cameras, without big drops in the accuracy for difficult cameras such as 0or 180, which are appreciated on other approaches. In our case, NM results are practically the same among cameras, oscillating by ±0.5% with respect to the average. Similar behavior is observed for BG where only the lateral view (90), which is one of the most affected by carrying objects, shows a drop of 2.3% with respect to the average. Finally, CL is the most challenging case since the subjects wear long coats and heavy clothes producing many occlusions depending on the viewpoint. Our results show more variations among cameras, especially in the back viewpoints (from 90 to 180) with differences of up to 5% in the most challenging case with the camera 180. 22 Method Rank-1 Rank-5 Rank-10 Rank-20 GaitSet [6] 46.3 63.6 70.3 76.8 GaitPart [7] 44.0 60.6 67.3 73.5 GaitGL [9] 47.3 63.6 69.3 74.2 CSTL [48] 50.6 65.9 71.9 76.9 GaitCoTr [42] 55.6 70.9 76.2 80.4 TransGait [40] 56.3 72.7 78.1 82.5 GaitGCI [49] 68.5 80.8 84.9 87.7 AttenGait-OF (ours) 70.7 82.9 86.9 89.2 Table 4: State of the art on GREW. Rank-nidentification rate (%). The best results are marked in bold. GREW. AttenGait achieves the best results for all rank-naccuracies as shown in Tab. 4. Note that results for GaitSet and GaitPart are obtained from [14]. Since this dataset comprises many more samples than CASIA-B, our model can improve the state-of-the-art by a large margin. This is especially clear for Rank-1 accuracy, where AttenGait improves previous approaches by 2.2%. As the Rank level increases, accuracy improves but less than other approaches since our Rank-1 value is already noticeably higher than previous approaches. 4.5. Ablation study In this section, we evaluate the behavior of AttenGait when different modifications are applied to the final model: baseline,Spatial Attention HPP,Temporal Attention HPP, and AttenGait without final fine-tuning. Note that in these experiments, for brevity, we present results only for CASIA-B and using optical flow as input. Results are obtained using a NN classifier. The final fine-tuning is applied only in the row ‘AttenGait’, which is our final version. Baseline. Here, we train a model removing attention mechanisms, just keeping the convolutional and HPP layers without attention masks. Thus, the architecture 23 Experiments NM BG CL Avg Baseline 98.2 95.3 82.3 91.9 Baseline w/ Attention+Spatial Attention HPP 98.6 97.6 88.8 95.0 Baseline w/ Attention+Temporal Attention HPP 98.7 97.4 89.9 95.3 AttenGait w/o ft 98.8 97.4 90.8 95.6 AttenGait 98.8 97.7 91.0 95.8 Table 5: Ablation study on CASIA-B. Comparison between the full AttenGait and different modifications. Each row represents a different version. Each column represents a different scenario of CASIA-B. The Rank-1 identification rate (%) is reported excluding identical-view cases. is composed of the same blocks of Fig. 2without attention masks. This can be considered as the baseline obtained with the basic version of AttenGait without attention mechanisms. Row ‘Baseline’ of Tab. 5contains the results of this experiment. Comparing the average result with our final model (row ‘AttenGait w/o ft’), the contribution of our attention mechanism is clear, obtaining an average improvement of 3.7%. If we focus on the different scenarios, CL gets the highest boost in accuracy (8.5%) followed by ‘BG’ (2.1%), which shows the capabilities of our attention mechanism to deal with changes in the walking conditions. Spatial Attention HPP. In this case, the architecture contains Attention Convs and the Spatial Attention HPP, removing the Temporal Attention HPP. The results of this experiment can be found in row ‘Baseline w/ Attention+Spatial Attention HPP’ of Tab. 5. Comparing the results with the Temporal Attention HPP (row ‘Baseline w/ Attention+Temporal Attention HPP’), the results are lower on average and for NM and CL scenarios. However, for the BG scenario, the model using only spatial information achieves better accuracy with an improvement of (0.2%). Therefore, spatial information allows the model to deal with small changes in 24 walking conditions like carrying bags. Temporal Attention HPP. In this case, Attention Convs and Temporal Attention HPP are kept in the architecture to asses a different way of dealing with spatiotemporal information. Row ‘Baseline w/ Attention+Temporal Attention HPP’ of Tab. 5contains the results of this experiment. In this case, the average results are better (0.3%) than the obtained by the Spatial Attention HPP (row ‘Baseline w/ Attention+Spatial Attention HPP’). Moving to the different scenarios, CL obtains a remarkable improvement of (1.1%) while the NM scenario obtains similar results. Thus, spatio-temporal information seems beneficial for difficult situations with extreme changes in the shape of the subjects. AttenGait without fine-tuning. We evaluate the performance of applying the final version of AttenGait, including all modules but without the fine-tuning process explained above. Thus, this experiment allows us to measure the impact of having both Attention HPP modules simultaneously. The results of this experiment can be found in row ‘AttenGait w/o ft’ of Tab. 5. Comparing the results with the obtained using only a kind of Attention HPP (rows ‘Baseline w/ Attention+Spatial Attention HPP’ and ‘Baseline w/ Attention+Temporal Attention HPP’), this version improves all scenarios and the average accuracy, taking the best features from each kind of Attention HPP module to produce better gait signatures. Finally, if we include the fine-tuning process (row ‘AttenGait’), the final average accuracy is even better (95.8% vs 95.6%) like for BG (97.7% vs 97.4%) and CL (91.0% vs 90.8%) scenarios. 25 [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021. 3,7,18 [17] C. Fan, J. Liang, C. Shen, S. Hou, Y. Huang, and S. Yu, “Opengait: Revisiting gait recognition towards better practicality,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9707–9716. 5 [18] Z. Wu, Y. Huang, L. Wang, X. Wang, and T. Tan, “A comprehensive study on cross-view gait based human identification with deep CNNs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 2, pp. 209–226, 2017. 5,16 [19] R. Delgado-Esca˜ no, F. M. Castro, J. R. C´ ozar, M. J. Mar´ ın-Jim´ enez, N. Guil, and E. Casilari, “A cross-dataset deep learning-based classifier for people fall detection and identification,” Computer Methods and Programs in Biomedicine, vol. 184, p. 105265, 2020. 6 [20] Z. Meng, S. Fu, J. Yan, H. Liang, A. Zhou, S. Zhu, H. Ma, J. Liu, and N. Yang, “Gait recognition for co-existing multiple people using millimeter wave sensing,” in Proceedings of the AAAI conference on artificial intelligence, 2019. 6 [21] W. An, S. Yu, Y. Makihara, X. Wu, C. Xu, Y. Yu, R. Liao, and Y. Yagi, “Performance evaluation of model-based gait on multi-view very large pop32 ulation database with pose sequences,” IEEE transactions on biometrics, behavior, and identity science, vol. 2, no. 4, pp. 421–430, 2020. 6 [22] R. Liao, S. Yu, W. An, and Y. Huang, “A model-based gait recognition method with body pose and human prior knowledge,” Pattern Recognition, vol. 98, p. 107069, 2020. 6 [23] X. Li, Y. Makihara, C. Xu, Y. Yagi, S. Yu, and M. Ren, “End-to-end modelbased gait recognition,” in Proceedings of the Asian Conference on Computer Vision, 2020. 6 [24] T. Teepe, J. Gilg, F. Herzog, S. H¨ ormann, and G. Rigoll, “Towards a deeper understanding of skeleton-based gait recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1569–1577. 6 [25] L. Wang, J. Chen, and Y. Liu, “Frame-level refinement networks for skeleton-based gait recognition,” Computer Vision and Image Understanding, vol. 222, p. 103500, 2022. 6 [26] J. Zheng, X. Liu, W. Liu, L. He, C. Yan, and T. Mei, “Gait recognition in the wild with dense 3D representations and a benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 6 [27] C. Xu, Y. Makihara, X. Li, and Y. Yagi, “Occlusion-aware human mesh model-based gait recognition,” IEEE Transactions on Information Forensics and Security, 2023. 6 33 [28] F. M. Castro, M. J. Mar´ ın-Jim´ enez, N. Guil, S. L´ opez-Tapia, and N. P. de la Blanca, “Evaluation of CNN architectures for gait recognition based on optical flow maps,” in International Conference of the Biometrics Special Interest Group, 2017, pp. 251–258. 6 [29] P. Kumar, S. Mukherjee, R. Saini, P. Kaushik, P. P. Roy, and D. P. Dogra, “Multimodal gait recognition with inertial sensor data and video using evolutionary algorithm,” IEEE Transactions on Fuzzy Systems, vol. 27, no. 5, pp. 956–965, 2018. 7 [30] M. J. Mar´ ın-Jim´ enez, F. M. Castro, R. Delgado-Esca˜ no, V. Kalogeiton, and N. Guil, “Ugaitnet: Multimodal gait recognition with missing input modalities,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 5452–5462, 2021. 7 [31] J. Liang, C. Fan, S. Hou, C. Shen, Y. Huang, and S. Yu, “GaitEdge: Beyond plain end-to-end gait recognition for better practicality,” in European Conference on Computer Vision, 2022. 7,21 [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017. 7,8 [33] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J´ egou, “Training data-efficient image transformers & distillation through attention,” in International Conference on Machine Learning, 2021, pp. 10 347–10 357. 7 34 [34] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 7 [35] K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao, “Uniformer: Unified transformer for efficient spatiotemporal representation learning,” in International Conference on Learning Representations, 2022. 7 [36] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 7 [37] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “Cvt: Introducing convolutions to vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 7 [38] Y. Zhang, Y. Huang, S. Yu, and L. Wang, “Cross-view gait recognition by discriminative feature learning,” IEEE Transactions on Image Processing, 2019. 7 [39] M. Wang, B. Lin, X. Guo, L. Li, Z. Zhu, J. Sun, S. Zhang, Y. Liu, and X. Yu, “Gaitstrip: Gait recognition via effective strip-based feature representations and multi-level framework,” in Proceedings of the Asian Conference on Computer Vision, 2022. 7,21 [40] G. Li, L. Guo, R. Zhang, J. Qian, and S. Gao, “Transgait: Multimodal-based gait recognition with set transformer,” Applied Intelligence, pp. 1–13, 2022. 7,21,23 35 [41] J. N. Mogan, C. P. Lee, K. M. Lim, and K. S. Muthu, “Gait-vit: Gait recognition with vision transformer,” Sensors, vol. 22, no. 19, p. 7362, 2022. 8 [42] J. Li, Y. Zhang, H. Shan, and J. Zhang, “Gaitcotr: Improved spatial-temporal representation for gait recognition with a hybrid convolution-transformer framework,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, 2023. 8,21,22,23 [43] Y. Cui and Y. Kang, “Multi-modal gait recognition via effective spatialtemporal feature fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 8,21,22 [44] T. Teepe, A. Khan, J. Gilg, F. Herzog, S. H¨ ormann, and G. Rigoll, “Gaitgraph: Graph convolutional network for skeleton-based gait recognition,” in IEEE International Conference on Image Processing, 2021. 8 [45] C. Fan, S. Hou, Y. Huang, and S. Yu, “Exploring deep models for practical gait recognition,” arXiv preprint arXiv:2303.03301, 2023. 8 [46] J. Liang, C. Fan, S. Hou, C. Shen, Y. Huang, and S. Yu, “Gaitedge: Beyond plain end-to-end gait recognition for better practicality,” in European Conference on Computer Vision. Springer, 2022, pp. 375–390. 9 [47] G. Farneb¨ ack, “Two-frame motion estimation based on polynomial expansion,” in Image Analysis: 13th Scandinavian Conference, vol. 2749, 2003, pp. 363–370. 17 [48] X. Huang, D. Zhu, H. Wang, X. Wang, B. Yang, B. He, W. Liu, and B. Feng, “Context-sensitive temporal feature learning for gait recognition,” in Pro36 ceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 909–12 918. 21,23 [49] H. Dou, P. Zhang, W. Su, Y. Yu, Y. Lin, and X. Li, “Gaitgci: Generative counterfactual intervention for gait recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 21,22,23 [50] K. Ma, Y. Fu, D. Zheng, C. Cao, X. Hu, and Y. Huang, “Dynamic aggregated network for gait recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 21,22 [51] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2017, pp. 618–626. 26 [52] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv, 2018. 26 37