UGaitNet: Multimodal Gait Recognition With Missing Input Modalities

JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 UGaitNet: Multimodal gait recognition with missing input modalities Manuel J. Marin-Jimenez, Member, IEEE, Francisco M. Castro, Ruben Delgado-Esca˜ no, Vicky Kalogeiton, and Nicolas Guil Abstract—Gait recognition systems typically rely solely on silhouettes for extracting gait signatures. Nevertheless, these approaches struggle with changes in body shape and dynamic backgrounds; a problem that can be alleviated by learning from multiple modalities. However, in many real-life systems some modalities can be missing, and therefore most existing multimodal frameworks fail to cope with missing modalities. To tackle this problem, in this work, we propose UGaitNet, a unifying framework for gait recognition, robust to missing modalities. UGaitNet handles and mingles various types and combinations of input modalities, i.e. pixel gray value, optical flow, depth maps, and silhouettes, while being camera agnostic. We evaluate UGaitNet on two public datasets for gait recognition: CASIA-B and TUM-GAID, and show that it obtains compact and state-of-the-art gait descriptors when leveraging multiple or missing modalities. Finally, we show that UGaitNet with optical flow and grayscale inputs achieves almost perfect (98.9%) recognition accuracy on CASIA-B (same-view “normal”) and 100% on TUM-GAID (“ellapsed time”). Code will be available at https:// github.com/ avagait/ ugaitnet Index Terms—Gait, multimodal, deep learning, biometrics. F 1 INTRODUCTION Like fingerprints or handwritten signatures, gait is another kind of biometric feature that allows for people identification. Its main advantages are that it does not require the collaboration of the subject and it can be performed at certain distance. Therefore, over the past decades, great effort has been put in gait recognition [1], [2], [3], [4]. Typical approaches use a single modality, or input data type, such as silhouettes [5], Gait Energy Images [6], optical flow [7], etc. Following a complementary line of research, recent works [4], [8], [9], [10], [11] explore the use of multiple modalities, showing that there are cases where these multimodal approaches improve upon the single-modality baselines. Nowadays, multimodal systems can benefit from the fact that is easy to find devices with multiple sensors and hence multiple output modalities (e.g. Microsoft Kinect, mobile phones, etc.) or techniques able to produce different kinds of data like depth [12], optical flow [13] or skeletons [14]. This makes the study of multimodal systems easier and more attractive. Nevertheless, despite their remarkable results [4], [10], the main common limitation of these multimodal systems is their inability to handle missing modalities. They require all modalities at the same time and therefore, they cannot be used in cases where one or more modalities are missing, due to sensors’ failure or unavailability. For instance, let us consider an heterogeneous setup for surveillance, e.g. in a •MJ Marin-Jimenez is with IMIBIC and the Department of Computing and Numerical Analysis, University of Cordoba, Cordoba, Spain, 14011. E-mail: mjmarin AT uco DOT es •F. Castro, R. Delgado and N. Guil are with the Department of Computer Architecture, University of Malaga, Malaga, Spain. •V. Kalogeiton is with the Computer Science Laboratory, Ecole Polytechnique, France. Manuscript received xxx, 2020; revised xxx, 2020. large industrial facility, where not all available devices produce the same kind of data (e.g. some provide RGB, others are event cameras, other are infrared cameras). Depending on the location of the subject, there will be moments where we can use only a subset of the modalities. However, we still want systems able to recognize the target subjects when they move from different areas of the surveilled scenario. This is not feasible with previous single or multimodal systems, and hence it is the main topic of study of this work. In this paper, we propose UGaitNet, a multimodal network for gait recognition using a variable number of input modalities. In contrast to previous methods [8], [10], [15], UGaitNet is a single network that handles and combines various input modalities and is robust to missing modalities. In that way, the modalities required at test time are a subset of the modalities used at training. For instance, an RGBD camera could be used to create a model able to fuse gray, optical flow, depth information and/or silhouettes, but UGaitNet could deal with sequences from an RGB camera (with no depth information) at test time. Such a framework is particularly useful when considering acquisition devices with computational or energy consumption constrains (such as cellular phones) that cannot deal with the computation of complex input data in real-time, such as optical flow or silhouettes. In this work, we experiment with up to four modalities for gait recognition: optical flow, gray, depth and silhouettes. A simplified overview of UGaitNet is shown in Fig. 1, where three possible modalities are used: optical flow depicted in red, silhouette depicted in blue and depth depicted in green. The input of each branch starts processing its corresponding input data (optical flow, silhouettes or depth) together with a binary value indicating if the input is available at test time (OF?, Silhouette? or Depth? ellipses in Fig 1). The output of each branch is its corresponding signature. Finally, UGaitNet combines the available signatures to produce a JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 2 multimodal one, which is used to predict the subject identities. Note that, although optical flow, silhouettes and gray are shown in the figure, any other combination and number of modalities can be used. During training, the whole model is trained end-to-end, using different combinations of modalities per batch, e.g. one or multiple modalities. In that way, all branches are trained in parallel and the final layers learn to combine them. Besides the standard classification loss, for better generalization and robustness, UGaitNet is also trained with a triplet loss between all the signatures, i.e. features from all different modalities. Moreover, using the triplet loss allows UGaitNet to implicitly learn two important properties: view-independent gait signatures and gender classification (Sec. 4.6). A preliminary version of this work [16] proposed the use of a gate mechanism together with a merge operation (i.e. max) to combine information from modality-specialised branches. In that way, when a modality is missing, the gate can disable the input of such modality to its corresponding branch, so as not to penalize the performance of the model. In this work, we propose an extension of [16] with several changes to improve the performance of the approach and completeness of the paper: (i) We propose a novel signmax merge operation, that uses both positive and negative values of the unimodal signatures; in contrast, in [16], only positive values are used, thus limiting the representation capabilities of the model. (ii) We propose a branch-agnostic model where branches can be built using different architectures, while [16] just provides a set of predefined branches per modality. (iii) A silhouette-based modality is added to the three previously considered modalities (optical flow, gray and depth) employed in [16]. (iv) The BatchAll [17] triplet selection (for triplet loss) is used instead of SemiHard one, thus improving the training process, since more triplets contribute to the training gradients. (v) We provide a comprehensive experimentation on the TUM-GAID [8] and CASIA-B [18] datasets, while [16] mainly focus on TUM-GAID. (vi) We include a cross-view experimentation on CASIA-B, while [16] just focus only on multi-view experiments. (vii) Finally, we study and analyze the effect of combining datasets during training, and performing crossdataset testing (i.e. training on CASIA-B and testing on TUM-GAID). The main contributions of this paper can be summarized in: (i) a novel multimodal gait recognition model robust to missing modalities; (ii) generation of robust gait signatures for multi-view and cross-view setups; and, (iii) state-of-theart results on TUM-GAID dataset. 2 RELATED WORK Gait is a behavioral biometric, a kind of attractive feature for human identification at a distance. Gait recognition has been an active research topic for the past decades due to various applications, e.g. surveillance, crime prevention, forensic identification and social security. Here, we review models for gait recognition [6], models based on multiple modalities [9], and some works dealing with missing modalities applied on different tasks [19], [20], [21], [22]. Gait recognition. The typical trend in gait recognition is to rely on appearance and period-based representations, such as silhouette-based methods (Gait Energy Image [1], [2]) and various extensions such as frequency (FrequencyDomain Feature [23]), entropy (Gait Entropy Image [24]) or even optical flow (Gait Flow Image [25]). Nevertheless, silhouettes are not invariant to changes in body shape or clothing, and can be subject to drastic variations, due to illumination changes, dynamic backgrounds or other factors. Other works rely on various sensors [26], such as floor-sensors [27], accelerometer [28], wave-sensors [29], etc. However, [30], [31] show that motion sensor-based systems are vulnerable when attacked with adversarial perturbations. An alternative to silhouette and sensor-based systems are model- [32], [33], [34] or skeleton-based [35], [36] ones. These, however, are very challenging to acquire especially for low image resolution [33]. Another line of works focus on different aspects of gait recognition, such as cross-view recognition [37], [38]. For instance, [39] propose a gait-related loss function on a simplified spatial transformer network [40] to learn discriminative gait features, [41] propose a view-resistant approach using optical flow, and [42] propose a method operating on multiple scales. [43] propose a view-invariant gait representation framework for cross-view gait recognition using the spatiotemporal motion characteristics of human walk. Unlike most approaches that use an image-like gait template or a gait sequence [34], [44], [5], [45], [46], [47], [48] treat gait as a set of independent frames and therefore the proposed method is immune to frame permutation and can integrate frames from different videos. Finally, some works use gait to extract information, e.g. age [49], [50], or to produce gait patterns, e.g. part movements [51]. Multiple-modalities for gait recognition. More recently, several works start to rely on additional input modalities [8], [9], [10]. Besides optical flow [41], [52], the most commonly-used modalities are infrared [11], pose [4], [53] and depth [54], [55], whereas others, such as audio, are less used due to their high complexity and mostly lack of suitable datasets [9]. For instance, [10] trains several multimodal CNNs using optical flow, gray pixels and depth information, while [4] proposes GaitNet, an AutoEncoder to automatically learn the disentangled gait features (pose and appearance) from a video. Missing modalities. There exist several multimodal algorithms dealing with missing modalities. [19] propose to impute lost data using the relations between samples from different modalities. The authors employ an autoencoder that captures the correlation between several modalities and is able to generate impute samples if missed. Autoencoders are also used in [56] to account for missing data in multimodal recommending systems. [20] presents a method for learning a joint text-video embedding, which accounts for missing video modalities (visual or audio cues) by simultaneously learning from heterogeneous data sources. When a modality is missing (e.g. audio), the expression that calculates the similarity between text and video removes the expected information. [21] develops an action recognition system using RGB and infrared videos as input. Assuming that infrared data can be lost during testing, a GAN is developed to generate missing infrared samples from RGB images. Others works, [22], [57], have also used GANs for synthesizing missing data. Note that hallucinating missing JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 3 input modalities is not a valid choice in a biometric context. Therefore, our proposal clearly differs from this type of generative approaches. 3 PROPOSED MODEL Given a sequence of video frames containing a walking person, the goal of gait recognition is to identify the person or, more formally, to obtain its biometric information. In this paper, we propose UGaitNet, a multimodal network for gait recognition robust to missing modalities at test time (Fig. 1). The main idea is to learn multimodal gait signatures that are similar for samples of the same subject, and different for samples of different subjects, regardless of the combination of sources used to compute them. In this section, we first describe the UGaitNet architecture (Sec. 3.1) and then, the training losses (Sec. 3.2). 3.1 UGaitNet architecture UGaitNet is illustrated in Fig. 1. It takes as input videos in various modalities and outputs the predicted identity of the person in the video. In this work, we experiment with four input modalities, i.e. optical flow, gray, depth and silhouettes. Therefore, UGaitNet consists of one branch per modality. Note that the number of modalities depends on the target dataset, since it may include different modalities. More details about the architecture of the input branches can be found in Sec. 3.1.1. At test time, we provide UGaitNet with an additional input: three binary units (one for each branch) indicating whether one input modality is available or not. Therefore, not all input modalities are required and at test time UGaitNet comprises as many branches as available modalities. The output of each branch is passed through a gate mechanism, and then these gated intermediate representations are merged to a single multimodal signature (merge operation). These mechanisms together with the additional input allow the network to deal with missing modalities and are described in detail in Sec. 3.1.2. Finally, the multimodal signature is fed to the classification layers that predict the identities (Sec. 3.1.3). 3.1.1 Input Branches UGaitNet contains one branch per input modality: optical flow, gray, depth, silhouettes, etc. Since the merge operation is performed at the feature level, once the output of each branch is obtained, any CNN can be used as backbone for each branch of UGaitNet. The only requirement is that all branches produce activations with the same shape, in order to merge them. Thus, in order to maximise UGaitNet performance for the target dataset and modality, each branch can be selected according to the best state-of-the-art approach. 3.1.2 Gate mechanism and merge operation Here, we describe the gate and merge operations that allow the network to deal with missing modalities. Gate mechanism. Recall that at test time, we provide UGaitNet with an additional input: kbinary units uindicating whether one input modality is available or not. Each binary unit uiacts as a gate, allowing or not the information coming from its corresponding modality to flow within the network. Notations. For simplicity of the explanation, we assume that we only have two input modalities. However, extending to any number of inputs is straight forward. Let B1 and B2be the backbone networks that extract features from the modalities m1and m2, respectively. Let f1and f2of dimensionality dbe the output vectors obtained from those backbones, respectively. Merge operation. To obtain a multimodal vector while keeping the same dimensionality d, we apply the merge operation to all outputs of the gate mechanism. In this work, we define the merge operation to be the aggregation function z(·)defined as: z(fi 1, u1, fi 2, u2, ..., fi k, uk) = sign max({u1·fi 1, u2·fi 2, ..., uk·fi k}), (1) where sign max({vj 1, vj 2, ..., vj k}) = arg max v∈{vj 1,vj 2,...,vj k} |v|(2) where vi j=uj·fi jand fi jthe i-th component of the feature vector fj; resulting in a d-dimensional output vector z. The intuition behind this choice for z(·)is the following. When ujis 1, the components of fjwill compete to be part of the output multimodal vector z. In contrast, when ujis 0, regardless the value of fjthat information will not become part of the output z, and the modalities will have to collaborate to produce multimodal signatures that are similar even though one or more modalities are missing. Note that the function sign max(·)takes into account negative values of the feature vectors. In contrast, the regular function max(·), used in [16], directly discards high responses with negative sign. One might think that z(·)could be implemented with the average function; however, the missing modalities would affect the response values of the function, and the choice of the input values for the missing values should be much more careful in order not to contaminate the average. In summary, a gait signature zfrom two modalities is obtained as: z=z(B1(m1, θ1), u1,B2(m2, θ2), u2) = sign max(u1· B1(m1, θ1), u2· B2(m2, θ2)),(3) where θ1and θ2are the parameters of the B1and B2 networks, respectively, that are learnt during training. The vector zis L2-normalized before further processing. Discussion. By definition, the B1and B2backbone networks may have different architectures. This is in line with multimodal systems [15], where the input modalities may come from different domains, such as videos, audio, depth maps. Given that gait recognition is mostly a visual process, in this work, we focus on visual modalities, e.g. gray pixels. We argue, however, that the same pipeline can be extended to modalities coming from other sources. 3.1.3 Classification layers The merge operation (Sec. 3.1.2) is followed by a series of fully connected layers that help to either obtain a higherlevel encoding from the output vector zor to reduce its dimensionality. On top of them, we add a soft-max layer that learns the human identities. In this work, we use two fullyconnected layers (see supplementary material for details). 3.2 Loss function for training To deal with missing modalities, UGaitNet needs to learn that the gait signatures for samples of the same subject JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 4 Depth? Silhouette? OF? Gate Gate Gate Merge Multimodal Signature Identities Triplet Loss Cross-entropy loss OF Branch OF Signature Silhouette Branch Silhouette Signature Depth Branch Depth Signature 2Lx60x60 Lx60x60 Lx60x60 ID #23 dx1 dx1 Cx1 FC1 FC2 Fig. 1: UGaitNet: multimodal gait recognition network robust to missing modalities. Input: (ellipses) binary input units (i.e. OF?, Silhouette?, Depth?) will inform the model whether one modality is available or not – in this example, depth is not available (dashed red cross); (volumes) sequences of Lframes for the different available modalities. After fusing the single-modality signatures, a multimodal gait signature of ddimensions is further compressed by layer FC1. The final layer FC2 contains C classes. Note that the binary units are used both at training and test time. are similar (i.e. have a minimum distance), whereas the ones from different subjects are different (i.e. maximize their distance), regardless of the viewpoint or clothing. To achieve this, we deploy the Triplet Loss [58] (Fig. 1): LT(A, P, N) = max kg(A)−g(P)k2− kg(A)−g(N)k2+α, 0,(4) where g(·)is a deep neural network, Ais an anchor sample, Pis a positive sample w.r.t. A,Nis a negative sample w.r.t. A, and αis a margin value to be cross-validated. Additionally, we use the standard Cross-Entropy loss LCto train the feature extractor of UGaitNet and classify the human identities (Fig. 1). The overall loss LMused for training of UGaitNet is a combination of both loss functions: LM=LT+β· LC,(5) where βis a positive weight, experimentally chosen, that controls the contribution of LCin the final loss. 4 EXPERIMENTS In this section, we present experimental results of UGaitNet. For all experiments, we use up to three input modalities with complementary information [10]: optical flow, gray pixels, depth maps, and silhouettes (when available in the dataset). Unlike other methods that fine-tune their models on the gallery samples of the test partition [10], [35], we experiment without any fine-tuning, unless stated otherwise. We directly apply the pre-trained model on the test samples and classify them using a simple kNN classifier, thus validating the generalization of the feature extractor. Here, we first report some implementation details, and then we present the datasets and metrics used in our experiments (Sec. 4.1). Then, we report the experimental results of UGaitNet and we compare it to the state of the art (Sec. 4.2). After that, we compare UGaitNet with several baseline cases (Sec. 4.3) trained without missing modalities, and study the robustness of our approach to missing modalities (Sec. 4.4). Then, we present an ablation study on the model (Sec. 4.5), and examine the learned properties of UGaitNet (Sec. 4.6). Finally, we propose a novel experimental setup, where we train a single model simultaneously with two different datasets using our missing modality strategy (Sec. 4.7). In that way, the model learns to identify all subjects despite the different modalities or the different conditions, such as camera, scenario, etc. Implementation details. To obtain the best possible results, we use state-of-the-art backbones for each branch and dataset. For TUM-GAID, we use CNN architectures derived from the state-of-the-art networks published in [10], [59]. Specifically, for the optical flow branch we use the 2D model from [10] that is based on a sequence of 2D convolutions and max pooling layers, ending with fully connected ones. For gray and depth branches, we use the 3D model from [59] that consists of a sequence of 3D convolutional filters with ReLU activations. For CASIA-B, we use GaitSet [5] for all branches since its code is public and is widely used [46], [47], [48] as baseline in gait recognition due to its outstanding results. Note that, although we use state-of-the-art backbones for each branch, they are trained simultaneously following our missing modality setup explained in Sec. 3.1. Finally, the number of parameters for these branches is summarized in the tables included in the supplementary material. We implement UGaitNet using the Keras version of TensorFlow 2.2 [60]. After cross-validation, the parameters α(Eq. 4) and β(Eq. 5) are set to 0.5and 0.1, respectively. The learning rate starts at 0.001 and is reduced by a factor of 0.2when the validation loss plateaus. During training, each minibatch contains a balanced number of samples JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 5 of the different covariate factors (e.g. normal, bag, shoes) and, for each sample, we have different versions of it, i.e. with all modalities, first modality missing, second modality missing, etc. Data augmentation is applied, including mirror samples. The size of the gray and depth maps is set to L×60 ×60, whereas the size of optical flow is 2L×60 ×60 (stack of the xand ycomponents). For robustness and better generalization, we extract samples from the video sequences with an overlap of 80% with the previous one. In our experiments we set L= 25, as it is enough to capture a complete gait cycle [61]. To obtain a gait descriptor at video level, we average the descriptors obtained from the samples of 25 frames. Results are obtained using a kNN classifier with k= 3. Pre-processing of input modalities. To deal with realworld videos of varying length containing people at different scales (if visible), we need to isolate the target subject from the rest of the scene and normalize their size. To this end, first, we spatially detect the subjects using FasterRCNN [62], pre-trained on MS-COCO, and group them into tracks based on their feature distance. Then, following [7], we align the stacked subsequences of 25 maps in such a way that the body is x-located in the middle of the central frame (i.e. #13) according to the obtained tracks of detections. Finally, we scale down the input maps to 80×60 pixels, keeping the original aspect ratio, and unnecessary background is removed by cropping the maps to 60 ×60 (the full height is kept). To construct the input volumes, we follow [10] and stack 25 consecutive frames. Optical flow is obtained from SpyNet [13] pre-trained on MPI Sintel. Depth maps are represented as gray-scale images, i.e. scaling depth values to [0,255]. 4.1 Datasets and Metrics To validate our approach, we experiment on the two public datasets for gait recognition that provide the original RGB video sequences: CASIA-B [18] and TUM-GAID [8]. Note that the other datasets (e.g. “The OU-ISIR Large Population Gait Database with real-life carried object” [63]) do not provide RGB video sequences, but only silhouettes or GEI descriptors and, hence, they are not suitable for this work. CASIA-B: In CASIA-B, 124 subjects perform walking trajectories in an indoor environment (left part of Fig. 2). The action is captured from 11 viewpoints (i.e. from 0◦to 180◦ in steps of 18◦) with a video resolution of 320 ×240 pixels. Three situations are considered: normal walk (NM), wearing a coat (CL), and carrying a bag (BG). Following the same criterion used in [64], we use the first 74 subjects for training and validation, and the remaining 50 for testing. TUM-GAID: It contains 305 subjects performing two walking trajectories in an indoor environment (right part of Fig. 2), captured by a Microsoft Kinect sensor (resolution of 640 ×480 pixels and 30 fps). Four situations are considered: normal walk (N), carrying a backpack (B), wearing coating shoes (S) and, for 32 subjects, there is an elapsed time case where the subjects were recorded wearing different clothes (TN-TB-TS). In this work, we follow the same experimental protocol of [8], and we use the same train and test splits: 150 subjects for training and 155 subjects for testing. Metrics: We use Rank-1 (R1) accuracy, i.e. the percentage of correctly classified videos: R1=#correct/#total. 4.2 Comparison of UGaitNet to prior state of the art TUM-GAID. Tab. 1 reports the comparison to the state of the art for TUM-GAID. We note that [16] and our UGaitNet are the only flexible frameworks, able to work with a variable number of modalities, i.e. as many as available in the dataset, since other state-of-the-art approaches are limited to solely a single modality or several modalities at the same time, but they do not cope with missing modalities. In all cases, UGaitNet outperforms all other methods. For TUM-GAID, it obtains a global average of 99.2% over all scenarios. We also observe that UGaitNet outperforms MissGait [16] thanks to the new merge function and L2normalization (more details in Sec. 3.1.2) together with the training strategy. CASIA-B. Recall that our goal is dealing with missing modalities. Thus, for CASIA-B we perform two types of experiments. First, we use the same camera for gallery and probe sets (i.e. no cross-view at test time). This enables decoupling the behaviour of the missing modalities from the missing cameras in the gallery set. Second, we use different cameras for gallery and probe sets (i.e. cross-view at test time). This experiment allows us to properly compare with other state-of-the-art approaches. Tab. 2 (top) reports the results of the first experiment, i.e., using the same camera for gallery and probe sets. We observe that UGaitNet with optical flow and gray (O+G) outperforms the competitor [16], in particular for the BG scenario. Specifically, we achieve the new state of the art for missing modalities, going from an average of 79.2% for MissGait [16] to 84.4% for UGaitNet. Tab. 2 (bottom) reports the results of the second experiment, i.e., comparison to other approaches using a cross-view setup. We observe that UGaitNet using Gray or Gray + OF successfully outperforms GaitSet, which is the baseline used for the UGaitNet branches. Other approaches like [46], [47], [48] achieve better results on CASIA-B. This is expected, first because these methods are newer than GaitSet and some of them rely on the GaitSet architecture, and second because they just focus on dealing with multiple viewpoints using a single modality while we must deal with missing modalities and multiple viewpoints. However, we expect that updating UGaitNet branches with recent models would bring similar performance improvements, which we leave for future work. Finally, in both experiments, comparing UGaitNet with optical flow and gray (O+G) to UGaitNet with flow and silhouettes (O+S), we observe that gray achieves the best performance, due to the extra information contained in this modality compared to the one that silhouettes encompass. However, for the CL scenario, where subjects wear coats, optical flow and silhouettes (O+S) obtains the best results. In this case, removing the extra information of the gray can benefit the generalization capabilities of the model to clothing. Note that in our comparison, we only include approaches that use missing or multiple modalities, as this is the goal of this paper. Finally, on TUM-GAID, UGaitNet achieves the new state of the art for all scenarios. For the ‘temporal scenario’, it results in perfect performance, i.e. from 78.6% [35] to 100%, and for the average on the whole test set, it increases from 96.5% to 99.2%. JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 6 Fig. 2: Datasets for gait recognition.(left) CASIA-B. People walking indoors recorded from 11 viewpoints and three situations: (from left to right) normal walking, carrying a bag and wearing coats. (right) TUM-GAID. People walking indoors under four situations: (from left to right) normal walking, carrying a bag, wearing coating shoes and wearing coats. Input Size Method N B S Avg TN TB TS Avg Global Avg 640 ×480 SiameseAE [35] 98.7 93.6 98.0 96.8 81.4 76.2 78.1 78.6 95.1 PFM [65] 99.7 99.0 99.0 99.2 78.1 62.0 54.9 65.0 96.0 60 ×60 MTaskCNN [49] 99.7 97.4 99.7 98.9 59.4 62.5 68.8 63.6 95.6 3D-CNN+Fusion [10] 100 99.4 99.4 99.6 75 62.5 62.5 66.7 96.5 MissGait [16] 99.7 98.1 98.1 98.6 100 100 100 100 98.8 Ours (OF+G+D) 99.7 99.4 98.1 99.1 100.0 100.0 100.0 100.0 99.2 TABLE 1: State of the art on TUM-GAID. Rank-1 identification rate (%) at video level. Each column corresponds to a different scenario. Best results are marked in bold. Method 0 18 36 54 72 90 108 126 144 162 180 Avg Identical-View NM MissGait [16] 96 98 97 96 97 98 99 96 95 95 99 96.9 Ours (O+G) 99 100 100 99 100 100 100 98 98 96 98 98.9 Ours (O+S) 95 98 98 97 96 97 97 96 95 96 96 96.5 BG MissGait [16] 81 85 81 75 78 72 78 78 79 82 78 78.8 Ours (O+G) 98 97 95 94 88 90 88 91 93 95 91 92.7 Ours (O+S) 86 80 80 78 73 74 77 79 84 78 80 79.0 CL MissGait [16] 44 51 58 67 69 69 74 73 66 60 51 62.0 Ours (O+G) 61 65 66 63 62 68 67 62 60 56 48 61.6 Ours (O+S) 59 64 70 67 68 70 71 69 71 68 57 66.7 Cross-View NM GaitSet [5] 90.8 97.9 99.4 96.9 93.6 91.7 95.0 97.8 98.9 96.8 85.8 95.0 GaitPart [46] 94.1 98.6 99.3 98.5 94.0 92.3 95.9 98.4 99.2 97.8 90.4 96.2 GLN [47] 93.2 99.3 99.5 98.7 96.1 95.6 97.2 98.1 99.3 98.6 90.1 96.9 GaitGL [48] 96.0 98.3 99.0 97.9 96.9 95.4 97.0 98.9 99.3 98.8 94.0 97.4 Ours (G) 93.5 95.0 96.7 97.5 97.3 97.4 97.6 96.9 98.0 96.2 93.9 96.4 Ours (O+G) 88.9 92.1 95.0 96.6 96.2 95.3 95.6 96.4 96.5 94.8 91.3 94.4 BG GaitSet [5] 83.8 91.2 91.8 88.8 83.3 81.0 84.1 90.0 92.2 94.4 79.0 87.2 GaitPart [46] 89.1 94.8 96.7 95.1 88.3 94.9 89.0 93.5 96.1 93.8 85.8 91.5 GLN [47] 91.1 97.7 97.8 95.2 92.5 91.2 92.4 96.0 97.5 95.0 88.1 94.0 GaitGL [48] 92.6 96.6 96.8 95.5 93.5 89.3 92.2 96.5 98.2 96.9 91.5 94.5 Ours (G) 91.6 89.9 92.4 93.3 94.3 93.8 92.5 93.9 94.5 94.1 90.1 92.8 Ours (O+G) 84.9 88.2 90.7 92.5 91.3 89.0 90.2 91.6 92.6 94.1 86.4 90.1 CL GaitSet [5] 61.4 75.4 80.7 77.3 72.1 70.1 71.5 73.5 73.5 68.4 50.0 70.4 GaitPart [46] 70.7 85.5 86.9 83.3 77.1 72.5 76.9 82.2 83.8 80.2 66.5 78.7 GLN [47] 70.6 82.4 85.2 82.7 79.2 76.4 76.2 78.9 77.9 78.7 64.3 77.5 GaitGL [48] 76.6 90.0 90.3 87.1 84.5 79.0 84.1 87.0 87.3 84.4 69.5 83.6 Ours (G) 73.5 75.6 77.4 74.7 74.9 72.3 72.3 72.3 68.9 68.0 59.9 71.8 Ours (O+G) 69.7 74.1 76.6 76.7 77.9 74.8 75.3 76.3 71.1 68.8 62.8 73.1 TABLE 2: State of the art on CASIA-B. Rank-1 identification rate (%) at video level using gallery and probe sets for identicalview cases (top) and cross-view setup excluding identical-view cases (bottom). Best results are marked in bold. Classifier Modality N B S Avg BL-single: 3NN Gray 96.3 92.9 94.8 94.7 OF 75.6 61.5 68.8 68.6 Depth 82.3 68.2 76.1 75.5 BL-all: 3NN-256 Gray+OF+Depth 84.1 70.9 75.1 76.7 BL-late: AVG-SM Gray+OF+Depth 99.8 98.7 99.7 99.4 UGaitNet Gray+OF+Depth 97.7 94.8 95.3 95.9 TABLE 3: Baseline experiments on TUM-GAID. Rank-1 identification rate (%) at subsequence level. Each row represents a different baseline approach. Each column corresponds to a different scenario. 4.3 Baseline models In this section, we present three different baselines trained with a standard approach where missing modalities are not supported. All baseline models share basic architectural aspects with UGaitNet, thus allowing for a fair comparison that focuses on the contributions related to handling missing modalities rather than architecture differences. First, we train models with a single modality to verify that training with multiple modalities (i.e. UGaitNet) improves upon single ones. Secondly, we train a model with all three modalities available during training and test, i.e. without our missing modality strategy. Note that this model can be considered as an early fusion strategy since it combines features from different modalities thanks to the merge operation. Finally, as a third baseline, we propose a late fusion strategy combining the probabilities at the softmax layer obtained by each single modality type. Recall that to obtain a descriptor at the video level, we average the descriptors obtained from the subsequences of 25 frames. To eliminate any possible effects introduced by this average operation, in this study we report results at the subsequence level, i.e. considering each 25-frame subsequences an independent sample. Single modality. For CASIA-B, we train three independent networks (i.e. one per available modality) only with TripletLoss. This results in gait descriptors of 15,872 dimensions, i.e. an architecture equivalent to the corresponding branch used in UGaitNet. Following the standard experimental protocol [5], we use these networks to extract gait descriptors for both the gallery and probe samples of the test partition. The results at the subsequence level of this baseline are summarized in the rows ‘BL-single: 3NN’ of Tab. 4. Comparing the results between modalities, it is clear that gray modality achieves the best results, followed by silhouettes. For TUM-GAID, we repeat the same process for the available modalities. A smaller gait descriptor with 1024 dimensions is enough to represent properly the lower variability of this dataset. We follow the experimental protocol proposed in [8] and the subsequence level results are summarized in ‘BL-single: 3NN’ rows of Tab. 3. Like in CASIAB, gray information achieves the best results, followed by Depth and OF. Finally, bottom rows in Tabs. 4 and 3 report UGaitNet results so that they can be compared to the ‘BL-single’ results. We observe that for both datasets UGaitNet improves or achieves similar results to the single modality cases. No missing modalities. We train a single UGaitNet with no modality missing at either training or test time. Therefore, JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7 Case Classifier Modality 0 18 36 54 72 90 108 126 144 162 180 Avg NM BL-single: 3NN Gray 92.3 94.1 96.7 96.8 96.7 96.1 96.6 96.8 96.9 95.0 87.5 95.0 BL-single: 3NN OF 21.8 43.9 59.1 61.8 66.3 68.4 67.7 64.9 62.1 50.5 25.3 53.8 BL-single: 3NN Silhouette 44.6 54.2 68.7 65.9 67.0 66.1 69.0 71.6 74.4 63.3 44.0 62.6 BL-all: 3NN Gray+OF 84.6 87.8 93.5 94.8 93.2 92.8 93.2 94.5 93.6 90.0 84.1 91.1 BL-all: 3NN Sil+OF 56.9 69.3 78.4 78.4 78.8 78.0 79.0 82.2 82.2 75.6 58.0 74.3 UGaitNet Gray+OF 84.2 87.7 93.3 93.9 93.2 92.6 93.3 93.6 93.6 90.4 83.3 90.8 UGaitNet Sil+OF 57.2 68.7 78.6 78.2 79.1 78.2 79.0 81.9 82.1 78.4 56.9 74.4 BG BL-single: 3NN Gray 90.4 90.8 92.2 91.2 90.1 89.8 91.3 92.1 93.0 90.8 82.4 90.4 BL-single: 3NN OF 19.7 36.7 49.4 50.3 53.5 54.8 54.1 56.4 56.2 44.2 21.1 45.1 BL-single: 3NN Silhouette 34.1 40.5 50.2 45.5 46.3 44.9 48.9 53.0 57.6 51.4 33.0 45.9 BL-all: 3NN Gray+OF 81.0 83.1 87.8 86.9 86.1 85.0 85.9 88.8 89.7 87.4 76.8 85.3 BL-all: 3NN Sil+OF 46.9 55.8 63.2 61.4 62.2 62.7 65.0 70.6 73.6 64.0 46.5 61.1 UGaitNet Gray+OF 80.3 82.4 87.4 86.6 89.6 85.7 86.2 88.9 89.0 88.1 76.2 85.5 UGaitNet Sil+OF 48.5 55.8 63.6 61.3 63.7 63.0 64.8 70.8 74.3 64.4 44.7 61.4 CL BL-single: 3NN Gray 70.7 73.0 75.8 73.9 71.2 69.6 66.4 66.5 64.5 61.5 52.1 67.7 BL-single: 3NN OF 16.5 31.0 40.8 43.2 46.4 48.1 48.0 47.3 44.0 35.8 19.2 38.2 BL-single: 3NN Silhouette 26.6 34.9 45.3 44.8 44.3 45.0 46.5 46.9 47.5 39.7 27.0 40.8 BL-all: 3NN Gray+OF 62.6 67.1 70.7 71.0 72.1 68.3 68.8 68.9 68.0 62.6 54.4 66.8 BL-all: 3NN Sil+OF 44.9 53.7 63.2 61.7 63.6 62.9 66.6 68.7 65.2 57.7 44.1 59.3 UGaitNet Gray+OF 62.6 66.9 70.8 71.0 73.8 70.7 70.9 71.5 68.0 62.8 54.4 67.6 UGaitNet Sil+OF 43.4 52.4 62.7 61.9 64.3 62.8 66.3 67.6 65.6 58.3 43.6 59.0 TABLE 4: Baseline experiments on CASIA-B. Rank-1 identification rate (%) at subsequence level. Cross-view setup excluding identical-view cases. Each row represents a different baseline approach. Each column corresponds to a different camera. the output descriptor is the max fusion of the available modalities, i.e. three for TUM-GAID and two for CASIA-B. The subsequence level results of this baseline are summarized in the rows ‘BL-all’ of Tab. 4 and Tab. 3. We observe that on average, ‘UGaitNet’ outperforms this baseline. This is probably because the compete and collaborate process that UGaitNet uses when training with missing modalities helps the network learn representations that generalize better than this baseline. In the silhouette case (Tab. 4), since each modality alone (i.e. optical flow and silhouettes) obtains low results, the combination of both modalities also obtains low accuracies, i.e., 53.8% and 62.6%, respectively, for the NM scenario; however, UGaitNet still helps to improve them, i.e. 74.4%. Late fusion. In this work, we are interested in addressing the problem of multimodal systems with possible missing modalities. However, for the sake of completion, we compare UGaitNet with late fusion strategies, which allow to combine different modalities at test time. Note that we consider the late fusion approaches as an upper-bound, since they require training three different models and combining them at test time. Thus, we define the following baseline: we train N single and independent networks, one per modality with the Crossentropy loss; at test time, we average the output (softmax predictions) of the M≤Navailable modalities to cast a prediction. Therefore, for this baseline: (a) there exist a different model per modality; (b) to recognize the test subjects, each model requires fine-tuning, i.e. an additional step; and, (c) the descriptors from different models are not directly comparable as they belong to different feature spaces. In contrast to this baseline, for UGaitNet (Sec. 3.1) (a) all input modalities co-exist, as it is a unified model, (b) no fine-tuning is required because we use a kNN classifier to identify the different subjects; and, (c) all modalities share a common feature space. The results of this baseline are summarized in rows ‘BL-late: AVG-SM’ of Tab. 3 for TUM-GAID. This baseline improves upon the ‘BL-single’ one, showing that combining multiple modalities boosts the results. By comparing the baseline to UGaitNet, we observe that ‘BL-late’ obtains betAvailable modality N B S Avg Depth (miss OF+G) 80.7 67.6 76.4 74.9 Gray (miss OF+D) 99.0 97.9 97.7 98.2 OF (miss G+D) 52.4 35.4 46.5 44.8 Gray+Depth (miss OF) 97.7 94.8 95.3 95.9 OF+Depth (miss G) 80.7 67.6 76.4 74.9 OF+Gray (miss D) 99.0 97.9 97.7 98.2 OF+Gray+Depth 97.7 94.8 95.3 95.9 TABLE 5: Missing modalities: Rank-1 identification rate (%) at subsequence level on TUM-GAID. ter results. This is expected as the baseline uses information from three different models trained and fine-tuned on the test subjects, whereas UGaitNet trains one unified model and does not require any fine-tuning. We argue that the computational cost of the baseline is significantly higher than the one of UGaitNet. This clearly demonstrates that this late fusion technique cannot be applied in an open-world setup, which is not limited to N subjects, as required by the ones using softmax. Note that we do not report ‘BL-late’ results for CASIAB. Because, after many trials, the GaitSet-like models did not converge properly with the cross-entropy loss. In our opinion, the softmax classifier was not able to deal with so large feature vectors. 4.4 Robustness to missing modalities The goal of this experiment is to evaluate the possible drop in performance when one or more modalities are missing at test time. Note that, to see the effect of different modalities, we report results at the subsequence level, thus eliminating the effect derived by combining subsequences. TUM-GAID. The results of this experiment are summarized in Tab. 5. We observe that the combination of gray and depth is competitive enough (i.e. 95.9%), and the contribution of the OF is in general very limited. For the all modalities case, we observe that the performance drops (95.9%) compared to using just gray (98.2%). By observing the results, two patterns emerge: (a) training with two modalities outperforms training with more (three), as less combinations of modalities have to compete during training, and (b) in all cases the dominant modality is gray, suggesting that using JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 8 depth or optical flow is not crucial, given the image and camera conditions of this dataset. CASIA-B. A similar analysis is performed on CASIA-B, summarized in Tab. 6. In this case, we have two groups of experiments: using gray+OF (white rows) or silhouettes+OF (gray rows). Comparing both groups, it is clear that graybased results outperform silhouette-based ones, specially for the BG scenario. Focusing on the missing modalities, when silhouettes are used, the model benefits from using an additional modality like OF, improving missing modality cases. On the other hand, when gray is used, it achieves better performance if OF is not included as input. In our opinion, the differences observed between gray and silhouettes are due to the fact that gray provides richer information to the model, what minimizes the impact of complementary modalities like OF. 4.5 Ablation studies Here, we compare the original UGaitNet with different modifications of the approach: merge operation,branch initialization and early stopping of the branches. Note that in these experiments, for brevity we present results at the video level and only for TUM-GAID. Effect of the merge function. In this experiment, we investigate the impact of the merge function used to combine different modalities) on the performance of the model. In detail, we examine three different merge functions: signmax, i.e., the default from Eq. 3, average and maximum per component on the feature vectors of the input modalities. The results are summarized in Tab. 7. By comparing the average (row ‘MAX-Merge’) and the maximum (row ‘AVGMerge’) functions, we observe that the maximum obtains slightly better results, especially for the difficult scenarios like carrying a bag (B) or wearing coating shoes (S). However, sign-max achieves the best results, since it exploits the positive and negative activation values of the unimodal signatures used for merging, thus losing less information during the data merge step. Initialization of the branches. We train three independent gait recognizers (one per modality) on TUM-GAID, using the same architectures used for the corresponding branches of UGaitNet. After training the individual models we transfer their weights to UGaitNet and train it. The results are summarized in Tab. 7, where we consider two different training processes: (a) fine-tuning using a small learning rate (‘UGaitNet + IB-FT’) and, (b) freezing the convolutional layers (‘UGaitNet + IB-FC’). By comparing both options, we observe that if the whole model is fine-tuned with a small learning rate (‘UGaitNet + IB-FT’) the average performance drops, probably due to interferences among branches during the fine-tuning process. On the other hand, if only the convolutional layers are frozen (‘UGaitNet + IB-FC’), the average performance has a bigger degradation than finetuning. Finally, by comparing the results with the original UGaitNet ones (‘UGaitNet’), we observe that none of the initialization strategies improves upon original UGaitNet. Contribution of the early stopping mechanism. During training, it might happen that one modality reaches an individual performance much higher than others – we consider that it converged. In such cases, we want the remaining training steps to focus on improving the performance of the other modalities. To this end, we propose to freeze the weights of the converged branch. For this purpose, we monitor the accuracy and the equal error rate (EER) of each individual branch on the validation set. During the experimental process, we observe that (a) the EER of the individual branches tends to be unstable, causing the branches not to stop their training, and (b) the results obtained by EER are lower than the ones using accuracy as control metric. Hence, in this paper we focus on the accuracy. For stopping the training process of each branch, we follow a cross-validation strategy on the validation set and select the value of the threshold to be 90%. The results of this experiment are summarized in Tab. 7, row ‘UGaitNet + ES-90’. We observe that the gray branch stops at a very early stage of training (before epoch 10) and that the depth branch stops, in both cases, at epoch 60. The optical flow branch never stops because it needs more time to converge. These results clearly demonstrate that this technique does not improve the original results obtained by UGaitNet. 4.6 Understanding what UGaitNet has learned Given a model trained on the train partition of CASIAB (74 subjects), we are interested in understanding what the network has learned. To this end, we apply the model on the other partition of the dataset whose samples are not seen during training (i.e. gait signatures of gallery samples of the test partition in CASIA-B with 50 subjects) and visualize the results (2D projection using UMAP [66] in Fig. 3). The left plot shows that, in contrast to finding different camera viewpoints in different parts of the feature space, the different viewpoints are evenly spread within the subject clusters. The right plot shows that UGaitNet groups the gait signatures based on the gender (green and orange colors), without having seen a gender label at training; this suggests that in order to learn a discriminative gait signature, UGaitNet learns an additional task, i.e. gender classification (implicit supervision). To validate our finding, we train a binary SVM for gender classification and obtain AUC = 92.1% and EER = 14.3% on the test partition, where the error chance is 24.0%. This shows that UGaitNet implicitly learns rich and robust gait signatures for gender classification. Finally, additional Figures are included in the supplementary material showing the effect of missing modalities in the feature space. 4.7 Supplementary material The supplementary material includes the following information: (a) further details on the network architectures used in the experiments; (b) additional UMAP figures on the generated multimodal signatures; and (c) experimental results on joint and cross-dataset setups (i.e. training models with combined datasets, and training on CASIA-B and testing on TUM-GAID). 5 CONCLUSIONS Exploiting multiple modalities helps to improve results in a range of problems [15], [67], [68]. Having access to all modalities at test time is not always the case, as one or more modalities may fail or be unavailable (e.g. missing sensor in a new device). Therefore, to deal with these situations JOURNAL OF L A T EX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 9 Case Available modality 0 18 36 54 72 90 108 126 144 162 180 Avg Gray (missing OF) 89.9 91.4 94.6 95.3 95.7 95.4 96.0 95.6 95.8 93.9 88.2 93.8 OF (missing Gray) 36.8 53.8 66.0 67.1 69.7 71.6 72.6 69.7 66.4 58.9 40.3 61.2 Gray+OF 84.2 87.7 93.3 93.9 93.2 92.6 93.3 93.6 93.6 90.4 83.3 90.8 Sil (missing OF) 47.3 56.5 69.1 66.8 71.4 71.3 71.8 74.2 75.4 64.6 47.2 65.1 OF (missing Sil) 32.6 50.5 63.0 66.6 67.1 68.4 70.0 69.6 67.4 58.5 35.3 59.0 NM Sil+OF 57.2 68.7 78.6 78.2 79.1 78.2 79.0 81.9 82.1 78.4 56.9 74.4 Gray (missing OF) 86.5 86.4 89.9 89.2 91.2 90.3 89.4 90.9 91.6 90.1 82.2 88.9 OF (missing Gray) 32.8 48.3 56.9 56.7 61.1 61.3 60.2 63.6 60.9 52.6 35.1 53.6 Gray+OF 80.3 82.4 87.4 86.6 89.6 85.7 86.2 88.9 89.0 88.1 76.2 85.5 Sil (missing OF) 38.4 43.5 53.4 49.6 54.5 54.7 57.3 61.1 64.1 55.1 36.6 51.7 OF (missing Sil) 28.4 42.3 52.6 53.7 57.4 59.0 57.8 60.7 60.9 48.3 28.1 49.9 BG Sil+OF 48.5 55.8 63.6 61.3 63.7 63.0 64.8 70.8 74.3 64.4 44.7 61.4 Gray (missing OF) 67.2 68.5 72.1 69.5 70.2 68.5 67.2 67.9 65.3 61.1 53.5 66.5 OF (missing Gray) 25.3 38.3 44.7 47.1 54.1 54.4 53.8 52.2 46.5 38.7 27.7 43.9 Gray+OF 62.6 66.9 70.8 71.0 73.8 70.7 70.9 71.5 68.0 62.8 54.4 67.6 Sil (missing OF) 32.5 39.7 52.0 48.6 54.1 54.7 55.2 56.2 55.6 49.2 33.5 48.3 OF (missing Sil) 23.7 37.3 46.4 49.4 51.0 53.1 54.8 53 50.4 41.3 27.5 44.4 CL Sil+OF 43.4 52.4 62.7 61.9 64.3 62.8 66.3 67.6 65.6 58.3 43.6 59.0 TABLE 6: Missing modalities: Rank-1 identification rate (%) at subsequence level on CASIA-B. Cross-view setup excluding identical-view cases. White rows (i.e., first three rows in each case) are obtained with gray+OF and gray-shaded rows (i.e., last three rows in each case) with silhouettes+OF. Fig. 3: UMAP projection of gait signatures for CASIA-B samples.(left) Colorized by camera viewpoint: our gait signatures are viewpoint independent. (right) Colorized by gender: clusters indicate that female gait style is different from male gait style. (Best viewed in digital format.) Experiments N B S Avg UGaitNet 99.7 99.4 98.1 99.1 AVG-Merge 99.7 97.7 97.7 98.4 MAX-Merge 99.7 98.1 98.1 98.6 UGaitNet + IB-FT 99.0 97.7 98.4 98.4 UGaitNet + IB-FC 99.0 96.8 98.4 98.1 UGaitNet + ES-90 98.4 90.0 95.2 94.5 TABLE 7: Ablation studies. Comparison between original UGaitNet and different techniques applied to our approach. Each row represents a different technique. Each column represents a different scenario of TUM-GAID. Rank-1 identification rate (%) at video level. while focusing on the task of gait recognition, we introduced UGaitNet, a single network that handles and combines various types of input modalities for gait recognition: pixel gray value, optical flow, depth maps and/or silhouettes. Although UGaitNet is trained with several input modalities, at test time it is robust to missing ones. UGaitNet obtains state-of-the-art gait descriptors when evaluated on TUM-GAID. Additionally, we showed that UGaitNet with optical flow and grayscale inputs achieves almost perfect (98.9%) recognition accuracy for same-view “normal” and top (96.4%) recognition accuracy for cross-view “normal” on CASIA-B, and perfect (100%) recognition accuracy on TUM-GAID (“ellapsed time”). Finally, we present the results of a new challenging experimental setup where two existing gait datasets are mixed and a single model is trained on it. The results indicate that using gray-based descriptors helps obtain promising results. As future work, we plan to investigate additional modalities, such as body skeletons. Moreover, we are interested in experimenting with more challenging scenarios, with more data and potentially multiple people. Another line of research would be the incremental addition of new modalities to already trained UGaitNet models. ACKNOWLEDGMENTS This work has been funded by Junta de Andaluc´ ıa (P18FR-3130 and P20 00430) and the Spanish Ministry of Science and Technology (PID2019-105396RB-I00 and RED2018102511-T). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research. REFERENCES [1] W. Zeng, C. Wang, and F. Yang, “Silhouette-based gait recognition via deterministic learning,” Pattern Recognition, vol. 47, no. 11, pp. 3568 – 3584, 2014. [2] J. Han and B. Bhanu, “Individual recognition using gait energy image,” IEEE PAMI, vol. 28, no. 2, pp. 316–322, 2005. [3] C. Wan, L. Wang, and V. V. Phoha, “A survey on gait recognition,” ACM Computing Surveys (CSUR), vol. 51, no. 5, p. 89, 2019. [4] Z. Zhang, L. Tran, X. Yin, Y. Atoum, X. Liu, J. Wan, and N. Wang, “Gait recognition via disentangled representation learning,” in CVPR, 2019, pp. 4710–4719. [5] H. Chao, Y. He, J. Zhang, and J. Feng, “Gaitset: Regarding gait as a set for cross-view gait recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2019. [6] K. Shiraga, Y. Makihara, D. Muramatsu, T. Echigo, and Y. Yagi, “Geinet: View-invariant gait recognition using a convolutional neural network,” in Intl. conference on biometrics (ICB), 2016, pp. 1–8. [7] F. M. Castro, M. J. Mar´ ın-Jim´ enez, N. Guil, and N. P. de la Blanca, “Automatic learning of gait signatures for people identification,” in IWANN, vol. 10306, 2017, pp. 257–270. [8] M. Hofmann, J. Geiger, S. Bachmann, B. Schuller, and G. Rigoll, “The TUM Gait from Audio, Image and Depth (GAID) database: Multimodal recognition of subjects and traits,” J. of Visual Com. and Image Repres., vol. 25, no. 1, pp. 195 – 206, 2014.