Toward graph-based semi-supervised face beauty prediction

Toward Graph-based Semi-supervised Face Beauty Prediction Fadi Dornaika 1,2 1University of the Basque Country (UPV/EHU), Spain 2IKERBASQUE, Basque Foundation for Science, Spain EMAIL: [email protected] TEL: 0034 943018034 Kunwei Wang 1,3 1University of the Basque Country (UPV/EHU), Spain 3Northwestern Polytechnic University, Xian, China EMAIL: wkwke[email protected]du.cn Ignacio Arganda-Carreras 1,2 1University of the Basque Country (UPV/EHU), Spain 2IKERBASQUE, Basque Foundation for Science, Spain EMAIL: ignacio.ar[email protected] Anne Elorza 1 1University of the Basque Country (UPV/EHU), Spain EMAIL: [email protected] Abdelmalik Moujahid 1 1University of the Basque Country (UPV/EHU), Spain EMAIL: ab[email protected] 1 This is the accepted manuscript of the article that appeared in final form in Expert Systems with Applications 142 : (2020) // Article ID 112990, which has been published in final form at https://doi.org/10.1016/j.eswa.2019.112990. © 2019 Elsevier under CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/) (?;?). The generic semi-supervised learning methods using graph-based label propagation attracted much attention in the last decade. All of them impose that samples with high similarity should share similar labels. They differ by the regularization term as well as by the loss function used for fitting label information associated with the labeled samples. All of these methods use the graph similarity matrix and the initial labels of some samples. Some recent label propagation algorithms (they can also be called classifiers (Sousa et al., 2013)) are: Gaussian Fields and Harmonic Functions (GFHF) (Zhu et al., 2003), Local and Global Consistency (LGC) (Zhou et al., 2004), Laplacian Regularized Least Square (LapRLS) (Belkin et al., 2006), Robust Multi-class Graph Transduction (RMGT) (Liu and Chang, 2009), Flexible Manifold Embedding (FME) (Nie et al., 2010). These techniques can be either transductive (defined for training samples only) or inductive (defined for both training and unseen samples). The method proposed in (?), learns a unified graph via a structural regularization term. Instead of weight regularization which is adopted by previous works, the work presented in (?) learns a unified graph and weights from a priori individual graphs. Thinking about beauty, it seems quite reasonable to assume that when two faces resemble each other they should have similar attractiveness scores. This abstract idea can be materialized by constructing a weighted graph, in which nodes are images (or their descriptors) and the weights between each pair of nodes represent their similarities. Therefore, in our case, we exploit manifold structure of face images (both labeled and unlabeled) via graphs. In this assumption, similar images should share similar beauty scores. In this paper, we introduce the semi-supervised paradigm to face beauty prediction field. We explore some manifold based semi-supervised algorithms to the specific problem of automatic facial beauty assessment. Moreover, we propose a non-linear Flexible Manifold Embedding for carrying out the score propagation. This proposed method can achieve state-of-the-art results. 0.1 Notations and preliminaries In the sequel, capital bold letters denote matrices and bold letters denote vectors. Assume that x1,x2,...,xlare llabeled face images (or their descriptors) and that xl+1,xl+2,...,xN are the uunlabeled face images. Here, the vector xirefers to the ith face image. The data matrix Xis defined by X= [x1,x2,...,xN]∈RD×N. The total number of training images is N=l+u. The semi-supervised algorithms we are using were originally developed for classification tasks, where the ground truth labels Yand the predicted labels Fare matrices in RN×C, where Yij = 1 if sample ibelongs to class jand Yij = 0 otherwise. Cdenotes the number of classes. Since our problem is essentially a regression problem, the labels are real numbers that can be represented as a column vector y∈RN. The first lrows of ywill contain the scores of the llabeled images, while the last urows will generally be 0, since they correspond to the uunlabeled images. In addition to the initial label (or score) vector y, we consider the unknown label vector f∈RNthat should be estimated. As already stated, the way of exploiting the information contained in unlabeled data is to consider a similarity graph that encodes the pairwise similarity between images. To this end, we have to introduce a similarity matrix S∈RN×N(which will be symmetric, so our graph has to be undirected). Each element Sij of Sis the similarity between samples 2 iand j(i.e., face iand face j). This graph is assumed to capture much information about the data manifold. In our work, without loss of generality, the affinity matrix Sis set to the KNN graph similarity matrix as it offers a simple and very efficient method for graph construction. It proceeds as follows. First, the adjacency matrix is constructed (the edges are set). Second, the weights of the edges are estimated. For adjacency matrix construction, K-Nearest Neighbor can be used in order to find the neighbors of a datum. There is a function that defines the distance (similarity) of one input with respect to the others. In the second phase, a weight should be assigned to each constructed edge. In general, this weight should quantify the similarity between two connected nodes. Let sim(xi,xj) be the similarity score between neighbors xiand xj, then the elements of the graph weight matrix Sare given by Eq. (1). Sij =sim(xi,xj) if xiand xjare neighbours 0 otherwise (1) There are several choices for sim(xi,xj). For instance, in (Belkin and Niyogi, 2003) the authors use the heat kernel sim(xi,xj) = e− kxi−xjk2 twhere tcan be set to the average of squared distances in the training set. We adopt the above similarity and set the neighborhood size of the KNN graph to 10 as in many studies (Dornaika and El Traboulsi, 2016)). We emphasize that more sophisticated graph construction methods can be used in order to estimate the similarity matrix S(e.g., (Cheng et al., 2010; He et al., 2011; Dornaika et al., 2013; Dornaika and Bosaghzadeh, 2015; Dornaika et al., 2016; Nie et al., 2016)). The Laplacian matrix of Sis given by L=D−S, where Dis the diagonal matrix whose elements are the row sums of S. The normalized Laplacian matrix is given by ˆ L=I−D−1/2SD−1/2, where Iis the identity matrix of size N. Finally, 1,0∈RNdenote vectors with all elements as 1 and 0 respectively. The norm ||·|| denotes the Euclidean norm. 0.2 Problem statement The input data are given by a set of face images or their descriptors x1,...,xl,xl+1,xl+2,...,xN. land u=N−lrepresent the numbers of labeled and unlabeled face images, respectively. The labels are given by real scores y1, y2, . . . , yl. The goal is to infer the scores of the unlabeled face images. For illustration, Figure 1 shows a toy example that demonstrates the principle of graph-based score propagation. In this example, we have seven face images, from which only 4 images are labeled with face beauty score. The remaining images are unlabeled. The objective is to recover the face beauty score of all unlabeled images by performing score propagation over the graph. The similarity matrix of the graph, S, is constructed using all training images. Unless stated otherwise, the paper targets learning from single-valued scores. Thus, learning from discrete classes or from label distribution is beyond the scope of the paper. 3 Figure 1: Principle of graph-based beauty score propagation. 4 1 Graph-based score propagation schemes In this section, we will describe some existing label propagation methods that were developed for discrete classification. We also show their adaptation for the score propagation, where the score is a continuous variable. All existing graph-based label propagation scheme use either a non normalized graph or a normalized graph. We emphasize that the three semi-supervised schemes that are presented and developed in Sections 3 and 4 use a normalized graph. In other words, the objective functional of each method uses a normalized Laplacian matrix. 1.1 Local and Global Consistency The Local and Global Consistency (LGC) method was introduced in (Zhou et al., 2004). It aims at predicting the discrete labels of all labeled and unlabeled instances, F, by minimizing the following function: q(F) = N X i,j=1 Sij      fi √Dii −fj pDjj      2 +µ N X i=1 kfi−yik2,(2) where fiis the i-th row of Fand Dii is the sum of the i-th row of Sor, in other words, the sum of the similarities of sample iwith all the other images. The first term is the smoothness constraint. The second term is the fitting constraint and µis the parameter which controls the trade-off between them. The optimal solution to this problem can be found analytically by vanishing the first derivatives w.r.t. the unknown. It is given by F= (I+ˆ L/µ)−1Y, where Iis the identity matrix of Ndimensions. Once the matrix F is estimated, the predicted class of an instance iwill be the maximum index jof the i-th row of F. Our revisited LGC should predict the scores of all images, namely the vector f. It minimizes the following: q(f) = N X i,j=1 ˆ Sij(fi−fj)2+µkf−yk2,(3) where the matrix ˆ Sis given by ˆ S=D−1S. As it can be seen, the criterion is similar to the one presented in (2). However, there are two significant differences. First, the distance in scores between any pair of nodes xiand xjis not any more depending on the degree of these two nodes. Indeed, empirically, we found that keeping the degrees in the pairwise distance lead to worse results. This is due to the fact that our problem is score estimation that should fit some fixed values provided by the labeled images. Therefore, whenever the degrees are different the vectors fi √Dii and fj pDjj will not be equally significant in the distance depicted in the first term of Eq. ( (2)). By recalling the definition of Dii (it is the sum of the i-th row in S), a simple analysis of this term fi √Dii −fj pDjj will show that the most influential samples are the ones being the least similar to the rest. Second, unlike discrete label propagation where it is safe to set the label distribution Yij associated with the unlabeled images to zero vectors, in our case these values (i.e., 5 yi, i = 1, ..., u) are set to the average scores that can be easily known from the labeled images. The solution for fis again similar to the one provided by the LGC discrete label propagation in which the Laplacian matrix is now associated with the graph ˆ S+ˆ ST 2: f= (I+ˆ L/µ)−1y 1.2 Flexible Manifold Embedding Similarly to the LGC method, the Flexible Manifold Embedding (FME) method (Nie et al., 2010) estimates the labels Fby minimizing the following cost function: g(F,W,b) = tr(FTL F) + βtr[(F−Y)TU(F−Y)] + µ(||W||2+γ||XTW+1bT−F||2),(4) where Uis an indicator matrix, that is a diagonal matrix, with its first ldiagonal elements, corresponding to labeled instances, equal to 1, while the last udiagonal elements, corresponding to unlabeled instances, are equal to 0. This assumes that the llabeled images are the first lsamples in the data matrix Xand in the similarity matrix S.W and bdenote the unknown linear regressor which maps the original samples to the label space. As with the LGC method, a closed-form solution can be found by setting the derivatives of gwith respect to W,band Fas 0 (Nie et al., 2010). The solution is given by: b=1 l+uFT1−WTX1(5) W=γ(γXHcXT+I)−1XHcF(6) F=β(βU+L+µγHc−µγ2Q)−1UY,(7) where Q=XT cXc(γXT cXc+I)−1,Xc=X Hcand Hc=I−(1/(l+u))11T. Flexible Manifold Embedding and regression As with the LGC method, this algorithm was originally designed for classification tasks. Nonetheless, it can easily be adapted to work as a regressor. If fand yare real-valued vectors representing the predicted labels and the ground-truth scores, the cost function becomes g(f,w, b) = fTL f +β(f−y)TU(f−y) + µ(||w||2+γ||XTw+b1−f||2).(8) The first term controls the label smoothness, the second one the label fitness and the last term fits a linear regression between features and labels, where ||w||2is a regularization term controlling the complexity of the model (thus, avoiding over-fitting). β,µand γare the parameters controlling the trade-off between all the terms. The solution is given by: b=1 NfT1−wTX1(9) w=γ(γXHcXT+I)−1XHcf(10) f=β(βU+L+µγHc−µγ2Q)−1U y.(11) 6 FME and unseen data One advantage of the FME method, which distinguishes it from many proposed label propagation methods, e.g., from the LGC method, is its capacity of dealing with unseen data. Apart from predicting the labels of the unlabeled data in vector f, it can predict unseen data using the regression model that is already learned in (8). Given the features of the unseen data Xuns, its scores would be: funs =XT unsw+b1. 2 Proposed approach: Non-linear Flexible Manifold Embedding The FME model works directly on the data samples. In many recent machine learning works, it was shown that working with a non-linear representation of the data can improve the final performance of the learner. In our work, we propose to use the column generation trick in order to get the non-linear representation of the original data matrix X. Column generation replaces each sample xiby a vector of similarities of that sample with the samples contained in a fixed set of samples (Klare and Jain, 2013). Very often, the latter set is given by the training samples or a subset of them (Zhang et al., 2018). In our work, we use all training samples as reference samples. The data matrix X is thus replaced by the matrix G= [g1,g2,...,gN], where each vector giis formed by the similarities, i.e. gi=sim(xi,x1), ..., sim(xi,xN)∈RN×N. In our case, we use the Gaussian similarity. This means that Gij =e −||xi−xj||2 2t0σ2, where σ2is a measure of the variability of the data. Concretely, it is set to the mean of the squares of the distances between all pairs of samples. t0is a kernel parameter that can control the similarity function. Therefore, the Non-linear Flexible Manifold Embedding (NFME) can be formulated as follows: g(f,w, b) = fTL f +β(f−y)TU(f−y) + µ(||w||2+γ||GTw+b1−f||2).(12) The NFME method can cope with non-linear data when a linear regression may have a poor performance in the FME. A closed-form solution can be found again by setting the derivatives of gwith respect to f,w, and bas 0. The solution is given by: b=1 NfT1−wTG 1(13) w=γ(γG HcGT+I)−1G Hcf(14) f=β(βU+L+µ γ Hc−µ γ2Q)−1U y (15) where Q=GT cGc(γGT cGc+I)−1,Gc=G Hc, and Hc=I−(1/(l+u))11T. 7 NFME and unseen data Similarly to the FME method, in the NFME method one can easily handle unseen data using the regression term in (12). To do so, given a set of unseen samples xuns i|m i=1, one has to build the similarity matrix of the unseen samples Guns ∈Rm×N, where the element (i, j) is the similarity function of the unseen sample i, xuns i, and the training sample j,xj. Then, the predicted scores, funs ∈Rm, would be funs =GT unsw+b1.(16) 3 Experimental setup 3.1 Datasets Three datasets are used in this work: the SCUT-FBP dataset (Xie et al., 2015), the Multi Modality-Beauty (M2B) dataset (Nguyen et al., 2013), and the SCUT-FBP5500 dataset (Liang et al., 2018). The first was specifically designed for automatic facial beauty perception and contains high resolution front-on face portraits of Asian females. Moreover, the second was developed to evaluate beauty via a face, dressing and/or voice on both Eastern and Western females and each instance in the dataset contains information about the three modalities. However, we are only focusing on the facial images, which unlike the ones in the SCUT-FBP dataset, show very different poses and expressions. This complicates, in consequence, the beauty assessment, which could be found difficult even by a human rater. Figure 2: Examples of face portraits of SCUT-FBP dataset from (Xie et al., 2015). SCUT-FBP dataset: The SCUT-FBP dataset contains 500 high resolution front-on face portraits of Asian females with neutral expressions, simple background and minimal occlusion, as can be seen in Figure 2. These characteristics prevent from taking into account irrelevant factors in the beauty classification task. The beauty rankings (scores) lie in the interval (1, 5) and are the result of averaging various ratings. The ratings were collected among 75 individuals using a web-based tool with an average number of 70 raters per image. The scores approximately follow a normal distribution (Figure 3) with a small peak around 4.5. Raters’ consistency and self-consistency are checked in different ways by the authors of the paper. For instance, low standard deviations in the ratings of each image indicate 8 Figure 3: Histogram of the rating distribution from (Xie et al., 2015). Figure 4: User interface of the attractiveness ranking tool from (Nguyen et al., 2013). rater’s agreement in the perception of beauty. M2B dataset: The Multi-Modality Beauty dataset has been developed to study beauty perception in three different modalities, in dressing, in the face and in the voice, as well as the global beauty perceived when any of these three aspects are combined. Therefore, the dataset contains one face photo, one full body photo and one voice snippet of 1240 females belonging to two ethnic groups: westerners and easterners (620 individuals in each group). In addition, each of the females of the dataset is rated, in the different modalities and their combinations, with various scores in the interval [1, 10]. The ratings were collected among 40 participants, which were split into two groups depending on their ethnicity, so that each of the participants rated females of their own ethnic group. The web tool used for this purpose can be seen in Figure 4. The ratings were obtained using k-wise comparison, which means that the raters are asked to sort kfemales according to their beauty, and then these k-wise ratings were converted into global ratings in the interval [1, 10] by solving an optimization problem to preserve as many pairwise preferences as possible. The drawback of this method of collecting the labels is that, unlike the SCUT-FBP dataset, where we had the ratings of various raters per image, here we have a unique rating. Thus, we cannot really measure the uncertainty of each of the labels, even if it seems to be important, since beauty is not an absolute concept. 9 Table 5: Summary of the performances of the supervised and semi-supervised methods on SCUT-FBP with a 10%–90% data partition. Method MAE ↓RMSE ↓PC %↑-error ↓ 1-NN 0.0956 0.1248 51.59 0.2423 Ridge Regression 0.0946 0.1310 23.89 0.2328 Gaussian -SVR 0.0735 0.0955 71.51 0.1761 LGC 0.0983 0.1336 17.91 0.2470 FME 0.0707 0.0923 72.91 0.1682 NFME 0.0724 0.0946 72.90 0.1711 4.2 M2B dataset In this section, we present the results obtained on M2B dataset, with a configuration of 50% of the samples as labeled data and the other 50% of the samples as the unlabeled/test data. All experiments are carried out doing 10 stratified splits of the data. Table 6 shows the results of applying the three different graph-based label propagation schemes on M2B dataset. In this experiment, we consider three sets: the first set contains 620 images of Eastern subjects, the second set contains 620 images of Western subjects. The third one contains the whole M2B dataset. The best performances are shown in bold. As it can be seen, the FME and NFME methods achieved the best performances. We can also observe that when the two types of faces Eastern and Western are mixed, the performance of all schemes dropped. This suggests that for a computational model the features about beauty are not the same for every ethnicity. Recall that, in M2B dataset, the eastern faces were rated by eastern subjects, and western faces were rated by western subjects. Since machine learning tries to imitate human expertise, using face images belonging to mixed ethnicities and rated by more than one ethnicity will be more difficult than using face images belonging to one ethnicity and rated by that ethnicity. We can also observe that the NFME method gave the best performances for the mixed case. Compared to the SCUT results, the performances obtained on M2B dataset are worse than those obtained with SCUT-FBP dataset. This is due to two main reasons: (i) the images in M2B dataset are more challenging (some faces correspond to mannequin faces), (ii) the rating process is based on ordering ten images at a time using 40 raters. On the other hand, in SCUT-FBP dataset every face image got the opinion of 70 raters on average. Table 7 depicts the comparison between the best MAEs of some supervised methods (Nguyen et al., 2013), which were carried out with 2-fold cross-validation, and ours, which has the same data proportions, i.e., train/test is 50%/50%. The FAT method is a cascaded estimation of the regression defined the DFAT method. Our MAEs are multiplied by 10, because we used the normalized labels (dividing the original score by 10). As it can be seen, the NFME method has provided the best MAE. 16 Table 6: Average performances obtained with three graph-based score propagation schemes: LGC, FME, and NFME. The dataset used is M2B. Set Method MAE ↓RMSE ↓PC %↑ Eastern LGC 0.1502 0.1827 20.51 FME 0.1352 0.1668 44.82 NFME 0.1358 0.1671 44.55 Western LGC 0.1438 0.1742 34.99 FME 0.1141 0.1422 63.38 NFME 0.1132 0.1424 63.22 Both LGC 0.1484 0.1801 22.90 FME 0.1346 0.1665 43.58 NFME 0.1303 0.1624 48.05 Table 7: MAEs obtained with supervised schemes and the proposed three graph-based score propagation schemes: LGC, FME, and NFME. The dataset used is M2B. Method Eastern Western State-of-the art 1-NN 2.11 1.92 Ridge Regression 1.95 1.87 Neural Network 1.80 1.76 F-A-T 1.80 1.69 DFAT 1.77 1.66 Our schemes LGC 1.50 1.42 FME 1.35 1.14 NFME 1.35 1.13 17 4.3 SCUT-FBP5500 dataset In this section, we present the results obtained on the SCUT-FBP5500 dataset. Table 8 shows the results of applying six different methods on the SCUT-FBP5500 dataset. This table depicts the performance obtained with five different experiments. The first four experiments correspond to learning from a given gender and ethnicity. These experiments correspond to Asian Female data (2,000 images), Asian Male data (750 images), Caucasian Female data (2,000 images), and Caucasian Male data (750 images), respectively. The fifth experiment corresponds to the use of the whole dataset (5,500 images). All experiments were conducted using the five-fold cross validation scheme in which 80% of images are used for training and the remaining 20% are used for testing. The features are given by the layer fc6 of the VGG-Face net. From the results depicted in Table 8, we can observe that the best supervised method was the non-linear -SVR method, and the best semi-supervised method was the NFME method. The best PCs were obtained for Asian Female and Caucasian Female models. This can be explained by the fact that these two datasets have 2000 images each, making the size of labeled images equal to 1600 images. On the other hand, the size of the labeled images for the Asian Male and Caucasian Male cases is 600 images. It is interesting to note that the model learned on a mixture of genders and ethnicities have provided a performance that can be slightly worse than that obtained in the other cases. Nevertheless, this performance has not been significantly deteriorated. This can be plausible since ethnicity and gender have different representations. 4.4 Score-based propagation versus label distribution based propagation For some datasets, the face photos may have label distributions instead of a single score. In our case, these label distributions are available for SCUT-FBP and SCUT-FBP5500 in which the distribution is over five levels of beauty. We compare the performance of the NFME method when it propagates single valued scores and when it propagates the label distribution over the same graph. The NFME was chosen since it gave the best results among the semi-supervised methods and since it can perform a basic label distribution propagation. When the NFME method is used for label distribution propagation, the label distributions of labeled images are given by the label matrix Y, and the unknown soft label distributions are given by the matrix F. In these cases, each of these matrices has five columns. In order to compare the performances of the NFME that outputs single valued predicted scores and the NFME that outputs label distributions, the latter are converted to a single valued score. This is achieved by averaging the levels using the obtained probability distribution. Table 9 summarizes the performance of the two NFME variants on the Asian-Female data of the SCUT-FBP5500 dataset. These results were obtained using the five-fold cross validation scheme. As it can be seen, the performance of the variant that estimates label distribution is worse than the variant that propagates a single valued score. One plausible explanation is that NFME estimates soft class label distribution that can be more suitable for discrete classification. While label distribution can be useful for uncertainty estimation, it is still not clear what can be gained when a single valued score should be used. 18 Table 8: Average performances obtained with three graph-based score propagation schemes: LGC, FME, and NFME. The dataset used is SCUT-FBP5500. Subset Method MAE ↓RMSE ↓PC %↑-error ↓ Asian Female 1-NN 0.0923 0.1206 63.09 0.2610 Ridge Regression 0.0580 0.0742 85.41 0.1362 Gaussian -SVR 0.0550 0.0714 86.57 0.1264 LGC 0.1202 0.1427 58.52 0.3554 FME 0.0564 0.0707 86.09 0.1309 NFME 0.0550 0.0713 86.72 0.1256 Asian Male 1-NN 0.0863 0.1182 59.29 0.2248 Ridge Regression 0.0569 0.0739 82.88 0.1249 Gaussian -SVR 0.0538 0.0701 84.68 0.1136 LGC 0.1032 0.1316 53.08 0.2768 FME 0.0557 0.0722 83.72 0.1202 NFME 0.0540 0.0704 84.80 0.1150 Caucasian Female 1-NN 0.0825 0.1080 70.71 0.2133 Ridge Regression 0.0624 0.0778 83.98 0.1395 Gaussian -SVR 0.0550 0.0701 87.27 0.1161 LGC 0.1226 0.1424 62.64 0.3553 FME 0.0554 0.0701 87.07 0.1172 NFME 0.0553 0.0702 87.52 0.1181 Caucasian Male 1-NN 0.0752 0.0978 68.71 0.2003 Ridge Regression 0.0567 0.0733 81.32 0.1316 Gaussian -SVR 0.0505 0.0650 85.53 0.1064 LGC 0.0983 0.1232 59.50 0.2721 FME 0.0506 0.0650 85.46 0.1071 NFME 0.0525 0.0663 85.67 0.1173 Whole dataset 1-NN 0.0864 0.1153 64.52 0.2330 Ridge Regression 0.0570 0.0736 84.50 0.1293 Gaussian -SVR 0.0539 0.0693 86.41 0.1167 LGC 0.1137 0.1376 56.45 0.3239 FME 0.0570 0.0734 84.60 0.1283 NFME 0.0535 0.0691 86.60 0.1151 Table 9: Performance of two NFME variants on Asian-Female data of the SCUT-FBP5500 dataset. MAE ↓RMSE ↓PC ↑-error ↓ NFME (score) 0.0550 0.0713 86.72 0.1256 NFME (label dist.) 0.0750 0.0926 85.35 0.1714 19 5 Conclusions and discussions Current methods for face beauty assessment are fully supervised. A limitation of these approaches is the scarcity of labeled face images. The paper has introduced two main contributions. Firstly, semi-supervised paradigms are proposed for the face beauty prediction problem. We exploit graph-based score propagation methods in order to enrich model learning without the need of additional labeled face images. Secondly, a non-linear flexible manifold embedding for solving the score propagation was proposed. The proposals were tested on three public datasets for face beauty analysis: SCUTFBP, M2B, and SCUT-FBP5500. These experiments as well as many comparisons with supervised schemes show that the non-linear semi-supervised scheme compares favorably with the best supervised scheme. Obviously, the proposed semi-supervised schemes have the limitation that the similarity graph should be computed prior to the estimation of the beauty prediction model, namely the semi-supervised learning model. In other words, if the graph quality is bad (e.g., the graph is very dense), the prediction accuracy might be affected. Therefore, as a future work we envision to overcome this limitation by deploying a joint estimation of the pairwise similarity graph and the unknown prediction model. The other limitation is related to the data themselves. By nature the face beauty prediction problem suffers from imbalanced data. For instance, it is well known that the number of faces with average attractiveness is very high. A remedy to this limitation is to use state-of-the-art data augmentation techniques in order to increase the the least and most attractive faces. The proposed semi-supervised scoring framework opens the door to other image-based applications. It paves the way to virtually all applications to adopt continuous scores instead of the usual discrete labels that are usually used. These applications can be pain level estimation, driver drowsiness detection, subject concentration detection, facial expression recognition, etc. In addition, all applications whose natural output is a number (age estimation, number of present objects, etc.) can directly benefit from the proposed framework. Future work may envision exploring the following research directions. Firstly, we may exploit image similarities in order to rectify some human generated ground-truth beauty scores. Secondly, we may target the joint estimation of the pairwise similarity graph and the unknown prediction model. Thirdly, we would use multiple views as inputs (i.e., multiple image descriptors) to the objective functional that estimates the prediction model in order to improve the prediction results over the use of one single descriptor. In this case, the unknown model will be estimated from a fused graph where the fusion is carried out by auto-weighted schemes. Fourthly, we intend to use multi-task estimation for solving the face beauty prediction problem in a more accurate way. In this kind of schemes, several outputs are simultaneously estimated. For instance, the output can be the gender, the ethnicity and the face beauty score. 20 References datatang. url: http://datatang.com/. Bainbridge, W., Isola, P., and Oliva, A. (2013). The intrinsic memorability of face images. Journal of Experimental Psychology: General, 142(4):1323–1334. Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput., 15(6):1373–1396. Belkin, M., Niyogi, P., and Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. The Journal of Machine Learning Research, 7:2399–2434. Cheng, B., Yang, J., Yan, S., Fu, Y., and Huang, T. (2010). Learning with l1-graph for image analysis. Trans. Img. Proc., 19(4):858–866. Dornaika, F. and Bosaghzadeh, A. (2015). Adaptive graph construction using data selfrepresentativeness for pattern classification. Information Sciences, 325:118–139. Dornaika, F., Dhabi, R., Bosaghzadeh, A., and Ruichek, Y. (2016). Dynamic adaptive graph construction: Application to graph-based multi-observation classification. In IEEE International Conference on Pattern Recognition. Dornaika, F. and El Traboulsi, Y. (2016). Learning flexible graph-based semi-supervised embedding. IEEE transactions on cybernetics, 46(1):206–218. Dornaika, F., Traboulsi, Y. E., and Assoum, A. (2013). Adaptive two phase sparse representation classifier for face recognition. In LNCS 8192. Advanced Concepts for Intelligent Vision Systems. He, R., Zheng, W.-S., Hu, B.-G., and Kong, X.-W. (2011). Nonnegative sparse coding for discriminative semi-supervised learning. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 2849–2856. Klare, B. and Jain, A. (2013). Heterogeneous face recognition using kernel prototype similarities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2(6):1410– 1422. Liang, L., Lin, L., Jin, L., Xie, D., and Li, M. (2018). SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction. arXiv:1801.06345v1 [cs.CV] 19 Jan 2018. Liu, W. and Chang, S.-F. (2009). Robust multi-class transductive learning with graphs. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 381–388. IEEE. Nguyen, T. V., Liu, S., Ni, B., Tan, J., Rui, Y., and Yan, S. (2013). Towards decrypting attractiveness via multi-modality cues. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 9(4):28. 21 Nie, F., Wang, X., Jordan, M. I., and Huang, H. (2016). The constrained laplacian rank algorithm for graph-based clustering. In AAAI Conference on Artificial Intelligence. Nie, F., Xu, D., Tsang, I. W.-H., and Zhang, C. (2010). Flexible manifold embedding: A framework for semi-supervised and unsupervised dimension reduction. IEEE Transactions on Image Processing, 19(7):1921–1932. Parkhi, O. M., Vedaldi, A., Zisserman, A., et al. (2015). Deep face recognition. In BMVC, volume 1, page 6. Raducanu, B. and Dornaika, F. (2014). Embedding new observations via sparse-coding for non-linear manifold learning. Pattern Recognition, 47(1):480–492. Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. (2014). Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813. Sousa, C., Rezende, S., and Batista, G. (2013). Influence of graph construction on semisupervised learning. In European Conferene on Machine Learning, pages 160–175. Xie, D., Liang, L., Jin, L., Xu, J., and Li, M. (2015). Scut-fbp: A benchmark dataset for facial beauty perception. In Systems, Man, and Cybernetics (SMC), 2015 IEEE International Conference on, pages 1821–1826. IEEE. Zhang, Z., Liu, L., Shen, F., Shen, H. T., and Shao, L. (2018). Binary multi-view clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence. Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Sch¨olkopf, B. (2004). Learning with local and global consistency. In Advances in neural information processing systems, pages 321–328. Zhu, X., Ghahramani, Z., and Lafferty, J. (2003). Semi-supervised learning using gaussian fields and harmonic functions. pages 912–919. 22