Diagonal Partitioning Strategy Using Bisection of Rectangles and a Novel Sampling Scheme

MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX ISSN: 1803-3814 (Printed), 2571-3701 (Online) https://doi.org/10.13164/mendel.2023.k.097 A Hybrid Photorealistic Architecture Based on Generating Facial Features and Body Reshaping for Virtual Try-on Applications Tran Van Duc1, Pham Quang Tien1, Hoang Duc Minh Trieu1, Nguyen Thi Ngoc Anh2, Nguyen Tien Dat1,  1Viettel High Technology Industries Corporation, Hanoi, Vietnam 2VNU University of Engineering and Technology, Hanoi, Vietnam anhnguyenngo[email protected], [email protected]  Abstract Online shopping using virtual try-on technology is becoming popular and widely used for digital transformation because of sustainably sourced materials and enhancing customers’ experience. For practical applicability, the process is required for two main factors: (1) accuracy and reliability, and (2) the processing time. To meet the above requirements, we propose a state-of-the-art technique for generating a user’s visualization of model costumes using only a single user portrait and basic anthropometrics. To start, this research would summarize different methods of most virtual try-on clothes approaches, including (1) Interactive simulation between the 3D models, and (2) 2D Photorealistic Generation. In spite of successfully creating the visualization and feasibility, these approaches have to face issues of their efficiency and performance. Furthermore, the complexity of input requirements and the users’ experiments are leading to difficulties in practical application and future scalability. In this regard, our study combines (1) a head-swapping technique using a face alignment model for determining, segmenting, and swapping heads with only a pair of a source and a target image as inputs (2) a photorealistic body reshape pipeline for direct resizing user visualization, and (3) an adaptive skin color models for changing user’s skin, which ensures remaining the face structure and natural. The proposed technique was evaluated quantitatively and qualitatively using three types of datasets which include: (1) VoxCeleb2, (2) Datasets from Viettel collection, and (3) Users Testing to demonstrate its feasibility and efficiency when used in real-world applications. Keywords: Adaptive Skin Color, Body Reshaping, Head Swapping, Photorealistic, Virtual Try-on. Received: 28 July 2023 Accepted: 31 August 2023 Online: 11 September 2023 Published: 20 December 2023 1 Introduction Due to the global impact of Coronavirus disease (Covid-19) and environmental pollution, the fashion industry is changing quickly to adopt the requirement in driving consumers and sustainable fashion. Offline retailers and department stores had already seen massive declines in sales and met huge pressure in inventory during the lockdown period. As a result, Online purchase is widely booming and numerous apparel companies are leveraging digital transformation to boost their sales. However, it is not easy for consumers to see and feel their actual appearance if they can not physically try products. Generally, there are two main approaches to creating a reality and immersive shopping experience: (1) Using 3D Interactive Simulation, (2) Using 2D Photorealistic Generation. The first method produces 3D models with collisions in virtual space. It could generate an acceptable shape and interact with stable performance, but it requires the preparation for input requirements and the ability to scalable both in horizontal and vertical directions. For the second approach, the speed and input requirement are the main strong points, but, this method meets a big issue in controlling outcomes, which prevents it from being widely adopted around the world. In general, most methods face three main challenges: (1) The complexity to process and manage the input resource data, (2) the accuracy and reality of virtual try-on with clothes, and (3) Maintaining the system’s high stability and reliability. To address these issues, two start-up companies: Zeekit1and 3D Look2have been developed with their patented technology. Zeekit was founded by Yael Vizel, Nir Appleboim, and Alon Kristal in 2014. Zeekit divides a person’s image into thousands of parts using its patented real-time image processing technology. A similar process is applied to clothing, and the analogous points from the two simulations are re-mapped to create the final simulation. The final product depicts a virtual representation of a person wearing clothing 1https://zeekit.me/ 2https://3dlook.ai/ 97 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX that takes into account their body type, fit, and fabric. Meanwhile, the body shape is changed by 3D look algorithms’ creation of a geometrically accurate virtual try-on. The segmentation map of the subject is altered by the semantic generation module, which then warps the clothing mask to cover the appropriate portion of the body. The main contributions of this research are listed below: (1) Proposing a unique and fully automatic architecture system applied for virtual try-on clothes that combines facial feature models with body reshaping, (2) A state-of-the-art technique is presented with high accuracy, stable and ease to integrate and deploy in a cloud server. This research is divided into four main parts: Section 2summarizes previous works on virtual try-on clothes; the proposed method is provided in Section 3; Section 4contains a detailed description of the dataset used for evaluating and results in quantitative and qualitative to prove the method‘s efficiency; section 5concludes the paper and opens future works. 2 Related work Recent research has shown a significant improvement in reconstructing 3D human models based on multiple approaches. In this part, two main methods including (1) Interactive simulation between the 3D models, and (2) 2D Photorealistic Generation are introduced and analyzed in terms of their upsides and downsides. 2.1 3D interactive Simulation 2.1.1 Physics-based Simulation Aphysics-based simulation of clothing basically involves three main processes: Calculation of internal forces, collision detection, and collision reaction forces; the total simulation cost is the result of combining the effects of the three processes. Bender et al.[1] provides a survey to analyze the trade-off between accuracy and cost simulation. Approximation methods for video game applications are reasonable, but they cannot transfer the realistic material behavior required for virtual fitting. Another approach by Fratarcangeli et al. [6] is to improve the efficiency of computing forces and collision constraints using GPU-based while maintaining the simulation accuracy. These approaches provide very well-controllable results with a physics-based model. However, the high computational cost and the performance are big fences for application and scalability. 2.1.2 Network-based Li et al. [8] introduces a mesh-based network, named N-cloth. The main purpose of the method is to predict the 3D Cloth Deformation between cloth mesh and the target obstacle mesh for various scenes. The main advantages are an end-to-end neural network, high performance (30-45 fps with up to 100.000 triangles), and acceptable accuracy. This work inherits, and extends the encoder-decoder architecture [11] and creates a fusion network to deal with the deformation mesh from the input clothes and multi-type of obstacle meshes for improving the overall accuracy. However, preprocessing training data using a physics-based simulator is a time-consuming procedure. Besides, the training process must be re-done if the mesh topology of clothes is changed. Santesteban et al. [14] proposes a learning-based clothing animation with two major steps: (1) Preprocess Data: using a physics-based simulation to generate multiple animated human bodies wearing the same garment, (2) Process: Combining a global fit to a fixed body shape with local garment wrinkles. This research provides an efficient way to generate cloth simulation with high speed but struggles for processing input data and remaining completeness of clothes. 2.2 2D Photorealistic Generation Xiang et al. [20] presents a photorealistic and animated clothing approach by using neural rendering to physically-simulated garments. There are three modules in this method: (1) Base body avatar: predicts body geometry and texture, (2) Cloth simulator: simulates and creates clothing deformation from body geometry, (3) Clothing appearance model: predicts photorealistic clothing texture. This approach is much better than prior work, however, there are still issues in dealing with: (1) data process: The clothing appearance model requires real captured data with the registered body, and clothing geometry (2) the clothes with losing form or multi-layer clothing. Yang et al. [22] proposes a method named: Adaptive Content Generating and Preserving Network (ACGPN). It generates a photorealistic try-on composing of three main parts: (1) Semantic Generation: separate target clothing region and segments the body parts, (2) Clothes Warping: warps the target clothing image to predicted segmentation layout, (3) Content Fusion: Renders the results by utilizing the target clothing image, original clothing mask and body part image. Outcomes are acceptable, but ACGPN has trouble with various kinds of human poses and handles the texture of clothing input. Another approach is proposed by Sarkar et al. [16], called: StylePoseGan for generating a photorealistic multi-view of users from a single input image with explicit control over pose and per-body-part appearance. This method, which is basically an end-to-end model trained with image reconstruction loss and adversarial loss, applies multi-purpose such as garment transfer, head swap, and image interpolation. Due to its architecture, this method has a difficulty in reconstructing images/text in the clothes. DeepFaceLab (DFL) [10] focuses on photorealistic face-swapping using the fundamentals of GAN and StylGan. It would consist of three main phases: (1) Extraction: use face detection, alignment, and segmentation to extract a face from given data (2) Training: 98 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Duc-g2igHX,gA Hybrid Photorealistic Architecture Based on Generating Facial Features and Body Reshaping... Figure 1: System diagram of the proposed method. A structure named LIAE is proposed to achieve the generalization of src and dst (abbreviation of source and destination), through the shared Encoder and Inter and inherit fully information such as lighting (3) Conversion: An Encoder is used to blend and sharp for face generation and re-alignment. These results are quite well in the wild; however, only face regions are swapped and hairstyle, skin tones, and face shapes do not remain. Shu et al. [17] presents a method named: Head Swapper (HeSer) with the main contribution is to produce photorealistic head swapping on different scenes using two parts: (1) Head2Head aligner: is used to migrate expression and position from target to source head, (2) Head2Scene Blender: is used to custom and edit the facial skin color and background texture unmatches. Qualitative and quantitative results prove its efficient approach, but it only focuses on headswapping and does not mention body re-shape. Furthermore, there are several issues during processing with long-hair users and huge demand for input data. 3 Methodology 3.1 Overview The system diagram of the proposed method is illustrated in Fig. 1. Our framework begins with data pre-processing, which produces two database: Sample Head-Pose Images and Sample Head-Pose Key-points. In the main process, the target image and measurement are operated with the pre-processed data through three main modules: body reshaping, skin and facial feature matching, and head swapping. While the body reshaping module resize the source image body to fit the target measurement, the skin and facial feature matching module take the target image facial properties to apply to the source image. Finally, after the source body and facial features are visually suitable with the target, the head swapping module is executed to place the target head on the right acceptable of source body’s neck. Figure 2: 68 face landmarks of the Sample Head-Pose Key points. 3.2 Pre-Process The preprocess module generates a database for changing the head pose algorithm in the head-swap module: Sample Head-Pose Images and Sample Head-Pose Keypoints. Sample Head-Pose Images contain multi-head images of one person that cover almost the natural position of the head. To ensure the reality of the change head pose process, the rotation angle of the head in three axes is limited in range [−45◦, 45◦]. And for the sake of simplicity, the rotation angle is uniformly sampled in a limited range with a 5◦division. The Sample Head-Pose Key-points are 68 face landmarks extracted from Sample Head-Pose Images as shown on Fig. 2. The landmarks are normalized based on the area of the human face. In order to reduce facial distortion in photos captured by short focal length lens cameras, Sample Head-Pose Images are utilized to change the users’ faces from “near pose” to “far pose”. Sample Head-Pose Images are split into two sets: near and far depending on the capture distance from the camera to the user’s face, which is defined as 30cm and 90cm. For each “near” image, there is a head-angle corresponding “far” image and the image pair having the closest head angle to the user’s face is used for the pose-changing task with DaGAN [7] implemented. 99 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Figure 3: System diagram of the Body Reshape module. 3.3 Body Reshape 3.3.1 Generate 2D Triangle Mesh Firstly, a 3D human parametric model is fitted to human images. The fitting process follows SMPLify-X by Pavlakos et al.[9]. The following is a short introduction, and more details are described in [9]. SMPL-X model is fitted to the human body in an image using 2D key points and an iterative optimization approach. The total objective is: E(θ, β, ψ) = EJ(θ, β, K, Jest ) + λθbEθb+λθfEθf +λmhEmh+λαEα+λβEβ+λεEε +λweightEweight +λheightEheight (1) Where: •λθb, λθf, λmh, λα, λβ, λweight, λheight are scalar weights. •θ, β, ψ are the pose, shape, and facial expressions of SMPL-X model. •θb, θf, mhare the pose vectors for the body, face, and two hands, respectively. •K,Jest are the camera parameters and 2D detected key-points. •EJpenalty 2D euclidean distance between projected SMPL-X joints and Jest.Emh, Eθf, Eβ, Eε are simple L2 priors for the hand pose, facial pose, body shape, and facial expressions, penalizing deviation from the neutral state. •Eαis a simple prior penalizing extreme bending only for elbows and knees. Eθbis a VAE-based body pose prior. The interpenetration error term is removed due to making the fitting process very slow and having a little contribution to fitting performance. •Eweight , Eheight are added to the total objective function to penalize the difference between the predicted and real weight, and height of a human. When the optimization process is done, the 3D human model with optimized parameters is projected to 2D sample source points. To ensure the smoothness of the image warping process, the 2D sample source points only take visible points which are detected by using Ray Casting [12] method. Then, a 2D triangle mesh is attached to the image to control image warping. The 2D triangle mesh is constructed based on 2D sample source points and the image bounding box using Delaunay triangulation [13]. 3.3.2 Generate Target Sample Points A simple model is constructed to predict SMPL-X shape parameters from height and weight in this step. The dataset for training is generated by registered SMPL-X model to Vietnam human scan data (Viettel dataset). This dataset includes more than 900 samples with different gender, occupation, region, pose and body mass index to ensure the diversity of the population in Vietnam. The SMPL-X model has two important properties. Firstly, body vertices of SMPL-X have a linear relation with shape space because SMPL-X model is constructed using Principal Component Analysis (PCA). Secondly, the SMPL-X model explores the relationship between the weight and volume of 900 registered sample data, and the linear relation between them is shown in Fig. 4: the more heavy the body is, the more massive the body volume has. This linear relation between weight and volume, combined with the linear relation of body vertices of SMPL-X and shape space come to a conclusion that the relationship between height, weight, and shape space of SMPL-X model is assumed to be linear. Therefore, a simple linear regressor is utilized for the training process. The 2D target sample points are obtained by using projections like 2D source sample points. 3.3.3 Compute Sample Points Deformation The deformations of sample points are represented as affine transformation matrices obtained by optimizing the objective function below: EA=ωpEp(JS, JT) + ωsES(2) Where: •ωp, ωsare scalar weights. •Ep(Jm, Ju) = ∥JT−Js∥2penalty 2D euclidean distance between 2D sample target points JTand source points JS.ES=PiPj∥Ai−Aj∥the term 100 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Duc-g2igHX,gA Hybrid Photorealistic Architecture Based on Generating Facial Features and Body Reshaping... Figure 4: The linear relationship between weight and volume. enforces smooth surface deformation of 2D triangle mesh. •Ai, Ajare affine transformation matrices of two vertices on an edge of triangles. The final result after applying image pixels deformation is shown in Fig. 5. For convenience, the image pixel deformations are pre-computed for a sample height, and weight. And the body in the image can be changed in real-time by scaling the pixel deformations via target height, and weight. 3.4 Head Swapping The diagram of the HeadSwap module is illustrated in Fig. 6. The preprocess data, combined with source and target images, is utilized as the input data and processed in the HeadSwap module. The input data is processed in five stages: (1) Key point Extraction, (2) Best-frame Finding, (3) Animated Head, (4) Head Removal, and (5) Facial Landmark Transformation. Two sets of 68 facial key points corresponding to preprocess data are collected from the source image and target image by implementing 2D Face Alignment [2] in the key point Extraction stage. In total, there are three key point sets belonging to the source image, target image, and preprocess data. Sample Head-Pose Images are generated from multiple face images with the same identity, various poses, and expressions with the aim of representing the majority of human facial poses. The more diverse the Sample Head-Pose Images are, the more accurate the reference key points detection task is. In the next stage, these three key point sets are normalized by pupillary distance. Two key point sets from the source image and target image are compared in sequence with Sample Head-Pose key points to find the most identical frame in Sample Head-Pose Images. This technique performs improvement in creating animated faces compared to other methods [7][3][24]. Animated Head images are created with only one Figure 5: Results of body reshaping process with H and W indicate height and weight, respectively. 101 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Figure 6: Diagram of the HeadSwap module. source image and one destination image without the need for a source video or a few shot images and a target image. Therefore, the processing time is shortened with fewer images or frames. DaGan [7] is integrated into the Animated Head stage with the inputs including two sets of key points and the source image and the output is the animated source head with the angles of the target head. The Animated Image and target image are fed into a segmentation network to detect the head region in each image. The head region in the target image is removed and filled with colors of the surrounding region to avoid blank space before being replaced with the head region in the source image. In the facial landmark transformation stage, the transformation matrix is calculated from the key points obtained in the key point Extraction stage and the image distortion algorithm is utilized to transform the image of the user’s face to match the model’s face in the image. The matrix is calculated based on the spatial matrix transformations as follows: M=scale. R cT u−scale. R.cT m 0 1 (3) R=U.(V)TT(4) U, Σ,(V)T= SVD (lmkm)T.lmku(5) cu= mean (lmku) (6) cm= mean (lmkm) (7) ∥lmkm∥=lm km−cm std (lmkm−cm)(8) ∥lmku∥=lm ku−cu std (lmku−cu)(9) scale = std (lmku−cu) std (lmkm−cm)(10) Where: •Ris the rotation matrix. •lmkuand lmkmare the key point sets of the user and model, respectively. •mean is the average value of the dataset. •std is the standard deviation value. •SV D is the Singular Value Decomposition (SVD), which is a factorization of a matrix into orthogonal matrices and non-square diagonal matrices. 3.5 Skin and Facial Features Matching In terms of output image natural enhancement, a skinchanging technique is implemented in the pipeline to make models’ skin correspond to users’ skin. The skinchanging technique is split into 2 phases: skin tone detection and skin changing as Fig. 7and Fig. 8show, respectively. In the first phase, the human skin tone is determined by three color spaces: RGB (for red, green, and blue), HSV (for hue, saturation, and value), and YCRCB. The RGB color space is utilized for separating images into three channels: red, green, and blue. Three threshold values, which represent the human skin color range in different color spaces, are defined in the H channel of HSV space and two channels of YCRCB space to create three masks: one for HSV and two for YCRCB. After that, these three masks are merged by “AND” logical operators and applied to the input image. The region of the input image covered in the merged mask is considered as “skin region”. In “skin region”, the human skin tone in RGB is collected by dividing respectively three total values of red, blue, and green by the total number of pixels. In the proposed method, both user and model skin tones are calculated for the next phase: skin changing. In human faces, there are some non-skin regions such as eyebrows, eyes, lips, glasses, etc. When applying skin changes to these regions, there is a high risk for the output images to become unnatural. Therefore, a preprocessing stage is added before the color-changing stage to filter out these non-skin regions by using lower and upper thresholds of the human skin color range in the skin-changing phase. After the preprocessing stage, the remaining skin regions are altered to match 102 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Duc-g2igHX,gA Hybrid Photorealistic Architecture Based on Generating Facial Features and Body Reshaping... Figure 7: Tone detection phase. the models’ skin. Firstly, the disparity between two skin tones is expressed in three ways as follows: diff1=cref cori (11) diff2=255 −cref 255 −cori (12) diff3=255(cref −cori) 255 −cori (13) While diff1,diff2, and diff3are three values to define the difference between two skin tones, cref and cori refer to the reference and original color, respectively. After that, the result image iout is created by comparing with cori and processing the original image iin with three values diff1,diff2, and diff3: iout =(iin ×diff1,if iin < cori iin ×diff2+diff3,if iin ≥cori (14) Notice that these formulas above are applied in a single channel only. Therefore, the input needs to be split into three channels: red, green, and blue to calculate each channel’s iout and then compose three values to get the complete output image. Due to differences in light environment between model and user pictures such as intensity, directions, etc, inspired by Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes (EleGANt) [21], there is an additional makeup transfer module after skin changing phase to make the user image match the light condition of the model image. In the makeup transfer module, the skin-changed model image and the user image are defined as reference and source, respectively. Thanks to the makeup transfer module, dark and light regions in the model face are estimated and applied to the user’s face to create the light effect in the user’s face equivalent to the model body. 4 Results This section would divide the results into three separateparts: (1) Evaluate the quantitative in Head Swap, Figure 8: Skin changing phase. (2) Evaluate the quantitative in Reshaping module, (3) Evaluate the quantitative and qualitative in the whole process. 4.1 Datasets and Hardware Specification To evaluate our proposed method, four datasets: two self-collected and two sub-datasets from VoxCeleb2 [5] are utilized. The first self-collected dataset is Vietnam adult scan data (Viettel dataset), which contains scan data of 156 males and 250 females aged from 18 to 60. Viettel dataset is used for body reshape error ratio evaluation. The second dataset includes portraits and full-body images of five men and ten women for qualitative assessment tasks. The other two sub-datasets are extracted from VoxCeleb2 [5] dataset for quantitative result collection tasks. VoxCelebs [5] is a collection of short interview videos (which last less than five minutes) of celebrities around the world. The first sub-dataset is accumulated with 13079 pairs of corresponding source and destination images in VoxCeleb2 [5]. These image pairs are used for calculating the Structural Similarity Index measure (SSIM), the Learned Perceptual Image Patch Similarity (LPIPS), and Peak Signal to Noise Ratio (PSNR) of the output from our proposed method. Due to the fact that the Pose Reconstruction Error (Ep) is only evaluated on the same person, another sub-dataset is collected from these 13079 image pairs by sampling source and destination images on each pair of frames of the same video to create 228 new image pairs. Our proposed method is developed and assessed using an Internet-connected workstation with an Intel Xeon W-2245 CPU, 64GB of memory, and an Nvidia Quadro RTX 8000 GPU running Windows 10 Pro for Workstations with Pytorch, OpenCV, Numpy, and Pycharm installed. 4.2 Head Swap Our proposed method is compared to the previous state-of-the-art models which included: (1) HeSer[17], (2) FOMM[18], (3) LPD[3], (4) Siarohin et al.[19]. Four popular metrics [15][3] are used to make the evaluation: 103 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Table 1: Quantitative comparison on Head Swapping. No. Method EP↓SSIM↑LPIPS↓PSNR↑ 1 FOMM[18] 0.275 0.76 0.18 30.92 2 LPD[3] 0.063 0.52 0.50 28.84 3 Siarohin et al.[19] 0.137 0.73 0.20 30.01 4 HeSer[17] 0.026 0.77 0.19 31.33 5Ours 0.022 0.90 0.09 34.82 (1) Pose reconstruction Error (Ep): represents the difference head pose between synthesized and source image using the facial key points, (2) Structural similarity index measure (SSIM): Estimate the perceived quality of images such as luminance masking, contrast masking, hand contrast (3) Peak Signal-to-Noise Ratio (PSNR): Measure the quality of reconstruction of loss image compression codecs (4) Learned Perceptual Image Patch Similarity (LPIPS): Measure the Semantic perceptual similarity between 2 images via the AlexNet [23]. The table gives information about the quantitative results of every metric of five different methods. Overall, our approach outperforms other methods on every metric. As can be seen from Table 1, Pose Error and Structure similarity index are 0.022 and 0.9, better than approximately 15% and 17% respectively compared to “HerSer” model. The biggest difference comes from LPIPS, the best result before is around 0.18 [18], while ours is 0.094, nearly 48% higher. 4.3 Reshaping To identify the accuracy of the reshaping module, a measurement for quantitative via three ratios in the leg, shoulder, and arm is conducted. OpenPose[4] is used to detect 25 key points and formulas to calculate the ratios are illustrated as follows: legratio =b c(15) shoulderratio =a2 a1+a2 + a3(16) armratio =a1 + a2 + a3 c(17) when a1, a2, a3, b, and care the body part lengths shown in Fig. 9. The boxplots in Fig. 10 and Fig. 11 and Table 3illustrate the error ratio (%) by males and females which includes leg, shoulder, and arm. Overall, the mean error in the whole ratio is around 5% and maintained the same level for 406 samples. However, the maximum errors of leg ratios are approximately 18% both in males and females. As can be seen from Fig. 10 and Fig. 11, the number of errors staying below 8% is much larger than the values above. 4.4 Runtime and Performance Analysis The average processing time of the head-swapping task over 13079 image pairs is 4 seconds, and the response Table 2: Qualitative results of the proposed method Source Image Target Image Result Image H: 1.70m H: 1.68m W: 66kg W: 67kg BMI: 22.84 BMI: 23.74 H: 1.70m H: 1.68m W: 85kg W: 67kg BMI: 29.41 BMI: 23.74 H: 1.70m H: 1.68m W: 72kg W: 73kg BMI: 24.91 BMI: 25.86 H: 1.45m H: 1.62m W: 55kg W: 50kg BMI: 26.20 BMI: 19.05 H: 1.50m H: 1.62m W: 40kg W: 50kg BMI: 17.80 BMI: 19.05 H: 1.60m H: 1.62m W: 48kg W: 50kg BMI: 18.80 BMI: 19.05 104 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Duc-g2igHX,gA Hybrid Photorealistic Architecture Based on Generating Facial Features and Body Reshaping... Table 3: Body error ratio of males and females. Gender Leg error ratio (%) Shoulder error ratio (%) Arm error ratio (%) Mean (%) Male 5.96 4.72 4.80 5.16 Female 5.68 4.77 4.75 5.07 Mean (%) 5.82 4.75 4.78 5.12 Figure 9: Body part length used in body reshaping assessment. time of the body-reshape module on the Viettel dataset is 0.021 seconds, which almost achieves real-time speed. Table 2indicates the qualitative outcomes on different body types both male and female. More information and details are put in the Appendix B. Overall, the images after processing express high fidelity to capture the user’s characteristics with various body types while maintaining the feature of target images. Three body types of source and target are chosen to verify the efficiency of the algorithm with a multiclass of BMI: (1) Overweight: BMI > 25, (2) Normal weight: 18.5< BMI < 25, (3) Underweight: BMI < 18.5. The area in the neck and facial skin color are generated naturally and consistent with the whole body. Furthermore, our algorithm could handle and perform very well on long hair cases. The accuracy of segmentation and head swapping combined with skin color present the best results with high precision. Besides, the features of target images such as belly fat/thin or muscular chest,... are kept and performed nicely while changing the outlook to fit based on the measurement of users through the reshaping model. Figure 10: Male body error ratio. Figure 11: Female body error ratio. 5 Discussions and future work 5.1 Discussions In this research, we present a unique photorealistic framework for virtual try-on applications that combines: (1) Face swapping, (2) Body shape Reconstruction, and (3) Skin adaptive changing. Our approach achieves high-fidelity in quantitative results comparing others methods and ensures the visual appearance of users on clothing after try-on processing. The average runtime is around 5 seconds on a single thread and there is no specific requirement in preparing data for processing with an image size of 1280 x 720 when applying a new type of clothing. Besides that, the ease of package modules brings our work a huge potential to apply and integrate into a complete mobile application and system. Our research is the first all-in-one pho105