AI-Driven Specular Removal for 3D Asset Creation

AI-driven specular removal for 3D asset creation Marco Callieri∗, Massimiliano Corsini∗, Somnath Dutta∗, Daniela Giorgi∗, Marco Sorrenti† ∗National research Council of Italy – Institute of Information Science and Technology (CNR–ISTI) Email: daniela.gior[email protected].it †University of Pisa, Italy Abstract—Specular highlights negatively affect photogrammetric 3D reconstructions. To mitigate this problem, we developed an AI-driven image processing technique able to remove specular highlights. We created a synthetic image dataset that reflects the objects, viewpoints, and specular behaviors found in realworld photogrammetric campaigns, and used it to train a U-Net model that can batch-process input images for photogrammetric reconstruction. The process was tested on both synthetic and real-world photos, demonstrating superior results compared to existing models in the literature. I. INTRODUCTION A crucial element of eXtended Reality (XR) applications is the environment: 3D assets are vital for interaction and significantly impact the user experience. Photogrammetry, the process of creating 3D models from photos, is widely used for its versatility and cost-effectiveness. However, specular highlights can disrupt the reconstruction process: they can reduce calibration accuracy, generate false points during dense matching, and create visual artifacts in texture mapping. To mitigate these issues, controlled lighting environments or heuristics are employed during photography and processing. This paper proposes an alternative solution to mitigate the impact of specular reflections and highlights in 3D photogrammetric reconstruction. To preserve the standard workflow used in industry and research, we introduce an image processing step between image acquisition and 3D reconstruction. A neural network processes the acquired images, removing specular highlights and enhancing the subsequent 3D reconstruction. To train the network, we create a synthetic dataset using 3D models in a rendering engine with varying lighting setups. The resulting realistic visual representations serve as training data, with images containing highlights as input and corresponding images without highlights as targets. We train a U-Net architecture with a custom loss designed to remove highlights while preserving shape features and intricate details. Experiments on both synthetic and real data show promising results compared to three state-of-the-art techniques [1], [2], [3]. The result is a technique that streamlines asset generation from real-world objects without disrupting established tools and procedures. This approach makes the creation of XR environments more cost-effective and scalable. Our main contributions are: •A procedurally-created training dataset of highresolution images simulating the photogrammetry setup for 3D asset creation, featuring real-world objects under various lighting conditions, and image pairs with and without specular highlights; •A custom loss function for training a U-Net architecture to remove specular highlights, preserving details and handling high-resolution images, consistent with photogrammetry-based 3D asset creation; •Results on specular highlight removal from both synthetic and real objects, demonstrating the effectiveness of our approach. This work has been developed within the European project SUN – Social and hUman ceNtered XR [4], launched in December 2022 and involving 18 European academic and industrial partners. SUN is investigating techniques for highquality 3D asset creation to populate XR applications with content featuring faithfully reconstructed geometry and appearance. One use case is rehabilitation from psychosocial issues, where high-quality 3D assets support the variability and adaptation of exercises to patients’ needs. This allows therapists to enrich virtual scenarios with objects from the physical world rapidly. Digitization brings familiar objects into the virtual environment, making it more stimulating and comfortable for patients. II. STATE OF THE ART The challenge of detecting and removing specularities from images has been addressed through specular detection and specular removal. Specular detection identifies highlights to infer light direction, scene geometry, and camera position, thereby reducing the impact of reflections on 3D computation. Specular removal aims to produce images without specular components, enhancing 3D reconstruction. These tasks have been approached using classical computer vision and machine learning techniques [5], [6], [7], [8], [9], [10], and more recently, deep neural networks. Among recent neural-based solutions, Muhammad et al. [11] propose two deep learning architectures for facial specular highlight elimination: Spec-Net, which mitigates high-intensity reflections in low-color images, and Spec-CGAN, which generates diffuse images from RGB inputs. Shi et al. [12] introduce an encoder-decoder CNN for specular highlight removal. Hu et al. [13] utilize Cycle-GAN and matrix factorization to derive highlight masks for specular removal. Shen et al. [14] employ Pix2Pix to eliminate wheel hub surface defects. Guo et al. [15] combine U-Net and LaMa models to remove specular highlights from wine bottle surfaces. We compare our work against three competitors: JSHDR, M2-Net, and DHAN-SHR. JSHDR [1] is a multitasking network that simultaneously detects and removes highlights in natural images. M2-Net [2] employs a three-stage architecture, combining highlight feature extraction, coarse highlight removal, and refined highlight correction for more accurate and context-aware specular removal compared to singlestage methods. DHAN-SHR [3] uses a dual-hybrid attention mechanism to capture both local spectral dependencies and global contextual relationships, enabling specular highlight removal without additional priors or supervision. Despite its complex architecture with specialized attention mechanisms and interconnected hierarchical processing blocks, we achieve comparable or superior results with a much lighter architecture. Most of the works discussed are specialized in a particular domain or task and rely on synthetic datasets tailored to their context. Our goal is to develop a versatile solution that supports photogrammetric 3D reconstruction, applicable to a wide range of objects. Given the limited availability of suitable free datasets and the specific requirements of closerange photogrammetry, creating a context-specific photorealistic dataset has been essential. Section III outlines the process of designing and creating a training dataset for photogrammetric 3D reconstruction, used for training a network for image preprocessing and removal of specular highlights (Section IV). III. SYNTHETIC DATASET GENERATION Real datasets collected from photographic campaigns provide authentic representations but are limited, expensive, and laborious to obtain. Moreover, acquiring images with and without specular reflections requires specialized setups. In contrast, a synthetic dataset generated using 3D models and rendering engines offers greater control and variety and is easier to collect. Although it may not fully capture realworld complexity, it suits resource constraints and the need for specific image pairs with and without specular highlights. We generate a synthetic image dataset using collected 3D models and a rendering engine. Rendering, the process of generating 2D images from a 3D model, simulates light interaction with objects to create realistic visual representations. Rendering engines offer control over the environment and data acquisition conditions, allowing for the creation of specific and repeatable scenarios. We chose Blender [16] for its fast integrated rendering engine, capable of generating lifelike images with realistic optics, reflections, and shadows. Blender provides diverse lighting tools, including directional, area, and spotlights, to simulate various illumination setups. Additionally, its powerful Application Programming Interface (API) enables task automation and precise scene control. The dataset must closely resemble real photogrammetry input, in terms of objects, geometries, materials, colors, lighting conditions, viewpoints, and image resolution. A. 3D Object selection The 3D models in the dataset must meet several criteria: they should be small objects suitable for photogrammetry, created through digitization, and rich in surface information. They should represent common photogrammetry targets, including everyday items (e.g., toys), cultural heritage artifacts (e.g., vases), and natural objects (e.g., vegetables). They must feature well-defined geometries, realistic textures, and associated normal and specular maps to ensure accurate spatially-varying surface behavior. Additionally, they should exhibit a variety of materials, optical characteristics, shapes, colors, and details. Fig. 1. A subset of objects in the synthetic training dataset. Fig. 2. Two example views and two different lighting conditions for the Soccer Shoe object in the synthetic dataset: from left to right, combined (diffuse+specular), only diffuse, only specular. Our dataset comprises 25 3D models meeting the specified criteria. Most of these models (23 out of 25) are sourced from Sketchfab [17], an online platform for publishing 3D models, choosing models with a compatible license (free to use or Creative Commons) and marked as usable for AI applications by the creator. The remaining two models are from our institution’s archive of 3D scans. Figure 1 displays a subset of the dataset objects. B. Scene construction and composition Each model is standardized in terms of size, orientation, position, and rendering materials. We use Python scripts and Blender’s APIs to create scenes with different lighting setups. For each setup, Blender’s Compositing feature allows the simultaneous generation of the images needed to train our network (Figure 2): with highlights (diffuse + specular, input images), without highlights (diffuse only, target images), and highlights only (specular only, employed in the custom training loss defined in Section IV). A single rendering is sufficient to generate all the training data. We used three lighting setups: a single spotlight, four area lights positioned far above the object, and three area lights placed closer around the object. To prevent the bottom from being completely dark, an additional low-intensity light is placed under the model, set not to generate specular highlights. These varying lighting setups are designed to enhance the network’s generalization, as rendered images with diverse lighting conditions help the network learn to remove highlights Fig. 3. Comparative results on synthetic images (test objects unseen during training). For each row: original image, our method, JSHDR [1], M2-Net [2], DHAN-SHR [3]. Our method consistently produces higher-quality images compared to the competitors. JSHDR and M2-Net tend to fill the highlights with a noticeable grey color, and the image sharpness is lost. DHAN-SHR performs effective corrections but ignores some highlights. under different scenarios. For camera placement, we use a hemispherical arrangement, with the camera moving around the object at various heights, pointing towards the object center. This strategy is the most common in photogrammetric reconstruction. We set the number of cameras to 40 to ensure accurate 3D reconstruction and to facilitate learning of varied specular highlights. Camera and light distances are adjusted based on model size to ensure optimal framing and coherent lighting. The rendering resolution is set to 2000×2000, slightly lower than the standard images used in photogrammetry. However, it is sufficient to preserve details in the final images and maintain specular highlights of sufficient size and complexity, while ensuring reasonable rendering and training times. C. Training and test datasets Considering three lighting setups, one camera setup with 40 cameras, and three types of rendered images (with highlights, without highlights, and highlight masks only) for 25 3D models, the final synthetic dataset contains 9000 images. The test set comprises 4 out of the 25 objects, which were not seen during training. The images for the remaining 21 objects were divided into training (80%) and validation (20%) sets. IV. ARCHITECTURE AND LOSS DEFINITION We use a U-Net architecture [18], due to its ability to effectively capture both local and global information. The UNet architecture features an encoder-decoder structure with skip connections, which downsamples the input image and then upsamples the feature maps to generate the output. We also tested Pix2PixHD [19], but found no clear advantages. Therefore, we present the results of the U-Net only. Specular highlights images serve as input images I, and diffuse component images as targets T. We design a custom loss including two terms: L= MSE(I, T ) + k wX i,j σ(Mi,j)(Ii,j −Ti,j )2(1) where MSE is the Mean Squared Error value, Mis the specular-only image, σis a sigmoid activation function with high steepness, kis a constant (set equal to 10 in our experiments) and w=Pi,j σ(Mi,j). The first term in the loss encourages the fidelity of output images to input ones, preventing hallucinations and preserving details. The second term penalizes differences between input and target images in correspondence of highlights. The number of layers and filters in our U-Net is the same as the original implementation. Dropout is set to 50% to prevent overfitting. We use the Adam optimizer with an initial learning rate equal to 0.0002 and an Exponential Learning Rate scheduler with a decay factor of 0.95. Images are divided into overlapping 256×256 patches to handle high-resolution images while ensuring efficient model training and inference. The network is trained using a batch size of 32 for 120 epochs. The best performing network on the validation set is used; validation is done every 4 epochs. The overall training time is 3h 28m on a NVIDIA GeForce RTX 2070 equipped with 8 GB of VRAM. At inference, processing a single 256×256 patch with our method takes about 4.5 milliseconds, summing up to about 0.51 seconds for a 4 MP image with overlapping patches. V. RESULTS From a quantitative point of view, we evaluate the Structural Similarity Index Measure (SSIM) between the output images and the ground-truth images. SSIM assesses image similarity based on luminance, contrast, and structure. In the test set, SSIM is equal to 0.945 ± 0.08, indicating detail preservation and high fidelity of the corrected images to the input images. From a qualitative standpoint, we discuss results on both synthetic objects in the test set and real-world objects to assess the generalization ability. Evaluation criteria include the extent of specular reflection removal, texture preservation, and overall visual fidelity. Figure 3 shows the results on a vase Fig. 4. Our network applies a coherent correction across the different views of the same object, as required by use in photogrammetric reconstruction. Fig. 5. Comparative results on real-world objects. For each row: original image, our method, JSHDR [1], M2-Net [2], DHAN-SHR [3]. JSHDR produces darker highlights without interpolating the color, making its results less suitable for photogrammetry. M2-Net reduces the highlights but loses details. DHAN-SHR performs well, reducing a significant portion of highlights (see the back of the red buddha); however, it introduces noticeable noise (zoom in on the head to appreciate the quality of our results w.r.t DHAN-SHR). and a bretzel model in the test set, providing a comparison with JSHDR [1], M2-Net [2], and DHAN-SHR [3]. Our customized U-Net is effective in removing specular reflections and preserving texture details, resulting in higher overall visual fidelity. Figure 4 shows our performance across all the input images for the bretzel model, demonstrating coherence across the set. This is fundamental, since our goal is photogrammetric 3D reconstruction. Figure 5 shows comparative results on two real-world objects photographed for photogrammetric digitization. In both cases, our method significantly reduce highlights, making the processed images more suitable for subsequent stages of 3D reconstruction. This demonstrates the model’s effectiveness in handling real-world scenarios. In comparison, JSHDR generally produces lower quality results. M2-Net reduces the brightness of the specular, but does not correctly interpolate the image. DHAN-SHR produces results similar to our network, but more noisy in some cases, as shown by the head of the Buddha statue (2nd row of Figure 5). Importantly, we are able to obtain comparative or even better results than DHAN-SHR with a much lighter architecture. VI. CONCLUSIONS We presented a carefully-designed synthetic dataset and a neural network with a custom loss to reduce specular highlights, which hinder photogrammetric 3D model reconstruction. By intervening between image acquisition and reconstruction stages, we respect the standard photogrammetry process. Despite using a synthetic dataset somehow limited (just 25 objects), our solution demonstrated promising performance and generalization ability on both synthetic and real photos compared to state-of-the-art techniques. We plan to expand the dataset with more objects and simulated lighting conditions and make it available to the research community. The highlight removal tool is crucial for creating high-quality 3D assets for XR applications. Regarding practical applications, in the coming months, we will assess the 3D reconstruction performance of our augmented photogrammetric pipeline using real objects required by the rehabilitation pilot use case of the SUN project. ACKNOWLEDGMENT This paper has been supported by Social and hUman ceNtered XR (SUN) project that has received funding by the Horizon Europe Research & Innovation Programme under Grant agreement N. 101092612. Views and opinions expressed in this paper are those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the European Commission can be held responsible for them. REFERENCES [1] G. Fu, Q. Zhang, L. Zhu, P. Li, and C. Xiao, “A multi-task network for joint specular highlight detection and removal,” in Proceedings CVPR, 2021, pp. 7752–7761. [2] Z. Huang, K. Hu, and X. Wang, “M2-net: Multi-stages specular highlight detection and removal in multi-scenes,” ArXiv, vol. abs/2207.09965, 2022. [Online]. Available: https://api.semanticscholar. org/CorpusID:250698700 [3] X. Guo, X. Chen, S. Luo, S. Wang, and C.-M. Pun, “Dual-hybrid attention network for specular highlight removal,” in Proceedings ACM Int. Conf. on Multimedia. ACM, 2024, p. 10173–10181. [4] “Sun project,” Web site: https://www.sun-xr-project.eu/, accessed: 2025-04-30. [5] K. Ikeuchi, D. Miyazaki, R. T. Tan, and K. Ikeuchi, “Separating reflection components of textured surfaces using a single image,” Digitally Archiving Cultural Objects, pp. 353–384, 2008. [6] Q. Yang, J. Tang, and N. Ahuja, “Efficient and robust specular highlight removal,” IEEE Trans. on PAMI, vol. 37, no. 6, pp. 1304–1311, 2014. [7] Q. Lu, E. Fauvet, A. Zakharova, and O. Laligant, “Entire reflective object surface structure understanding based on reflection motion estimation,” Pattern Recognition Letters, vol. 68, pp. 176–182, 2015. [8] J. Guo, Z. Zhou, and L. Wang, “Single image highlight removal with a sparse and low-rank reflection model,” in Proceedings ECCV, 2018, pp. 268–283. [9] G. Fu, Q. Zhang, C. Song, Q. Lin, and C. Xiao, “Specular highlight removal for real-world images,” in Computer Gaphics Forum, vol. 38, no. 7. Wiley Online Library, 2019, pp. 253–263. [10] R. Li, J. Pan, Y. Si, B. Yan, Y. Hu, and H. Qin, “Specular reflections removal for endoscopic image sequences with adaptive-rpca decomposition,” IEEE Trans. on medical imaging, vol. 39, no. 2, pp. 328–340, 2019. [11] S. Muhammad, M. N. Dailey, M. Farooq, M. F. Majeed, and M. Ekpanyapong, “Spec-net and spec-cgan: Deep learning models for specularity removal from faces,” Image and Vision Computing, vol. 93, p. 103823, 2020. [12] J. Shi, Y. Dong, H. Su, and S. X. Yu, “Learning non-lambertian object intrinsics across shapenet categories,” in Proceedings CVPR, 2017, pp. 1685–1694. [13] G. Hu, Y. Zheng, H. Yan, G. Hua, and Y. Yan, “Mask-guided cycle-gan for specular highlight removal,” Pattern Recognition Letters, vol. 161, pp. 108–114, 2022. [14] Z. Shen, H. Dang, M. Sun, and X. Zhou, “Application of generating adversarial networks in high-light removal of wheel hub surface,” in Proceedings ICAMechS. IEEE, 2019, pp. 12–15. [15] S. Guo, X. Wang, J. Zhou, and Z. Lian, “A fast specular highlight removal method for smooth liquor bottle surface combined with u2-net and lama model,” Sensors, vol. 22, no. 24, p. 9834, 2022. [16] “Blender,” Web site: https://www.blender.org, accessed: 2025-04-30. [17] “Sketchfab - the leading platform for 3d & AR on the web,” Web site: https://sketchfab.com/, accessed: 2025-04-30. [18] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Cham: Springer International Publishing, 2015, pp. 234–241. [19] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.