Integrating digital factory twin and AI for monitoring manufacturing systems through synthetic data generation and vision transformers

Integrating digital factory twin and AI for monitoring manufacturing systems through synthetic data generation and vision transformers Marcello Urgo (2) a, *, Walter Terkaj b a Mechanical Engineering Department, Politecnico di Milano, Via La Masa 1, Milano, 20127, Italy b CNR-STIIMA, Via Alfonso Corti 12, Milano, 20133, Italy ARTICLE INFO Article history: Available online 27 April 2025 ABSTRACT Integrating Digital Twin and Artificial Intelligence technologies is reshaping manufacturing monitoring systems by leveraging synthetic data and advanced computer vision models. This paper presents an approach where a Digital Twin of a factory is used to generate synthetic datasets to train Vision Transformers for object detection and image segmentation in manufacturing processes. The study demonstrates improved accuracy in detecting and monitoring factory assets, validated through synthetic and real-world datasets. An industrial case study further illustrates its potential to identify anomalies. © 2025 The Authors. Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/) Keywords: Manufacturing Digital twin Artificial intelligence 1. Introduction and problem statement The adoption of Artificial Intelligence (AI) in manufacturing is constantly progressing as new models emerge and the reliability of the technology improves, entailing a considerable impact on existing workflows [1,2]. Among AI technologies, computer vision stands out as one of the most promising and mature areas due to its ability to provide accurate and reliable results, supporting a wide range of applications, from automation [3], to human monitoring [4], and quality control processes [5,6]. With advanced algorithms and improved hardware, computer vision systems can detect defects, monitor production lines, and ensure accuracy, ultimately leading to lower costs and increased productivity in manufacturing. The incorporation of deep learning techniques has significantly shaped recent advancements in object detection models. While traditional models based on convolutional neural network (CNN) architectures (e.g., Faster R-CNN and YOLO) have been widely adopted, vision transformers (ViTs) have emerged as a compelling alternative that exploits self-attention mechanisms to process images in a similar way to how transformers have revolutionised natural language processing (NLP) tasks [7]. ViTs demonstrated consistent performance across tasks such as image classification, object detection, and semantic segmentation. They also exhibited robustness to disturbances (e.g., occlusions), making them a reliable choice for real-world applications. Among these models, the Detection Transformer (DETR) uses a transformer encoder-decoder architecture to capture relationships between image features and object queries, thereby simplifying the training process [8]. These state-of-the-art techniques leverage the strengths of transformers to improve generalisation, yielding promising results for complex object detection tasks. However, they demand higher computational effort and larger datasets. The latter is the main barrier to adopting AI in manufacturing, as collecting and managing these large datasets can be challenging, especially with limited data infrastructure. The availability of a digital twin (DT) of the factory, i.e., a digital replica [9] including geometric characteristics but also products, processes, resources and their integrated behaviour [10], has been supporting several applications in manufacturing, ranging from processes to maintenance, ergonomics and system control [11,12], and facilitating the implementation of knowledge-based and datadriven approaches, especially AI. A DT of factory assets can be used to generate synthetic data that support the training of object detection AI models with a wide range of possible applications such as tracking products or general entities, supporting and optimise handling operations, monitoring the progress of production activities, enabling advanced safety and ergonomics analyses [13]. This capability is vital when the actual system is non-existent or inaccessible. This paper addresses monitoring factory objects by identifying their locations within a manufacturing system. It builds upon an existing framework [13] for object detection in manufacturing systems that relies on synthetically generated data produced through a factory DT, enhancing it by integrating ViTs models. The approach is validated within a virtual factory environment. An application to an industrial case is also explored, leveraging the capability of ViTs to perform object detection alongside image segmentation, supporting the implementation of structured vision pipelines enabling the detection of object and possible anomalies. * Corresponding author. E-mail address: [email protected] (M. Urgo). https://doi.org/10.1016/j.cirp.2025.04.037 0007-8506/© 2025 The Authors. Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/) CIRP Annals - Manufacturing Technology 74 (2025) 639643 Contents lists available at ScienceDirect CIRP Annals - Manufacturing Technology journal homepage: https://www.editorialmanager.com/CIRP/default.aspx 2. Methodology The proposed approach (Fig. 1) specialises and extends the training workflow presented in [13]. The first step involves creating a DT model of the relevant factory assets, following a specified data model and the necessary level of detail (activity A1). The data model can be a reference ontology [14]. For each factory asset that needs monitoring (e.g., parts, fixtures, pallets), synthetic data are generated as images (activity A2,Fig. 1) using the DT model and other digital technologies, such as virtual reality (VR) (Sect. 2.1). The synthetic data, possibly together with a set of real pictures of the same assets, are used to train a DETR model for object detection (activity A3, Fig. 1) by leveraging the functionalities of deep learning frameworks (Sect. 2.2). The trained DETR model is tested and validated (activity A4, Fig. 1) and, if the performance metrics (KPIs) meet the requirements, it can be used to detect the selected assets and support monitoring (Sect. 3). The conventional training of object detection models is based on activities A3 and A4 in Fig. 1, but leveraging synthetic data (activity A1 plus A2,Fig. 1) can provide significant benefits [13]. The trained DETR model can also provide panoptic segmentation [6], i.e., identify the pixels in the image associated with the detected objects. This enables the processing of a real dataset to remove the background from the images. The result can be used to effectively train a VGG [15] model for image classification (activity A5,Fig. 1) with reduced data requirements to achieve reasonable performance. 2.1. Generation of synthetic datasets VR tools typically generate a virtual representation of factory assets based on the corresponding DT model. In addition, the VR web application VEB.js [13] provides advanced functionalities to automatically capture the rendered visualisation according to the specific perspective defined by the camera configuration of the VR environment (Fig. 2). The VR geometric model enables the automatic identification of visible assets within the scene and the calculation of their bounding box coordinates as projected onto the viewport. This information can be directly obtained from the rendering engine, simplifying the creation of synthetic images with accurate object annotations. For each relevant visible asset, the annotation contains its identifier, class, and the position and size of the bounding box. VEB.js (Fig. 2) enables both manual and automatic generation of synthetic images. In manual mode, the user controls navigation and camera perspective, while automatic mode completely controls the VR camera, generating images of selected assets from various perspectives and distances. Automatic mode is ideal for creating large training datasets centred on specific assets, whereas manual mode is better suited for generating targeted testing datasets. Arealistic or complex strategy can be used to incorporate relevant factory assets into the VR scene. In the first strategy, the scene represents a realistic context that reflects the actual use of the assets within a manufacturing system. In the second strategy, the scene serves as a background consisting of many intersecting assets usually found in a manufacturing environment (Fig. 2). This strategy, proposed in [16], addresses the need to train object detection models using a dataset with adequate variety [13]. Both strategies can be illustrated in a case related to monitoring a pallet used in assembly operations. Fig. 3 (left) shows the pallet moving along a conveyor, while Fig. 3 (right) displays the pallet against a randomly generated background featuring parts of workstations, conveyors, and robots. 2.2. Training the object detection model A pre-trained DETR model [17] is used as an object detection model. A transfer learning approach has been implemented, initialising the DETR model with pre-trained weights for the transformer and backbone parts and using data for an additional training phase with reduced learning rates. Various synthetic datasets can be generated based on the strategies for the scene background (Sect. 2.1) and the image characteristics. Table 1 lists the datasets used to train various DETR models. The training phase was carried out over 60 epochs, using the PyTorch framework supported by the PyTorch-lightning [18], providing a simplified and direct interface to handle PyTorch, Fig. 1. IDEF0 diagram representing the training workflow based on [12]. Fig. 2. VR representation of a scene built according to the complex strategy using the VEB.js environment. Fig. 3. Generation of synthetic images and bounding boxes for factory objects in realistic (left) and complex (right) environments using VEB.js. 640 M. Urgo and W. Terkaj / CIRP Annals - Manufacturing Technology 74 (2025) 639643 and Supervision libraries [19], to prepare and preprocess the datasets, as well as calculating KPIs for the validation and testing phase. 3. Validation and testing This section demonstrates how a trained DETR model can be tested and validated (activity A4,Fig. 1), while also deriving insights to identify the most effective training strategy. The reference use case is related to monitoring pallets moving through an assembly line producing hinges for the furniture market [14]. Seven DETR models have been trained on the different datasets (Table 1) and tested using only synthetic images from the realistic VR environment, containing pallets in different positions and dimensions in the picture framing, as reported in Fig. 4. For the testing, the following KPIs have been estimated based on the concepts of precision True Positives True PositivesþFalse Positives  , and recall True Positives True PositivesþFalse Negatives  : AP IoU=0.50 : Average precision with IoU = 0.5. It measures the precision of the model by only considering predicted bounding boxes with a value of IoU (intersection-over-union between the detected and ground truth bounding boxes) of at least 50 %. AP: Average Precision. It is the AP across a range of IoU from 0.50 to 0.95. AR: Average Recall. The average fraction of objects detected across a range of IoU from 0.50 to 0.95. The results in Table 2 show that training on data from a realistic environment (D1, Table 2) only provides reasonable but inadequate results, with an AP = 0.917. Using images exclusively from a complex environment performs worse, with AP = 0.870 (D4, Table 2). Better performance is achieved by merging the two datasets, enabling the trained model to generalise more effectively and demonstrate greater robustness. Additionally, converting the images to a grey scale further improved performance by preventing the model from relying on colour features, which might not be relevant in some cases, and emphasising edge features. This might have helped reduce the dimensionality of the data, making it easier for the model to focus on the essential features of the images and increase the detection rate (D2 and D5, Table 2). Also, rescaling the images to a fixed size of 640 by 640 pixels was beneficial. This transformation helped make the datasets consistent so that possible differences in the resolution of the screenshots did not play any influence. This led to an AP equal to 0.961, which is considered an excellent performance (D7, Table 2). Values for AP IoU=0.5 were identical to AP, meaning that detections had a very high overlapping with the ground truth, and the ones used to calculate AP and AP IoU=0.50 were the same. Similar considerations apply to the AR, showing that the model trained with dataset D7 could detect 97.5 % of the objects in the testing images (see Fig. 5). However, the main emerging issue was the occurrence of false positives. Compared to alternative models (e.g., the YOLO class of models [12]), the DETR model demonstrated superior performances, experiencing a minimal number of false positives for non-existing instances (Fig. 5, bottom-right). However, multiple detections of the same pallet with different bounding boxes occurred (Fig. 6). The associated confidence score for these detections is usually very high (see the numbers in the detected bounding boxes in Fig. 6). Adjusting this parameter, i.e., increasing the threshold confidence for detections, is expected to reduce the number of multiple detections. However, completely eliminating this issue may not be possible, as multiple detections also occur with high confidence (Fig. 6, bottom-left). Further investigations will be conducted to address this by leveraging non-maximum suppression (NMS) thresholds [17]. 4. Industrial case The proposed computer vision workflow has been applied to support automatic quality control and anomaly detection in an industrial case. The focus is on electric motors, where winded coils must be assembled in the stator. The quality problem concerns the winding process, where a copper wire is winded around the coil. This can cause the paper to be included within the copper filament and Fig. 4. Characterisation of the test dataset according to the dimension and position of the objects (bounding boxes) to be detected. Table 1 Datasets. Dataset Description # images D1 Realistic env 100 D2 Realistic env + gray-scaling 100 D3 Realistic env + gray-scaling + rescaling 100 D4 Complex env 100 D5 Complex env + gray-scaling 100 D6 Realistic + Complex env + gray-scaling 200 D7 Realistic + Complex env + gray-scaling + rescaling 200 Table 2 KPIs of DETR models trained with different datasets. Dataset AP IoU=0.50 AP AR D1 0.917 0.917 0.952 D2 0.933 0.933 0.957 D3 0.921 0.921 0.952 D4 0.870 0.870 0.900 D5 0.891 0.891 0.923 D6 0.957 0.957 0.969 D7 0.961 0.961 0.975 Fig. 5. Examples of correct detections. Fig. 6. Examples of multiple or wrong detections. M. Urgo and W. Terkaj / CIRP Annals - Manufacturing Technology 74 (2025) 639643 641 compromise the insulation of the copper from the aluminium of the lamination stacks (Fig. 7, top). As these anomalies cannot be recognised during the winding process, non-conform coils would be assembled into the stator, causing the motor to be defective. The objective is to use a computer vision approach to identify anomalies (the defective coils) at the end of the winding process when they are still on the winding machine (Fig. 7, bottom) and discard them. Since multiple types of the same defect can occur, training the model to detect each would be complicated. Thus, a different approach was used. First, a DETR model has been trained (activity A3, Fig. 1) to detect coils using both synthetically generated (activity A2, Fig. 1) and real images of conforming and non-conforming parts, thus training it to detect the coils without discriminating between conforming and non-conforming ones. Hence, leveraging DETR capability to segment an image (activity A4,Fig. 1), the background is removed from real images, obtaining new coil images. These are further partitioned into conform and non-conform ones; thus, a VGG-16 model [15] can be trained for image classification (activity A5,Fig. 1). This model was trained for 10 epochs on a dataset of 400 images (50 % real and 50 % synthetically generated), also using data augmentation techniques leveraging a rotation of §10°; up to 10 % for zoom, width and height shifts; horizontal and vertical flipping; reaching an accuracy of 93.2 % (training) and 82.3 % (validation). The DETR and image classification models trained so far were then used for the described industrial case. Given an image of a coil after the winding phase, the DETR model segments the image and removes the background. The obtained image is analysed through the image classification model to detect non-conform coils. The testing was carried out on 190 images captured from the industrial environment, with different resolutions, framing and orientation of the coils, leading to the results summarised in Fig. 8 (left). The topleft cell in the matrix represents true negatives, i.e., images of conforming coils that are correctly predicted. The bottom-right cell corresponds to the fraction of true positives, i.e., defective coils correctly predicted as defective. The top-right cell represents the fraction of false positive cases, coils erroneously predicted as non-conforming. In contrast, the bottom-left cell reports false negatives, i.e., coils are incorrectly predicted to conform when they are not. Based on this confusion matrix, the precision obtained was 93.8 %, recall 80.0 %, accuracy 87.4 % and specificity (i.e., the true negative rate) equal to 94.7 %. A traditional approach based on edge detection was also implemented. A subset of conforming and defective coils was used to define reference classes, with classification performed using cosine similarity [20]. Due to the need for consistent image dimensions and framing, this method was applied to a limited sample set. A region of interest was defined, focusing on the bottom insulation, along with a threshold to trigger the edge detection. Fig. 8 (right) shows that the edge detection method achieved 64 % precision, 90 % recall, 70 % accuracy, and 50 % specificity. Although the recall was higher than the image classification approach, the low specificity is a critical issue, with half of the conforming coils misclassified as defective. This is mainly due to the flexibility of the insulation paper, which can affect the perceived shape, even if it has not been wound together with the copper filament. Fig. 9 shows an example where paper bending led to a false positive. This is problematic, as the company prioritises minimising false positives, as these coils are discarded immediately, leading to material waste and added labour. In contrast, false negatives can be easily caught during inspection before subsequent assembly stages. 5. Conclusions The proposed workflow leverages the strengths of synthetic data generation and ViTs to achieve improved object detection and image classification performance by leveraging segmentation. Combining synthetic and real datasets proved effective for training ViTs, achieving performance superior to the previous generation of object detection models [13], demonstrating that synthetic data can mitigate the impact of limited data. The industrial case study further affirmed the applicability of the proposed workflow in quality control applications. Nevertheless, assessing performance in real environments deserves further investigation. Future research will also address a more general and structured workflow and tools to support applications in other manufacturing contexts. Declaration of interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. CRediT authorship contribution statement Marcello Urgo: Writing review & editing, Writing original draft, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Walter Terkaj: Writing original draft, Visualization, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Acknowledgements The authors thank B.C. Morlock, B. B€ ohm, G. Pracella, E. Sala, and E. Scalvi for their support and contributions to datasets and methodology, and an anonymous company for the industrial case. This research has been partially funded by the EU Horizon Europe programme under GA 101058505 and 101138930. Fig. 7. Example of a non-conform coil (top) and a conform coil in the winding machine (bottom). Fig. 8. Results of the testing on the industrial case using image classification (left) and a traditional edge detection approach(right). Fig. 9. Example of a conform coil classified as non-conform through the edge detection approach and cosine similarity. 642 M. Urgo and W. Terkaj / CIRP Annals - Manufacturing Technology 74 (2025) 639643 References [1] Gao RX, Kr€ uger J, Merklein M, M€ ohring H-C, V ancza J (2024) Artificial Intelligence in Manufacturing: State of the Art, Perspectives, and Future Directions. CIRP Annals 73(2):723–749. [2] Cao J, Bambach M, Merklein M, Mozaffar M, Xue T (2024) Artificial Intelligence in Metal Forming. CIRP Annals 73(2):561–587. [3] Kr€ uger J, Lehr J, Schl€ uter M, Bischoff N (2019) Deep Learning for Part Identification Based on Inherent Features. CIRP Annals 68(1):9–12. [4] Urgo M, Berardinucci F, Zheng P, Wang L (2024) AI-Based Pose Estimation of Human Operators in Manufacturing Environments. in Tolio T, (Ed.) CIRP Novel Topics in Production Engineering, Lecture Notes in Mechanical Engineering,CIRP Novel Topics in Production Engineering, Lecture Notes in Mechanical Engineering,1, Springer, Cham, 3–38. [5] Caggiano A, Zhang J, Alfieri V, Caiazzo F, Gao R, Teti R (2019) Machine Learningbased Image Processing for on-Line Defect Recognition in Additive Manufacturing. CIRP Annals 68(1):51–454. [6] Zhang Y, Shan S, Frumosu FD, Calaon M, Yang W, Liu Y, Hansen HN (2022) Automated Vision-based Inspection of Mould and Part Quality in Soft Tooling Injection Moulding Using Imaging and Deep Learning. CIRP Annals 71(1):429–432. [7] Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, Xu C, Xu Y, Yang Z, Zhang Y, Tao D (2023) A Survey on Vision Transformer. IEEE Trans Pattern Anal Mach Intell 45:87–110. [8] Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-toEnd Object Detection with Transformers. in Vedaldi A, Bischof H, Brox T, Frahm JM, (Eds.) Computer Vision ECCV 2020, Springer International Publishing, Cham, 213–229. [9] Terkaj W, Annoni M, Martinez BO, Pessot E, Sortino M, Urgo M (2024) Digital Twin for Factories: Challenges and Industrial Applications, Springer Nature Switzerland, Cham255–274. [10] Terkaj W, Gaboardi P, Trevisan C, Tolio T, Urgo M (2019) A Digital Factory Platform for the Design of Roll Shop Plants. CIRP Journal of Manufacturing Science and Technology 26:88–93. [11] Nassehi A, Colledani M, K ad ar B, Lutters E (2022) Daydreaming Factories. CIRP Annals 71(2):671–692. [12] Urgo M, Terkaj W (2020) Formal Modelling of Release Control Policies as a Plug-in for Performance Evaluation of Manufacturing Systems. CIRP Annals 69(1):377– 380. [13] Urgo M, Terkaj W, Simonetti G (2024) Monitoring Manufacturing Systems Using AI: A Method Based on a Digital Factory Twin to Train Cnns on Synthetic Data. CIRP Journal of Manufacturing Science and Technology 50:249–268. [14] K ad ar B, Terkaj W, Sacco M (2013) Semantic Virtual Factory Supporting Interoperable Modelling and Evaluation of Production Systems. CIRP Annals 62(1):443– 446. [15] Simonyan, K., Zisserman, A., 2015, Very deep convolutional networks for largescale image recognition, https://arxiv.org/abs/1409.1556. arXiv:1409.1556. [16] Jhang, Y.-C., Palmar, A., Li, B., Dhakad, S., Vishwakarma, S.K., Hogins, J., Crespi, A., Kerr, C., Chockalingam, S., Romero, C., Thaman, A., Ganguly, S., 2020, Training a Performant Object Detection ML Model on Synthetic Data Using Unity Perception Tools, https://blogs.unity3d.com/2020/09/17/training-a-performant-objectdetection-ml-model-on-synthetic-data-using-unity-computer-vision-tools/. [17] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M., 2024, Transformers: State-of-the-Art Natural Language Processing, https:// github.com/huggingface/transformers. [18] Falcon W (2024) The PyTorch Lightning Team. PyTorch Lightning . v2.5 https:// github.com/Lightning-AI/pytorch-lightning. [19] Roboflow, 2024, Supervision, v0.25.0, https://github.com/roboflow/supervision. [20] Peng Y, Ruan S, Cao G, Huang S, Kwok N, Zhou S (2019) Automated Product Boundary Defect Detection Based on Image Moment Feature Anomaly. IEEE Access 7:52731–52742. M. Urgo and W. Terkaj / CIRP Annals - Manufacturing Technology 74 (2025) 639643 643