I-MPN: inductive message passing network for efficient human-in-the-loop annotation of mobile eye tracking data

I-MPN: inductive message passing network for efficient human-inthe-loop annotation of mobile eye tracking data Hoang H. Le1,2,3,7, Duy M. H. Nguyen1,4,5,7, Omair Shahzad Bhatti1, László Kopácsi1, Thinh P. Ngo2, Binh T. Nguyen2, Michael Barz1,6 & Daniel Sonntag1,6 Comprehending how humans process visual information in dynamic settings is crucial for psychology and designing user-centered interactions. While mobile eye-tracking systems combining egocentric video and gaze signals can offer valuable insights, manual analysis of these recordings is timeintensive. In this work, we present a novel human-centered learning algorithm designed for automated object recognition within mobile eye-tracking settings. Our approach seamlessly integrates an object detector with a spatial relation-aware inductive message-passing network (I-MPN), harnessing node profile information and capturing object correlations. Such mechanisms enable us to learn embedding functions capable of generalizing to new object angle views, facilitating rapid adaptation and efficient reasoning in dynamic contexts as users navigate their environment. Through experiments conducted on three distinct video sequences, our interactive-based method showcases significant performance improvements over fixed training/testing algorithms, even when trained on considerably smaller annotated samples collected through user feedback. Furthermore, we demonstrate exceptional efficiency in data annotation processes and surpass prior interactive methods that use complete object detectors, combine detectors with convolutional networks, or employ interactive video segmentation. Keywords Human-centered AI, Scene Recognition The advent of mobile eye-tracking technology has significantly expanded the horizons of research in fields such as psychology, marketing, and user interface design by providing a granular view of user visual attention in naturalistic settings1,2. By capturing intricate details of eye movement, this technology provides real-time insights into cognitive processes and user behavior during interactions with physical products or mobile devices. For instance, in educational research, mobile eye-tracking enables the exploration of learners’ gaze behavior in interactive, real-world environments like classrooms and science laboratories3,4. Insights into where students focus their visual attention, therefore, can guide the design of instructional strategies and foster improved learning outcomes5. In this study, we investigate a new approach aimed at enhancing object recognition under interactive mobile eye-tracking, specifically optimizing data annotation efficiency and advancing human-inthe-loop learning models (Fig. 2). Equipped with eye-tracking devices, users generate video streams alongside fixation points, providing visual focus as they navigate through their environment. Our primary aim lies in recognizing specific objects, such as tablet-left, tablet-right, book, device-left, and device-right, with all other elements considered background, as demonstrated in Fig.1. However, the manual analysis of these eye-tracking data is challenging due to the extensive volume of data generated and the complexity of dynamic visual environments, where target objects may overlap and be affected by environmental noise6,7. In clinical, real-world research contexts or educational, the variability of gaze patterns across participants further complicates the extraction of meaningful insights. Additionally, the dynamic nature 1Interactive Machine Learning Department, German Research Center for Artificial Intelligence (DFKI), 66123 Saarbrücken, Germany. 2Mathematics and Computer Science Department, University of Science, VNU-HCM, Ho Chi Minh City, Vietnam. 3Quy Nhon AI Research and Development Center, FPT Software, Quy Nhon, Vietnam. 4Max Planck Research School for Intelligent Systems (IMPRS-IS), 70569 Stuttgart, Germany. 5Machine Learning and Simulation Science Department, University of Stuttgart, 70569 Stuttgart, Germany. 6Applied Artificial Intelligence Department, University of Oldenburg, 26129 Oldenburg, Germany. 7Hoang H. Le and Duy M. H. Nguyen: These authors contributed equally to this work. email: [email protected]; ho_minh_duy[email protected] OPEN Scientific Reports | (2025) 15:14192 1 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports of visual scenes demands precise object identification and segmentation, often requiring extensive manual annotations to account for factors such as occlusion, shifting lighting conditions, and rapid scene changes. These challenges highlight the pressing need for autonomous analytical strategies that can leverage advanced computational techniques to streamline data processing, improve accuracy, and reduce the burden of human intervention. Among these strategies, techniques such as gaze-based clustering8,9, fixation heatmaps10,11, and predictive modeling12 are particularly promising, as they can not only enhance data interpretation but also facilitate real-time applications, such as adaptive learning systems or assistive technologies for individuals with visual or cognitive impairments. The algorithms beyond those methods are largely powered by machine learning, with state-of-the-art architectures such as convolutional neural networks (CNNs)13,14 and recurrent neural networks (RNNs)15,16 achieving remarkable accuracy in predicting gaze trajectories and identifying areas of interest (AOIs) across both static and dynamic environments17,18. Building on these successes, object detection models, particularly those employing multi-scale feature extraction techniques like YOLO19 and Faster R-CNN20, have further enhanced the efficiency of visual attention detection in complex scenes21. Other important directions involve graph neural networks (GNNs)22–24, which utilize graph structures to capture and model the spatial and semantic relationships among objects or regions in images, enabling robust object recognition in dynamic environments. Overall, by automating traditionally manual and time-intensive tasks, these models provide a scalable and robust approach to analyzing eye-tracking data, unlocking broader applications in dynamic and visually intricate environments. Nevertheless, these methods face notable challenges, many of which arise from the inherent variability in human eye movement patterns and contextual dependencies25,26. For example, gaze behavior is highly dynamic, varying across users, tasks, and environmental factors such as occlusions and changes in lighting conditions. This complex interplay of factors often compromises model robustness, particularly in real-world scenarios where such variability is prevalent. To overcome these obstacles, large-scale training datasets are essential for ensuring effective generalization; however, the process of acquiring such datasets is both labor-intensive and time-consuming, posing an additional hurdle for advancing these methods. Moreover, integrating user feedback with individual preferences and situational contexts into machine learning workflows remains a significant bottleneck27. These personalized adaptations are crucial for improving the usability and accuracy of mobile eyetracking systems, yet they often conflict with the need for computational efficiency and real-time responsiveness. Bridging this gap thus requires innovative approaches that balance adaptability with resource constraints, paving the way for models that can seamlessly customize to individual differences while remaining practical for realworld deployment. Fig. 1. Our setup for dataset collection is designed in a laboratory environment, featuring various Areas of Interest (AOIs), such as tablets and experiment stations for electrical circuits. A child equipped with a mobile eye tracker interacts with these stations, allowing for the collection of gaze data essential for research in educational contexts. The study aims to analyze children’s learning processes under both AR-supported and non-AR conditions. To achieve this, our algorithm processes the input data from the eye tracker in a backend service, predicting the real-time objects the user is focusing on and tracking their attention over time. Scientific Reports | (2025) 15:14192 2 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ In this work, we design a new method for interactive mobile eye-tracking as demonstrated in Fig. 2. The training process starts with initial data annotations by leveraging video object segmentation (VoS) techniques28,29. Users are prompted to provide weak scribbles denoting areas of interest and assign corresponding labels in initial frames. Subsequently, the VoS tool autonomously extrapolates segmentation boundaries closest to the scribbled regions, thereby generating predictions for later frames. During a period of time, users interact with the interface, reviewing and refining results by manipulating scribbles or area-of-effect (AoE) labels if they reveal error annotations. In the next phase, we collect segmentation masks and correspondence annotations provided by the VoS tool to define bounding boxes encompassing AoI and their corresponding labels to train recognition algorithms. Our approach, named I-MPN, consists of two primary components: (i) an object detector tasked with generating proposal candidates within environmental setups and (ii) an Inductive Message-Passing Network30–32 designed to discern object relationships and spatial configurations, thereby determining the labels of objects present in the current frame based on their correlations. It is crucial to highlight that identical objects may bear different labels contingent upon their spatial orientations (e.g., left, right) in our settings (Fig. 1, device left and right). This characteristic often poses challenges for methods reliant on local feature discrimination, such as object detection or convolutional neural networks, due to their inherent lack of global spatial context. I-MPN, instead, can overcome this issue by dynamically formulating graph structures at different frames whose node features are represented by bounding box coordinates and semantic feature representations inside detected boxes derived from the object detector. Nodes then exchange information with their local neighborhoods through a set of trainable aggregator functions, which remain invariant to input permutations and are adaptable to unseen nodes in subsequent frames. Through this mechanism, I-MPN plausibly captures the intricate relationships between objects, thus augmenting its representational capacity to dynamic environmental shifts induced by user movement. Given the initial trained models, we integrate them into a human-in-the-loop phase to predict outcomes for each frame in a video. If users identify erroneous predictions, they have the ability to refine the models by providing feedback through drawing scribbles on the current frame using VoS tools, as shown in Fig.3. This feedback triggers the generation of updated annotations for subsequent frames, facilitating a rapid refinement process similar to the initial annotation stage but with a reduced timeframe. The new annotations are then gathered and used to retrain both the object detector and message-passing network in the backend before being deployed for continued inference. If errors persist, the iterative process continues until the models converge to produce satisfactory results. We illustrate such an iterative loop in Fig.2. In summary, we make the following contributions: • Firstly, we introduce I-MPN, an efficient object recognition framework specifically designed to integrate with mobile eye-tracking systems for analyzing gaze behavior in dynamic environments. • Secondly, I-MPN demonstrates exceptional adaptability to user feedback within mobile eye-tracking applications. By leveraging only a small fraction of user feedback data (20%-30%), it achieves performance levels comparable to or surpassing conventional methods that rely on fixed data splits (e.g., 70% training data). Fig. 2. Overview of our human-in-the-loop I-MPN approach. Video frames are processed by an object detector to produce feature maps, bounding boxes, and semantic features, which are then analyzed by the Inductive Message Passing Network for object prediction. A domain expert annotates initial frames, and the video object segmentation module propagates these annotations, reducing manual effort. Users confirm or correct predictions, engaging in a feedback loop (dashed arrow) that updates the model iteratively until it achieves a predefined level of accuracy or reliability performance assessed by the domain expert. Scientific Reports | (2025) 15:14192 3 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ • Thirdly, a comparative analysis with other human-learning approaches, such as object detectors and interactive segmentation methods, highlights the superior performance of I-MPN, especially in dynamic environments influenced by user movement. This underscores I-MPN’s capability to comprehend object relationships in challenging conditions. • Finally, we measure the average user engagement time needed for initial model training data provision and subsequent feedback updates. Through empirical evaluation of popular annotation tools in segmentation and object classification, we demonstrate I-MPN’s time efficiency, reducing label generation time by 60% −70% . We also investigate factors influencing performance, such as message-passing models. Our findings confirm the adaptability of the proposed framework across diverse network architectures. We outline the structure of the paper with related work presented in Section "Related work", followed by a detailed description of our methodology in Section "Methodology", experimental results in Section "Experiments & results", and, finally, the conclusion and future directions in Section "Conclusion and future work". Related work Eye tracking-related machine learning models Many methods rely on pre-trained models to analyze localized features around fixation points. For instance, some map fixations to bounding boxes using object detection models33,34, while others classify small image patches around fixation points with image classification models21,35. These approaches, however, are typically limited to controlled settings where the training data closely matches the target domain21,36. Studies highlight substantial discrepancies between manual and automatic annotations for areas of interest in benchmark datasets like COCO37,38, emphasizing the challenges in applying pre-trained models to diverse real-world scenarios34. Some efforts to fine-tune object detection models for specific domains show promise39,40, yet these lack interactivity and cannot dynamically update during annotation. Global interaction-based methods focus on capturing and utilizing broader contextual information. Traditional semi-automatic annotation strategies often rely on non-learnable feature descriptors like color histograms or bag-of-SIFT features41,42, limiting adaptability. More recently, Kurzhals et al.43 proposed an interactive approach for annotating egocentric eye-tracking data by iteratively searching time sequences based on eye movements and visual features. Their method involves segmenting gaze-focused image patches, clustering them into representative thumbnails, and visualizing these clusters on a 2D plane. While innovative, such methods primarily operate on pre-segmented patches and lack the dynamic modeling capabilities needed for more complex, adaptive environments. Unlike these works, our I-MPN is designed to capture both detailed visual feature representations and broader relational interactions among objects through an inductive messagepassing network, enhancing model robustness under occlusion or significant viewpoint changes. Graph neural networks for object recognition Graph neural networks (GNNs) are neural models designed for analyzing graph-structured data like social networks, biological networks, and knowledge graphs44. Beyond these domains, GNNs can be applied in object recognition to identify and locate objects in images or videos by leveraging graph structures to encode spatial and semantic relations among objects or regions. Through mechanisms like graph convolution45 or attention mechanisms22, GNNs efficiently aggregate and propagate information across the graph. Methods such as GCN46, GAT22,47, KGN23, SGRN48, and RGRN24 demonstrate the ability of GNNs to incorporate contextual reasoning, spatial relationships, and real-time processing into object recognition workflows. Fig. 3. The video object segmentation-based interface allows users to annotate frames using weak prompts like clicks and scribbles, then propagate these annotations to subsequent frames. Scientific Reports | (2025) 15:14192 4 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ Other approaches extend GNNs to address specific challenges in spatial reasoning and dynamic scenarios. Relation Networks49 and Scene Graph Generation50 explicitly model object relationships and generate structured scene representations. Hierarchical GNNs, like HGRN51, integrate low-level visual features with high-level semantics for improved interaction modeling. Additionally, dynamic frameworks such as Dynamic Graph Neural Networks (DyGNN)52, Principal Neighbourhood Aggregation (G-PNA) 53, Gated Graph Sequence Neural Networks (GatedG)54, and Graph Transformer (TransformerG)55 capture both spatial and temporal dynamics for video-based object recognition. However, in mobile eye-tracking scenarios, these methods face two significant challenges. Firstly, the message-passing mechanism typically operates on the entire graph structure, necessitating a fixed set of objects during both training and inference. This rigidity implies that the entire model must be updated to accommodate new, unseen objects that may arise later due to user interests. Secondly, certain methods, such as RGRN24 or TransformerG55, depend on estimating the co-occurrence of object pairs in scenes using large amounts of training data. However, in human-in-the-loop settings, where users typically provide only small annotated samples, this information is not readily available. As a result, the co-occurrence matrices between objects evolve dynamically over time as more annotations are provided by the user. I-MPN tackles these issues by performing message passing to aggregate information from neighboring nodes, enabling the model to maintain robustness to variability in the graph structure across different instances. While there exist works have exploited this idea for link predictions30, recommendation systems56, or video tracking57, we the first propose a formulation for human interaction in eye-tracking setups. Human-in-the-loop for eye tracking Recent works on human-in-the-loop methods for mobile eye tracking have utilized CNNs for object detection and classification35,58,59. These methods incorporate user feedback to enhance model performance, making them more adaptive to real-world scenarios. However, they often face challenges such as high computational demands and the need for extensively annotated datasets59. Additionally, these models can struggle with environmental noise and varying object angles, which can reduce their accuracy60. In contrast, our I-MPN framework combines object detectors with inductive message-passing techniques, offering more robust performance in dynamic environments while being less resource-intensive than traditional CNN-based methods. Methodology Dataset We begin by detailing our setup, including the process of dataset recording and the generation of video-groundtruth annotations used to evaluate our method. Figure 1 illustrates our experimental setup where we record three video sequences captured by different users, each occurring in two to three minutes (Table 2). The users wear an eye tracker on their forehead, which records what they observe over time while also providing fixation points, showing the user’s focus points at each time frame. We are interested in detecting five objects: tables (left, right), books, and devices (left, right). Video ground-truth annotations To generate data for model evaluation, we asked users to annotate objects in each video frame using the video object segmentation tool introduced in Section"User feedback as video object segmentation". Following the cross-entropy memory method as described in29, we interacted with users by displaying segmentation results on a monitor. Users then labeled data and created ground truths by clicking the “Scribble” and “Adding Labels” functions for objects. Subsequently, by clicking the “Forward” button, the VoS tool automatically segmented the objects’ masks in the next frames until the end of the video. If users encountered incorrectly generated annotations, they could click “Stop” to edit the results using the “Scribble” and “Adding Labels” functions again (Fig. 3). Further analysis of the VoS tool is provided in Table 2, which includes a runtime comparison against other methods based on object detection and semantic segmentation. Overview systems Figure 2 illustrates the main steps in our pipeline. Given a set of video frames: (i) the user generates annotations by scribbling or drawing boxes around objects of interest, which are then fed into the video object segmentation algorithm to generate segment masks over the time frames. (ii) The outputs are subsequently added to the database to train an object detector, perform spatial reasoning, and generate labels for appearing objects using inductive message-passing mechanisms. The trained models are then utilized to infer the next frames until the user interrupts upon encountering incorrect predictions. At this point, users provide feedback as in step (i) for these frames (Fig.2 bottom dashed arrow). New annotations are then added to the database, and the models are retrained as in step (ii). This loop is repeated for several rounds until the model achieves satisfactory performance. In the following sections, we describe our efficient strategy for enabling users to quickly generate annotations for video frames (Section"User feedback as video object segmentation") and our robust machine learning models designed to quickly adapt from user feedback to recognize objects in dynamic environments (Section"Dynamic spatial-temporal object recognition"). Scientific Reports | (2025) 15:14192 5 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ User feedback as video object segmentation Annotating objects in video on a frame-by-frame level presents a considerable time and labor investment, particularly in lengthy videos containing numerous objects. To surmount these challenges, we utilize video object segmentationbased methods61,62, significantly diminishing the manual workload. By using cross-video memory29, this method achieved promising accuracy in various tasks ranging from video understanding63, robotic manipulation64, or neural rendering65. In this study, we harness this capability as an efficient tool for user interaction in annotation tasks, particularly within mobile eye-tracking, facilitating learning and model update phases. The advantages of using VoS over other prevalent annotation methods in segmentation are presented in Table 2. Generally, with VoS, users simply mark points or scribble within the Area of Interest (AoI) along with their corresponding labels (Fig. 3). Subsequently, the VoS component infers segmentation masks for successive frames by leveraging spatial-temporal correlations (Fig. 2-left). These annotations are then subject to user verification and, if needed, adjustments, streamlining the process rather than starting from scratch each time. Formally, VoS aims to identify and segment objects across video frames ( {F1,F 2,...,F T} ), producing a segmentation mask Mt for each frame Ft . In the first step, for each frame Ft , the model extracts a set of feature vectors Ft={ft1,f t2,...,f tn} , where each fti corresponds to a region proposal in the frame and n is the total number of proposals. Another memory module maintains a memory Mt={m1,m 2,...,m k} that stores aggregated feature representations of previously identified object instances, where k is the number of unique instances stored up to frame Ft . To generate correlation scores Ct={ct1,c t2,...,c tn} among consecutive frames, a memory reading function R(Ft,Mt−1)→Ct is used. The scores in Ct estimate the likelihood of each region proposal in Ft matching an existing object instance in memory. The memory is then updated via a writing function W(Ft,Mt−1,Ct)→Mt , which modifies Mt based on the current observations and their correlations to stored instances. Finally, given the updated memory and correlation scores, the model assigns to each pixel in frame Ft a label and an instance ID, represented by S(Ft,Mt,Ct)→{(lt1,i t1),(lt2,i t2),...,(ltn,i tn)} , where (lti,i ti) indicates the class label and instance ID for the i-th proposal. Dynamic spatial-temporal object recognition Generating candidate proposals Due to the powerful learning ability of deep convolutional neural networks, object detectors such as Faster R-CNN66 and YOLO19,67 offer high accuracy, end-to-end learning, adaptability to diverse scenes, scalability, and real-time performance. However, they still only propagate the visual features of the objects within the region proposal and ignore complex topologies between objects, leading to difficulties distinguishing difficult samples in complex spaces. Rather than purely using object detector outputs, we leverage their bounding boxes and corresponding semantic feature maps at each frame as candidate proposals, which are then inferred by another relational graph network. In particular, denoting fθ as the detector, at the i-th frame Fi , we compute a set of k bonding boxes cover AoE regions by Bi={bi1,b i2, ..., bik} and feature embeddings inside those ones Zi={zi1,z i2, ..., zik} while ignoring Pi denotes the set of class probabilities for each bounding boxes in Bi where {Bi,Zi,Pi}←fθ(Fi) . The fθ is trained and updated with user feedback with annotations generated from the VoS tool. Algorithm 1. I-MPN forward and backward pass. Scientific Reports | (2025) 15:14192 6 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ Inductive message passing network We propose a graph neural network gϵ using inductive message-passing operations30,31 for reasoning relations among objects detected within each frame in the video. Let Gi=(Vi,Ei) denote the graph at the i-th frame where Vi being nodes with each node vij ←bij ∈Vi defined from bounding boxes Bi . E is the set of edges where we permit each node to be fully connected to the remaining nodes in the graph. We initialize nodefeature matrix Xi , which associates for each vij ∈Vi a feature embedding x v ij . In our setting, we directly use x v ij =z ij ∈Zi taken from the output of the object detector. Most current GNN approaches for object recognition24,48 use the following framework to compute feature embedding for each node in the input graph G (for the sake of simplicity, we ignore frame index): H( l +1) =σ(˜ D − 1 2 ˜ A˜ D − 1 2 H( l )W( l )) (1) where: H( l ) represents all node features at layer , ˜ A is the adjacency matrix of the graph G with added selfconnections, ˜ D is the degree matrix of ˜ A , W( l ) is the learnable weight matrix at layer l, σ is the activation function, H( l +1) is the output node features at layer l+1 . To integrate prior knowledge, Zhao, Jianjun, et al.24 further counted co-occurrence between objects as the adjacency matrix ˜ A . However, because the adjacency matrix ˜ A is fixed during the training, the message passing operation in Eq(1) cannot generate predictions for new nodes that were not part of the training data appear during inference, i.e., the set of objects in the training and inference has to be identical. This obstacle makes the model unsuitable for the mobile eye-tracking setting, where users’ areas of interest may vary over time. We address such problems by changing the way node features are updated, from being dependent on the entire graph structure ˜ A to neighboring nodes N(v) for each node v. In particular, h(l) N(v)= AGG ( ℓ )({ h ( l ) u, ∀ u ∈N( v )}) (2) h(l+1) v = σ ( W (l)· CONCAT ( h (l) v,h (l) N(v))) (3) where: h(l) v represents the feature vector of node v at layer l, AGG is an aggregation function (e.g., Pooling, LSTM), CONCAT be the concatenation operation, h(l+1) v is the updated feature vector of node v at layer l+1 . In scenarios when a new unseen object vnew is added to track by the user, we can aggregate information from neighboring seen nodes vseen ∈N(vnew) by: h(l+1) vnew = σ (W(l)·CONCAT(h(l) vnew , AGG(ℓ)({h(l) vseen }) (4) and then update the trained model on this new sample rather than all nodes in training data as Eq.(1). The forward and backward pass of our message-passing algorithm is summarized in the Algorithm1. We found that such operations obtained better results in experiments than other message-passing methods such as attention network22, principled aggregation53 or transformer68 (Fig. 4b). Fig. 4. Comparative performance analysis. Scientific Reports | (2025) 15:14192 7 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ Algorithm 2. PyTorch-style I-MLE algorithm. End-to-end learning from human feedback In Algorithm2, we present the proposed human-in-the-loop method for mobile eye-tracking object recognition. This approach integrates user feedback to jointly train the object detector fθ and the graph neural network gϵ for spatial reasoning of object positions. Specifically, fθ is trained to generate coordinates for proposal object bounding boxes, which are then used as inputs for gϵ (bounding box coordinates and feature embedding inside those regions). The graph neural network gϵ is, on the other hand, trained to generate labels for these objects by considering the correlations among them. Notably, our pipeline operates as an end-to-end framework, optimizing both the object detector and the graph neural network simultaneously rather than as separate components. This lessens the propagation of errors from the object detector to the GNN component, making the system be robust to noises in environment setups. The trained models are deployed afterward to infer the next frames and are then refined again at wrong predictions, giving user annotation feedback in a few loops till the model converges. In the experiment results, we found that such a human-in-the-loop scheme enhances the algorithm’s adaptation ability and yields comparable or superior results to traditional learning methods with a set number of training and testing samples. Scientific Reports | (2025) 15:14192 8 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ Algorithm 3. User feedback propagation algorithm. Experiments & results Device Our hardware setup utilizes the Pupil Core eye-tracking device1, which users wear to observe their surroundings during the video recording process. The device outputs both video data and fixation points, capturing the user’s focal attention at each moment in time. The videos are displayed on a monitor, where a backend service powered by the VoS tool facilitates the annotation process. Dataset statistics Our study utilizes three distinct video sequences, each recorded by a different user within our controlled environment. The first video spans 169 seconds, yielding 3873 extracted frames. The second video, slightly longer at 183 seconds, comprises 3422 frames. The third sequence, shorter in duration at 118 seconds, contains 2340 frames. This diverse range of video lengths and frame counts ensures a comprehensive dataset for evaluating the performance and adaptability of our method. Metrics The experiment results are measured by the consistency of predicted bounding boxes and their labels with ground-truth ones. In most experiments except the fixation point cases, we evaluate performance for all objects in each video frame. We define AP @α as the Area Under the Precision-Recall Curve (AUC-PR) evaluated at α IoU threshold AP @α= ∫1 0 p(r) dr where p(r) represents the precision at a given recall level r. The mean Average Precision69 is computed at different α IoU ( mAP @α ), which is the average of AP values over all classes, i.e., mAP @α= 1 n n ∑ i=1 (AP @α) i . We provide results for α∈{50,75} . Furthermore, we report mAP as an average of different IoU ranging from 0.5→0.95 with a step of 0.05. To assess the overall accuracy of object classification across the entire scene (Section "Further analysis"), we compute the average accuracy, which reflects the model’s ability to correctly identify and classify objects within a given frame. Specifically, the average accuracy metric is calculated by averaging the classification accuracy for each object in all video frames. This measure is critical in tasks where not only the localization of objects but also their correct classification is important. Model configurations We use the Faster-RCNN66 as the network backbone for the object detector fθ and follow the same proposed training procedure by the authors. The message-passing component gϵ uses the MaxPooling and LSTM aggregator functions to extract and learn embedding features for each node. We use output bounding boxes and feature embedding at the last layer in fθ as inputs for gϵ . The outputs of gϵ are then fed into the Softmax and trained with cross-entropy loss using Adam optimizer70. Human-in-the-loop vs. conventional data splitting learning We investigate I-MPN’s abilities to interactively adapt to human feedback provided during the learning model and compare it with a conventional learning paradigm using the fixed train-test splitting rate. Baselines setup In the conventional machine learning approach (CML), we employ a fixed partitioning strategy, where the first 70% of video frames, along with their corresponding labels, are utilized for training, while the remaining 30% are reserved for testing purposes. We use I-MPN to learn from these annotations. In the humanin-the-loop (HiL) setting, we still utilize I-MPN but with a different approach. Initially, only the first 10 seconds 1 https://pupil-labs.com/products/core Scientific Reports | (2025) 15:14192 9 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ 58. Trajkovska, K., Kljun, M. & Pucihar, K.Č. Gaze2aoi: Open source deep-learning based system for automatic area of interest annotation with eye tracking data. arXiv preprint arXiv:2411.13346 (2024). 59. Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J. & Fernández-Leal, Á. Human-in-the-loop machine learning: a state of the art. Artificial Intelligence Review 56, 3005–3054 (2023). 60. Saeed, A., Spathis, D., Oh, J., Choi, E. & Etemad, A. Learning under label noise through few-shot human-in-the-loop refinement. arXiv preprint arXiv:2401.14107 (2024). 61. Yao, R., Lin, G., Xia, S., Zhao, J. & Zhou, Y. Video object segmentation and tracking: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 1–47 (2020). 62. Zhou, T., Porikli, F., Crandall, D. J., Van Gool, L. & Wang, W. A survey on deep learning technique for video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 7099–7122 (2022). 63. Song, E. et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 (2023). 64. Huang, W. et al. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973 (2023). 65. Tschernezki, V. et al. Epic fields: Marrying 3d geometry and video understanding. Advances in Neural Information Processing Systems 36 (2024). 66. Girshick, R. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, 1440–1448 (2015). 67. Jiang, P., Ergu, D., Liu, F., Cai, Y. & Ma, B. A review of yolo algorithm developments. Procedia Computer Science 199, 1066–1073 (2022). 68. Shi, Y. et al. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509 (2020). 69. Everingham, M., Van Gool, L., Williams, C. K., Winn, J. & Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303–338 (2010). 70. Kingma, D.P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). 71. Kelleher, J.D., Mac Namee, B. & D’arcy, A. Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies (MIT press, 2020). 72. Intel. Computer vision annotation tool (2021). 73. Kukkala, V. K., Tunnell, J., Pasricha, S. & Bradley, T. Advanced driver-assistance systems: A path toward autonomous vehicles. IEEE Consumer Electronics Magazine 7, 18–25 (2018). 74. Baldisserotto, F., Krejtz, K. & Krejtz, I. A review of eye tracking in advanced driver assistance systems: An adaptive multi-modal eye tracking interface solution. In Proceedings of the 2023 Symposium on Eye Tracking Research and Applications, 1–3 (2023). 75. Zhang, L. et al. Learning unsupervised world models for autonomous driving via discrete diffusion. International Conference on Learning Representations (2024). 76. Shi, J.-X. et al. Long-tail learning with foundation model: Heavy fine-tuning hurts. International Conference on Machine (2024). 77. Barz, M., Bhatti, O.S., Alam, H. M.T., Nguyen, D. M.H. & Sonntag, D. Interactive Fixation-to-AOI Mapping for Mobile Eye Tracking Data Based on Few-Shot Image Classification. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23 Companion, 175–178, https://doi.org/10.1145/3581754.3584179 (Association for Computing Machinery, New York, NY, USA, 2023). Event-place: Sydney, NSW, Australia. 78. Jiang, Y. et al. Ueyes: Understanding visual saliency across user interface types. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–21 (2023). 79. Yfantidou, S. et al. The state of algorithmic fairness in mobile human-computer interaction. In Proceedings of the 25th International Conference on Mobile Human-Computer Interaction, 1–7 (2023). 80. Shaily, R., Harshit, S. & Asif, S. Fairness without demographics in human-centered federated learning. arXiv preprint arXiv:2404.19725 (2024). 81. Marinó, G. C., Petrini, A., Malchiodi, D. & Frasca, M. Deep neural networks compression: A comparative survey and choice recommendations. Neurocomputing 520, 152–170 (2023). 82. Xu, C. & McAuley, J. A survey on model compression and acceleration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intelligence 37, 10566–10575 (2023). 83. Bolya, D. et al. Token merging: Your ViT but faster. In International Conference on Learning Representations (2023). 84. Tran, H.-C. et al. Accelerating transformers with spectrum-preserving token merging. arXiv preprint arXiv:2405.16148 (2024). Acknowledgements This work was funded, in part, by the European Union under grant number 101093079 (MASTER), the German Federal Ministry of Education and Research (BMBF) under grant number 01IW23002 (No-IDLE), and the Lower Saxony Ministry of Science and Culture (MWK) as part of the project zug.KI.The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Author contributions Hoang H. Le and Duy M. H. Nguyen implement the proposed method and run experiments on datasets. Omair, Laszlo, and Michael support the collection of datasets, providing tools for annotation. Thinh Ngo supports in providing data annotations Binh T. Nguyen, Michael and Daniel Sonntag guide the project. Declarations Competing interests The authors declare no competing interests. Ethical approval We confirm that this study used a dataset sourced with approval from the Ethics Board of the University of Saarland, ensuring adherence to all relevant ethical guidelines and regulations, and we have the right to use this dataset for our research. The dataset did not involve direct interaction with participants by the authors. The human annotations referenced in the study were performed by employed students who were compensated for their work and conducted the annotations as part of their professional duties. Additional information Correspondence and requests for materials should be addressed to H.H.L. or D.M.H.N. Scientific Reports | (2025) 15:14192 16 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/ Reprints and permissions information is available at www.nature.com/reprints. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit h t t p : / / c r e a t i v e c o m m o n s . o r g / l i c e n s e s / b y - n c - n d / 4 . 0 / . © The Author(s) 2025 Scientific Reports | (2025) 15:14192 17 | https://doi.org/10.1038/s41598-025-94593-y www.nature.com/scientificreports/