Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrations

Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrations Nicola Messina†1Rosario Leonardi†2Luca Ciampi1Fabio Carrara1 Giovanni Maria Farinella2Fabrizio Falchi1Antonino Furnari2 1CNR-ISTI, Pisa, Italy 2University of Catania, Italy †Equal contribution Abstract Pixel-level recognition of objects manipulated by the user from egocentric images enables key applications spanning assistive technologies, industrial safety, and activity monitoring. However, progress in this area is currently hindered by the scarcity of annotated datasets, as existing approaches rely on costly manual labels. In this paper, we propose to learn human-object interaction detection leveraging narrations – natural language descriptions of the actions performed by the camera wearer which contain clues about manipulated objects (e.g., “I am pouring vegetables from the chopping board to the pan”). Narrations provide a form of weak supervision that is cheap to acquire and readily available in state-of-the-art egocentric datasets. We introduce Narration-Supervised in-Hand Object Segmentation (NS-iHOS), a novel task where models have to learn to segment in-hand objects by learning from natural-language narrations. Narrations are then not employed at inference time. We showcase the potential of the task by proposing Weakly-Supervised In-hand Object Segmentation from Human Narrations (WISH), an end-to-end model distilling knowledge from narrations to learn plausible hand-object associations and enable in-hand object segmentation without using narrations at test time. We benchmark WISH against different baselines based on open-vocabulary object detectors and vision-language models, showing the superiority of its design. Experiments on EPIC-Kitchens and Ego4D show that WISH surpasses all baselines, recovering more than 50% of the performance of fully supervised methods, without employing fine-grained pixel-wise annotations. Code and data can be found at https://fpviplab.github.io/WISH/. 1. Introduction Egocentric vision offers a privileged view into humanobject interactions, enabling a fine-grained understanding Input RGB image Input RGB image Model Predictions Model Predictions WISH picking a container from the fridge WISH Training Inference Figure 1. The NS-iHOS task. Models designed to solve this task, like our proposed WISH, learn to segment in-hand objects using human narrations as the sole form of weak supervision; predictions are then made directly from images at test time. of which objects are manipulated and how the user engages with them from their first-person perspective. This ability to detect and segment in-hand objects, those actively manipulated by the user, can enable intelligent systems to gain a deeper understanding of human behavior, offering valuable insights into how individuals engage with their surroundings and supporting a wide range of applications [41], from daily activity analysis [12] to assistive technologies [4], robot-assisted surgeries [54], and industrial safety [44]. Despite the widespread availability of wearable devices, detecting manipulated objects remains significantly underexplored. This is largely due to the scarcity of public datasets and the high cost associated with acquiring high-quality annotations. Most existing approaches typically rely on fully supervised learning [2,12,43,44,49,63], requiring detailed manual annotations. Advancing research in this area necessitates novel strategies that reduce the reliance on manual labeling and facilitates scalable data collection in real-world scenarios. In this paper, we explore a different direction to tackle the lack of suitable fully annotated datasets for detecting 1 arXiv:2509.26004v1 [cs.CV] 30 Sep 2025 in-hand objects. In particular, the most prominent largescale datasets of egocentric videos [10,11,19,20] include narrations — densely-annotated, temporally-localized natural language descriptions of the events within each video moment, which are cheaper to obtain at scale compared to pixel-wise labels. Crucially, as descriptions of the user’s interactions with the environment, narrations inherently mention the specific objects the camera wearer interacts with (see Figure 1-top), offering a reliable source of weak supervision that scales effectively with the size of modern datasets. Towards this direction, we introduce NarrationSupervised in-Hand Object Segmentation (NS-iHOS), a novel task where models learn to recognize and segment objects manipulated by the camera wearer from egocentric images using paired narrations as supervision at training time, but requiring only image inputs at test time (Figure 1). NSiHOS builds on previous formulations [2,63], where methods were tasked with segmenting in-hand objects given full supervision. Unlike these works, we propose scaling these methodologies beyond the bottleneck of carefully crafted manual labels. To tackle the proposed NS-iHOS task, we propose WISH (Weakly Supervised In-Hand Object Segmentation from Human Narrations), an architecture designed to combine the supervisory signal of nouns mentioned in the narrations with the weak spatial understanding of CLIP [42] to determine whether hands are interacting with any of the objects present in the scene. WISH integrates two stages: 1) noun-object alignment, where a CLIP-based shared vision-language space matching nouns extracted from narrations and objects detected with an open-vocabulary detector is learned, and 2) hand-object matching, which distills pseudo-labels obtained from aligned noun-object pairs into a module capable of matching hands and objects, enabling the identification of in-hand objects at test time on raw images without relying anymore on narrations (see Figure 2). We benchmark WISH against baselines combining openvocabulary object detectors and Vision-Language Models (VLM), which offer a current understanding of what state-of-the-art multimodal language-vision approaches can achieve in this context. To mitigate the computational demand of large language models, we also investigate the possibility of using VLMs to process narrations and generate pseudo-labels, which can then be used to train a more compact, off-the-shelf HOI detection models. Experiments on EPIC-Kitchens [11,12] and Ego4D [19,63] show that WISH outperforms all baselines and recovers more than 50% of the performance of a fully supervised method without any access to fine-grained pixel-wise segmentation masks. In sum, the contributions of this paper are as follows: • We introduce NS-iHOS, the novel task of training models to segment in-hand objects using only narrations as supervision; • We propose WISH, a new architecture able to leverage narrations at training time to learn in-hand object segmentation, and able to perform inference on images only; • We present some baselines based on off-the-shelf openvocabulary object detectors and image-language models; • We establish a curated benchmark for NS-iHOS, relying on existing hand-object segmentation datasets based on EPIC-Kitchens and Ego4D datasets, and employing meaningful in-object detection metrics to quantitatively evaluate the proposed models. We aim that our problem definition, the WISH model, and this benchmark will provide a valuable resource for further developments of in-hand object segmentation. Code and benchmark details will be publicly released. 2. Related Works Egocentric Human-Object Interaction Detection Human-Object Interaction detection has been extensively studied within the domain of third-person vision [7,8,21,32,56,62], where algorithms are tasked with detecting humans and their interactions [18]. Significant progress in this area has been driven by the availability of large-scale annotated datasets [7,8,15,21] and standardized evaluation protocols, enabling benchmarking and methodological advancements. In contrast, Egocentric Human-Object Interaction detection remains comparatively underexplored. In this context, the problem of Hand-Object Interaction (HOI) detection was first formulated in [49] as a combined task involving the detection of hands their manipulated objects via bounding boxes, along with the estimation of hand contact states, and adopted by several follow-up studies [16,28,29,44]. The authors of [12] proposed Hand-Object Segmentation (HOS) as an alternative formulation of HOI detection, extending the detection objective to pixel-wise segmentation. A similar approach was followed in [63], where semantic segmentations of hands and interacted objects were predicted from a first-person perspective. Our NS-iHOS task builds on the formulation introduced in [2], which segments in-hand objects, distinguishing whether they are held by the left hand, right hand, or both. Only few approaches aimed to reduce the reliance on fully-supervised data, mainly focusing on synthetic data generation [14,30,38]. In contrast to these works, we seek to reduce reliance on fully supervised annotations by exploiting weakly supervised learning with natural language narrations, enabling scalable training without extensive manual labeling. Language-Guided Object Segmentation Contrastivelylearned vision-language representations like CLIP [42] paved the way towards a deep interaction between images and text and enabled tasks like open-vocabulary image classification [1,61] and image segmentation [13,17,31, 2 58,59], where classes are treated as natural language text rather than discrete categorical items. Similarly, other research branches focused on the weakly-supervised imagephrase grounding task [9,22,52,53], which requires localizing corresponding objects in an image given a phrase, or on fine-grained understanding of text-region correspondences from coarse image-caption pairs [24,37,40]. Recently, these fine-grained localization capabilities have been employed to obtain language-guided image segmentations employing training-free [5,23,27,51,57] or weaklysupervised methodologies [36,45,46]. Object grounding from natural language descriptions has also been recently investigated in egocentric vision [3,26,50]. While the aforementioned works provide an important foundation, the NS-iHOS task requires models to correctly identify and segment the in-hand object, rather than a specific object category provided in natural language. This requires correctly associating the objects with the person’s hands, and assumes the availability of narrations only at training time. Open-Vocabulary Visual Relationship Detection Our work also has affinities with visual relationship detection through vision-language models [6,48,64] and text-guided hand-object interaction [55,60]. We largely differ from these as they focus on category-level object detection and eventually employ ground-truth relational labels for both learning and inference, like structured scene graphs from large-scale manually annotated datasets. Differently, NSiHOS focuses on the specific object instanges manipualted by the camera wearer and employs language supervision only at training time with no narrations at inference. 3. NS-iHOS Task Definition and Benchmark We frame the Narration-Supervised in-Hand Object Segmentation (NS-iHOS) task as learning to segment objects manipulated by a user in egocentric images, relying solely on natural language narrations as supervision during the training phase. This formulation is based on the premise that narrations describing a user’s actions inherently encode informative cues about the objects being handled. For example (see Figure 1), a narration such as “picking a container from the fridge” offers a high-level, yet indirect signal linking the visual scene to interacted objects. The challenge for the model is to translate this coarse semantic guidance into accurate, pixel-level segmentation masks. At evaluation time, the model is required to perform inference using only the visual input, without access to the narration. Task Definition Let training and test sets be denoted as Dtrain ={(Ii,Ni)}Ntrain i=1 ,Dtest ={(Ij,{M∗ e,j}e∈E )}Ntest j=1 , (1) where Ii∈RW×H×3is an egocentric image, Niis its corresponding natural language narration, and M∗ e,j denotes the ground-truth pixel-level mask for interaction type e(available only for evaluation). Let E={L,R,B}denote the interaction types: left hand, right hand, and both hands. The goal of NS-iHOS is to learn a model parametrized by θ fθ:RW×H×3−→ {0,1}W×H×|E|.(2) At inference, for each I∈ Dtest, we obtain {Me}e∈E =fθ(I), Me∈ {0,1}W×H,(3) where ML,MR, and MBare the predicted binary masks for objects interacted with the left hand, right hand, and both hands, respectively, to be compared with ground truth M∗ L, M∗ R, M∗ Bfor evaluation. The narration Niis used only as supervision during training, whereas at inference, fθdepends solely on the input image. Benchmark To evaluate the proposed NS-iHOS task, we propose a benchmark based on two large-scale egocentric datasets: EPIC-Kitchens [10,11] and Ego4D [19]. For EPIC-Kitchens, we adopt the official EPIC-Kitchens VISOR [12] train/val splits1. Specifically, we source narrations from EPIC-Kitchens and ground truth pixel-level annotations of hand and active objects from VISOR. This set contains kitchen-related scenes and interactions with different food items and kitchen tools. For Ego4D, we construct our training set using the official annotations from the Hands and Objects benchmark2, which provides images of handobject interactions and narrations. For evaluation, we rely on the subset of EgoHOS [63] in overlap with Ego4D. From this set, we obtain ground truth pixel-wise masks for hands and interacted objects, while associated narrations are not provided. We ensure that our training set is fully disjoint from these splits to maintain a strict separation between training and evaluation data. This set includes images from different scenarios, inheriting the diversity of Ego4D. The EPIC-Kitchens subset of our benchmark contains 32,857 training and 7,747 test image/narration pairs, whereas the Ego4D subset contains 26,512 training and 1,499 val/test images. We will release all benchmark details to support future evaluations. Evaluation Metrics We evaluate each model by comparing the predicted masks (ML, MR, MB)with the corresponding ground-truth masks (M∗ L, M∗ R, M∗ B). Following the evaluation protocol of [2,63], we report the mean Intersection-over-Union (mIoU) computed over the three interaction classes E={L,R,B}. In addition to classspecific evaluation, we report performance on an auxiliary Either (E) class, defined by matching ground-truth and predicted object masks irrespective of the interacting hand. This aggregated category enables assessment of the model’s ability to detect in-hand objects independently of its accuracy in assigning them to a specific hand. 1The test set of VISOR is not publicly available, therefore we only report results on the validation set. 2https://ego4d-data.org/docs/benchmarks/handsand-objects/ 3 Detector “pouring olive oil into the plate while holding a spoon” CLIP Text Encoder + “in contact with the right hand” Nouns Extraction MLPv MLPt Mask-guided CLIP olive oil plate spoon + “in contact with the left hand” MLPC MLPI Pseudo-labels = object in interaction = in-contact = not in-contact σmax σ max max max max max max max max a) detector and visual backbone b) Stage 1: hand-specific noun-object alignment c) Stage 2: learning hand-object interactions Image I Narration Figure 2. The Architecture of WISH Our model operates in two stages sharing a common backbone. (a) An object segmenter and a CLIP-based backbone extract visual embeddings for all object and hand proposals. (b) In Stage 1, we learn a shared embedding space to align hand-specific noun phrases from narrations with their corresponding visual object embeddings. (c) In Stage 2, we generate pseudolabels from this alignment to train two specialized heads: a Contactness head (C) and a Matching head (M). At test time, only the backbone and Stage 2 are used for narration-free in-hand object segmentation. 4. WISH To tackle the proposed NS-iHOS task, we propose an endto-end weakly-supervised model called Weakly-Supervised In-hand Object Segmentation from Human Narrations (WISH). WISH is guided by two assumptions derived from common human-object interaction datasets [12,63]: (i) narrations describing an activity typically mention the objects being manipulated, grounding the text in the visual scene; and (ii) a person’s hand typically interacts with at most one primary object at any given moment. Based on these priors, WISH employs a two-stage architecture (Figure 2) that first learns a rich vision-language alignment space and then distills this knowledge into a lightweight, vision-only inference model. Specifically: •Stage 1: Hand-specific Alignment. This stage acts as a bridge between the textual and visual domains. It learns to associate specific noun phrases from the narration (e.g., “spoon”) with their corresponding visual object segments in the image, while also discerning which hand (left or right) is involved in the interaction. •Stage 2: Hand-Object Interaction Learning. This stage learns to perform the final NS-iHOS task without relying on narrations. It uses the alignments learned in Stage 1 to generate pseudo-labels, effectively teaching itself to identify which objects are being held and by which hand, using only visual cues. This two-stage design allows WISH to leverage textual supervision during training while performing efficient, narration-free segmentation at inference time. 4.1. Visual and Textual Representation Visual Backbone (Figure 2a) The visual backbone is shared across both stages of WISH. Given an input image I∈RW×H×3, our first goal is to localize all candidate objects and hands that may participate in an interaction. To this end, we apply a pre-trained, class-agnostic object segmenter D, producing a set of Nbinary masks {m1,m2,...,mN}, each highlighting a distinct object or hand region. We embed these regions into a semantically rich feature space using the vision-language model CLIP [42]. Specifically, for each mask mi, we compute a visual embedding vi∈RDv,vi= CLIPv(I,mi), where CLIPvis a mask-guided variant of the CLIP image encoder [50], which applies the mask mito the last attention layer, ensuring that the resulting CLS token captures object-specific features while still being contextualized by the entire scene. To adapt the general-purpose CLIP features to our task, we refine each object embedding viusing a learnable multilayer perceptron (MLP) adapter, denoted as MLPv. This produces the final object representation wi= MLPv(vi), which preserves the semantic richness of CLIP while adapting it to the interaction recognition setting. We apply the same procedure to left and right hands detected through an off-the-shelf hand detector (see Appendix), which are then encoded by the mask-guided CLIP encoder to yield hand embeddings hLand hR. During training, we update only the MLP adapters, keeping the CLIP backbone and object/hand detectors frozen. 4 Textual Backbone To connect visual representations with the narration N, we first extract its key semantic entities. Using standard NLP tools (see Appendix), we parse Nto obtain a set of Mnoun phrases {P1,P2,...,PM}, each corresponding to a distinct object or entity mentioned in the narration. For example, given the narration “I pick the scraper from the bucket of water”, the extracted phrases would include “scraper” and “bucket of water”. However, the extracted noun phrases are generic and do not indicate which hand, if any, is interacting with the referenced object. A key responsibility of the first stage is therefore to derive potential object–hand associations, compensating for this missing information in the narration. To this aim, we enrich each noun phrase Pjusing two hand-specific prompt templates: FL(“[noun] in contact with the left hand”) and FR(“[noun] in contact with the right hand”). For example, from Pj=“spoon”, we generate two distinct phrases: PL j=“spoon in contact with the left hand” and PR j=“spoon in contact with the right hand”. This augmentation step explicitly encodes possible hand–object relationships in language form, enabling the model to later align visual regions with the correct interaction type. These augmented phrases are then encoded using the pre-trained CLIP text encoder CLIPtto get embeddings pL j and pR j. Similar to the visual pipeline, we use a learnable textual adapter, MLPt, to fine-tune these embeddings for our task: q{L,R} j=MLPt(p{L,R} j). This step preserves the broad semantic knowledge of CLIP while specializing the text features for accurate alignment with the visual object and hand embeddings. 4.2. Stage 1: Hand-Specific Alignment The primary goal of this stage (Figure 2b) is to learn the parameters of the visual and textual adapters, MLPvand MLPt, so that embeddings of visually present objects align closely with the corresponding hand-specific noun phrases extracted from the narration. For example, if the user is holding a spoon in their right hand, the visual embedding wiof that spoon should exhibit high cosine similarity with the textual embedding qR jfor the phrase “spoon in contact with the right hand”. A challenge here is the absence of explicit ground-truth pairings between detected objects and noun phrases. To address this, we adopt a Noise-Contrastive Estimation (NCE) framework [39], which encourages correct image–narration alignment while implicitly learning object–phrase associations. The key idea is that the set of object embeddings from an image Igshould, as a whole, be more similar to its paired narration Ngthan to any other narration Ng′from the same training batch. Formally, let {wg1,wg2,...,wgN }denote the visual embeddings extracted from image Ig, and let {˜qh1,...,˜qh(2M)}={qL h1,...,qL hM ,qR h1,...,qR hM }be the re-indexed textual embeddings corresponding to the leftand right-hand noun phrases from narration Nh. We then compute a similarity matrix Agh ∈RN×2M, where each entry measures how well a given visual embedding matches a given textual embedding: Agh ij = sim(wgi,˜qhj ) = wgi ·˜q⊤ hj ∥wgi∥ ∥˜qhj∥.(4) Intuitively, Agh captures all possible pairwise affinities between detected visual regions in image Igand the handspecific noun phrases from narration Nh, forming the basis for our contrastive alignment objective. Given the assumption that each hand can be in contact with at most one object, we enforce that a hand-specific noun phrase is matched to only one visual embedding. Operationally, this means that for each textual embedding (each column of Agh), we retain only its strongest match across all visual embeddings. Formally, we define the vector Bas follows: Bgh j= max iAgh ij (5) Intuitively, this step collapses the object–phrase similarity matrix into a single best-match score per phrase, reflecting the most plausible visual grounding for that phrase under our one-object-per-hand assumption. Next, we aggregate the best-match scores in Bgh into a single global similarity score between image Igand narration Nhby averaging over all 2Mhand-specific noun phrases in Nh: Sgh =1 2M 2M X j=1 Bgh j=1 2M 2M X j=1 max iAgh ij .(6) Here, Sgh reflects the overall alignment between the set of detected objects in Igand the hand-specific phrases from Nh, under the one-object-per-hand assumption. We finally employ a symmetric contrastive objective based on the InfoNCE loss [39,42] to encourage alignment between each image and its paired narration, while pushing apart mismatched pairs in both directions: LNCE =−1 B B X g=1 log exp(Sgh/τ) PB h′=1 exp(Sgh′/τ) −1 B B X h=1 log exp(Sgh/τ) PB g′=1 exp(Sg′h/τ),(7) where τis a temperature parameter and Bis the batch size. 4.3. Stage 2: Learning Hand-Object Interactions With the vision–language alignment module trained in Stage 1, the second stage (Figure 2c) transfers this crossmodal knowledge into a purely visual model capable of 5 identifying in-hand objects at inference time, without relying on narration. We achieve this by first generating pseudolabels in the learned alignment space, which serve as supervisory signals for training two specialized heads designed to directly predict, from visual features alone, the objects being manipulated by the left and right hands, respectively. The first prediction head, referred to as the contactness head, outputs a binary score for each (object i, hand k) pair, answering the question: “Is object iin physical contact with hand k?”. While this head can indicate contact likelihood, it does not enforce the strong assumption that each hand interacts with at most one object. To impose this winnertakes-all constraint, we introduce a second prediction head, the matching head, which addresses the competitive question: “Among all candidate objects, which one is hand k most likely interacting with?” The two heads are complementary: the matching head assigns each hand to its most likely object, while the contactness head verifies whether that object is indeed in contact, thereby handling the nocontact case. The next paragraphs detail the pseudo-label generation process and the architectures of these heads. Pseudo-Label Generation. We define Ab≜Agg as the similarity matrix of matched image-noun pairs. Each entry Ab ij is the cosine similarity between object iand the j-th hand-specific phrase of the same sample. For each image–narration pair in the batch, we process Abto generate pseudo-labels. First, for each detected object iand each hand k∈ {L,R}, we identify the hand-specific noun phrase with the highest similarity to that object. To make the hand index explicit, we reshape Ab ij into Ab ijk, where the original index j∈ {1,...,2Mg}is de-flattened into j∈ {1, . . . , Mg}for the noun phrase and k∈ {1,2} for the hand. We then define the condensed score Ab ik = maxjAb ijk, which represents the strongest textual evidence that object iis in contact with hand k. We hence derive two sets of pseudo-labels: 1. Contact Pseudo-Labels (Lb): Binary labels indicating whether hand kis in contact with object i. We threshold the continuous scores Ab ik using a dynamic threshold ργ, defined as the γ-percentile of all Ab ik in the batch: Lb ik = 1 Ab ik > ργ. 2. Matching Pseudo-Labels (Hb): Object indices indicating the most likely match for each hand, under the oneobject-per-hand assumption: Hb k= arg maxiAb ik. Contactness Head To produce a binary score for each (object i, hand k) pair, we use a dedicated adapter MLPC projecting object features wbi and hand features hbk into a contact-specific space. The dot product of these projected features yields the contactness logit: Cb ik =wC bi ·(hC bk)⊤,(wC bi = MLPC(wbi) hC bk = MLPC(hbk).(8) Applying a sigmoid gives the contact probability Qb ik = σ(Cb ik). We train this head with the binary contact pseudolabels Lbusing the focal loss [33] to address the natural class imbalance: Lb C=−(1 −p)θlog(p), p =(Qb ik,if Lb ik = 1, 1−Qb ik if Lb ik = 0 , (9) where θcontrols the focusing strength. The batch loss is LC=1 BPB b=1 Lb C. Matching Head A second adapter, MLPI, projects object features wbi and hand features hbk into an interactionspecific space. The dot product of these projected features yields the matching logits: Mb ik =wI bi ·(hI bk)⊤,(wI bi = MLPI(wbi) hI bk = MLPI(hbk).(10) To select the single most likely object for each hand, we apply a column-wise softmax over Mb, producing probabilities Pb ik = Softmax(Mb :k)i. We train this head with a cross-entropy loss using the matching pseudo-labels Hb: LI=−1 B·K B X b=1 K X k=1 log Pb ck,where c=Hb k.(11) 4.4. End-to-End Training and Inference The entire WISH model is trained end-to-end by jointly optimizing the objectives from both stages. The total loss is a weighted sum of the three components: Ltot =λNLNCE +λILI+λCLC,(12) where λN,λI, and λCare hyperparameters that balance the contributions of the noun-object alignment, interaction matching, and contactness losses, respectively. At inference time, we discard Stage 1 and the narrations entirely. For a given image, we use Stage 2 to make a prediction. Specifically, we use the matching head to find the object most likely in contact with each hand. If the contactness head outputs a score below 0.5 for that object, the hand is considered in contact with none of the objects. 5. Experiments 5.1. Compared Baselines We compare WISH with different baselines. The first baseline is a state-of-the-art Hand-Object Segmentation (HOS) [12] method fine-tuned on fully-supervised annotations (section F in Table 1). The second set of baselines (section Zin Table 1) employs zero-shot vision-language understanding capabilities of state-of-theart open-vocabulary object detectors like Grounded-SAM 6 EPIC-Kitchens Ego4D Taxonomy E L R B E L R B F) Fully Supervised F1 HOS [12]✓50.29 37.86 37.34 19.25 55.29 34.55 39.43 30.18 O) Oracles with Narrations at Inference Time (EK100 only) O1 GSAM + IoU-contact ✓34.33 21.78 19.29 20.06 - - - - O2 GSAM + LLaVa-contact ✓31.63 22.79 15.56 4.00 - - - - O3 GSAM + LLaVa-props. 26.70 20.64 16.38 11.90 - - - - Z) Zero-shot Z1 SAM + IoU-contact 11.87 9.82 6.69 4.65 13.23 8.52 10.92 3.59 Z2 GSAM + IoU-contact ✓18.72 14.98 12.06 7.81 15.81 12.42 14.79 3.62 Z3 GSAM + LLaVa-contact ✓20.24 16.26 14.52 4.45 16.15 11.62 11.58 8.24 D) Distilled from Oracles D1 HOS [12]on O1 10.11 4.93 3.82 3.54 15.33 10.14 12.52 1.16 D2 HOS [12]on O2 ✓15.05 16.32 4.35 1.98 16.00 12.75 5.95 0.00 D3 HOS [12]on O3 ✓13.01 8.91 10.84 3.05 24.56 13.20 18.94 9.15 W) Ours W1 WISH SAM 21.31 15.73 11.91 13.17 22.55 13.78 16.51 9.96 W2 WISH GSAM ✓27.66 19.49 15.63 13.55 23.61 14.23 20.18 9.96 Table 1. Results on EK100 and Ego4D datasets on the Either (E), Left (L), Right (R), Both (B) mIoU metrics. Oracles cannot be evaluated on Ego4D as narrations are not available on the EgoHOS test set. Taxonomy checkmark indicates that the method employs the original object taxonomy from the underlying dataset to work. (GSAM) [47] and large vision-language models like LLaVa [35]. In particular, IoU-contact (Z1, Z2) employs SAM and GSAM using the dataset-specific taxonomy, respectively, to produce object segmentations; then it leverages the IoU scores to determine if the hand is in contact with the object; LLaVa-contact (Z3) is similar to Z2, but employs LLaVa as a visual reasoner to understand which objects are actually interacting with the hand. The third set of baselines (section D) employs narration-driven pseudo-labels to train a stateof-the-art HOS model [12]. The pseudo-labels are generated through similar pipelines employed in Z, with the difference that they have access to the narrations at inference time. For this reason, we call these pseudo-label generators oracles (section O). As in the case of F1, oracles are not directly comparable to other NS-iHOS methods due to the extra input or supervision, but reported for reference only. O1 and O2, are similar respectively to Z2 and Z3, but employ narrations at inference time instead of the specific dataset taxonomy; O3, instead, employs LLaVa with the frame narration injected within the context to produce a suitable taxonomy to feed GSAM, and then leverages the IoU score to determine the interactions. For fair comparisons, we instantiate WISH, with both SAM [25] and GSAM [47]. 5.2. Results EPIC-Kitchens On the left part of Table 1, we can notice that our method (WISH) achieves the best overall performance, significantly outperforming zero-shot and distillation-based approaches, and even surpassing some oracle baselines that have access to narrations at inference time in the Both metric. Compared to the fully supervised HOS [12], WISH recovers about 50% of the performance despite not relying on manually annotated data during training. Oracle methods (O1–O3) act as upper bounds for related zero shot methods by exploiting narrations at inference time, which are typically not available in realworld applications. Their strong performance in some categories highlights the potential of such privileged information, though it limits their practical applicability. Notably, O1 (GSAM + IoU-contact) achieves the best results among the oracles, indicating that simple spatial heuristics combined with narration-driven segmentation models can be extremely effective. GSAM variants outperform SAM variants (both in WISH and in the baselines) across all metrics, highlighting the benefit of using GSAM over the standard SAM. In particular, the taxonomy awareness of GSAM helps generate more semantically relevant object proposals, improving alignment with narrated actions. Ego4D In the right section of Table 1, we observe similar trends as in EPIC-Kitchens: WISH outperforms zeroshot and distillation-based baselines across most categories, achieving particularly strong results in the Right-O and Both classes. As in the previous benchmark, WISH clearly benefits from using GSAM over SAM, highlighting the role of taxonomy-aware proposals. Distilled models show competitive performance, especially when trained on richer 7 1 2 3 4 5 6 7 8 Figure 3. Qualitative results of WISH on EPIC-Kitchens (top) and Ego4D (bottom). pseudo-labels. In particular, D3 outperforms our method on the generic Either metrics, but still falls short when accurate discrimination between left/right/both is required. Overall, these results highlight the effectiveness and robustness of WISH, demonstrating strong generalization across scenarios without manual supervision or inference-time narrations—making it a promising solution for NS-iHOS. 5.3. Ablations and Qualitative Results Role of different components In Table 2, we validated the two main components of our architecture by enabling or disabling loss functions for Stage 1 (LNCE) and Stage 2 (LIand LC). Results highlight that stages can also work in isolation. However, end-to-end training that employs both components is crucial for achieving the best performance. Pseudo-labels evaluation In Table 3, we measured the quality of the pseudo-labels in output from the first stage of our architecture. Notice that, to perform this measurement, we require the assumption that narrations are available at inference time. Specifically, we reported the results using both SAM and GSAM detectors, comparing the generated pseudo-labels with the final outputs from the second stage. As we can notice, the pseudo-labels are overall weaker than the final results. This is reasonable, as the role of the second stage is to learn to distill relevant signals from the narrationguided pseudo-labels, which may contain strong noise. Qualitative Results Figure 3presents qualitative results of WISH on EPIC-Kitchens (top) and Ego4D (bottom) datasets. Our approach successfully segments objects of varying sizes, accurately identifying both large (images 3, 7) and small ones (images 2, 5, 6), demonstrating robustness to scale variations. WISH also handles complex scenes with multiple objects (images 1 and 8) and bimanual interactions (image 3). In the fourth image, WISH mistakenly identifies a non-interacted object, showing that distinguishing between interacted and nearby objects remains challenging. LNCE LCLIE L R B ✓16.61 11.87 10.35 5.45 ✓ ✓ 21.85 16.42 14.14 8.59 ✓ ✓ 16.06 11.12 9.37 5.50 ✓ ✓ ✓ 27.66 19.49 15.63 13.55 Table 2. Effect of the three losses on the EPIC-Kitchens dataset. E L R B Pseudo-labels from Stage-1 (Narrations at Inference Time) WISH SAM S1 14.73 11.04 8.98 10.06 WISH GSAM S1 26.28 17.82 14.71 13.83 Final output from Stage-2 WISH SAM 21.31 15.73 11.91 13.17 WISH GSAM 27.66 19.49 15.63 13.55 Table 3. Performance of pseudo-labels (obtained by employing narrations at inference time) on the EPIC-Kitchens dataset. 6. Conclusions We introduced NS-iHOS, a novel task for in-hand object segmentation from egocentric images supervised only by natural language narrations, addressing the limitations of manual annotation. To tackle this task, we proposed a two-stage framework that uses CLIP-based alignment to generate pseudo-labels and learn hand-object associations without the need for narrations at test time. Experiments on EPIC-Kitchens and Ego4D show that our model outperforms baselines and recovers over 50% of the performance of fully supervised methods. Our work establishes a new benchmark for scalable, narration-driven understanding, with future plans to extend this approach to egocentric videos and larger datasets. 8 Acknowledgments This work was partially funded by: Spoke 8, Tuscany Health Ecosystem (THE) Project (CUP B83C22003930001), funded by the National Recovery and Resilience Plan (NRRP), within the NextGeneration Europe (NGEU) Program; SUN – Social and hUman ceNtered XR (EC, Horizon Europe No. 101092612); the project Future Artificial Intelligence Research (FAIR) – PNRR MUR Cod. PE0000013 - CUP: E63C22001940006. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support (GEPPETHO project). References [1] Philipp Allgeuer, Kyra Ahrens, and Stefan Wermter. Unconstrained open vocabulary image classification: Zero-shot transfer from text to image via clip inversion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 8217–8228. IEEE, 2025. 2 [2] Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. CVPR, 2025. 1,2,3 [3] Siddhant Bansal, Michael Wray, and Dima Damen. Hoi-ref: Hand-object interaction referral in egocentric vision. arXiv preprint arXiv:2404.09933, 2024. 3 [4] Alejandro Betancourt, Pietro Morerio, Carlo S Regazzoni, and Matthias Rauterberg. The evolution of first person vision methods: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 25(5):744–760, 2015. 1 [5] Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding Everything: Emerging Localization Properties in Vision-Language Transformers. In CVPR, 2024. 3 [6] Emanuele Bugliarello, Aida Nematzadeh, and Lisa Hendricks. Weakly-supervised learning of visual relations in multimodal pretraining. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3052–3071, 2023. 3 [7] Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recognizing human-object interactions in images. In International Conference on Computer Vision, pages 1017–1025, 2015. 2 [8] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In Winter Conference on Applications of Computer Vision, pages 381–389, 2018. 2 [9] Keqin Chen, Richong Zhang, Samuel Mensah, and Yongyi Mao. Contrastive learning with expectation-maximization for weakly supervised phrase grounding. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8549–8559, 2022. 3 [10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, pages 720–736, 2018. 2,3 [11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. IJCV, pages 1–23, 2021. 2,3 [12] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In NIPS, pages 13745–13758, 2022. 1,2,3,4,6,7,12 [13] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022. 2 [14] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4497–4506, 2021. 2 [15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–308, 2009. 2 [16] Qichen Fu, Xingyu Liu, and Kris M. Kitani. Sequential voting with relational box fields for active object detection. In Conference on Computer Vision and Pattern Recognition, pages 2374–2383, 2022. 2 [17] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, 2022. 2 [18] Georgia Gkioxari, Ross Girshick, Piotr Doll´ ar, and Kaiming He. Detecting and recognizing human-object interactions. In Conference on Computer Vision and Pattern Recognition, pages 8359–8367, 2018. 2 [19] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Q. Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Z. Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonz´ alez, James M. Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Yu Heng Khoo, J´ achym Kol´ ar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran K. Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbel´ aez, David J. Crandall, Dima Damen, Giovanni Maria Farinella, 9