Fine-grained Open-vocabulary Object Detection

Fine-grained Open-vocabulary Object Detection Lorenzo Bianchi 1,2*, Fabio Carrara 1, Nicola Messina 1, Claudio Gennaro 1, Fabrizio Falchi 1 1CNR-ISTI,Via G. Moruzzi, 1, 56124, Pisa, Italy. 2University of Pisa, Computer Engineering Department, Via Caruso, 15, 56122, Pisa, Italy. *Corresponding author(s). E-mail(s): lorenzo.bianc[email protected]; Contributing authors: [email protected];[email protected]; [email protected];fabrizio.falc[email protected]; Abstract The emergence of vision-language models like CLIP has significantly advanced open-vocabulary object detection, enabling object recognition through free-text descriptions at inference time. However, existing approaches primarily focus on class-level discrimination, often failing to capture fine-grained object attributes such as color, pattern, and material. In this paper, we introduce Fine-Grained OpenVocabulary Object Detection and propose a benchmark suite to assess the ability of models to detect, differentiate, and describe objects with fine-grained attributes, even in the presence of challenging negative captions. Our benchmark suite covers multiple difficulty levels and attribute types, providing a comprehensive evaluation of state-of-the-art open-vocabulary object detectors. Extensive experiments reveal that most detection models struggle to capture subtle object attributes effectively. In order to mitigate the critical failures of the probed models, we prepare a weakly labeled training set and introduce a distillation-based adaptation method that balances attribute-level and class-level detection. This approach improves the trade-off between fineand coarse-grained recognition, helping to bridge the gap that emerges in current state-of-the-art models. Our results highlight current limitations and suggest promising directions for improving fine-grained open-world detection. Data and code are available at https://lorebianchi98.github.io/FG-OVD/. Keywords: Open-vocabulary detection, Fine-grained understanding, Benchmark, Dataset 1 Introduction Do you remember the iconic scene in Terminator where the T-800 scans the parking lot outside the Bar Corral? His vision is augmented by detailed data on each vehicle — make, model, condition, and whether it is suitable for escape. Although entertaining in a cinematic context, this glimpse into a robot’s perspective highlights a central challenge of modern computer vision, that is, understanding the fine-grained differences between objects. Of course, this is an amusing example that captures well the essence of fine-grained object detection tasks: identifying subtle differences in object properties, such as color, texture, or material, even in the presence of similar distractors. Unlike coarse-grained detection, where recognizing broad categories (e.g., ‘car’ versus ‘motorcycle’) is sufficient, fine-grained detection requires models to go 1 deeper, discerning the complex details that could define an object. Open-vocabulary object detection (OVD) is the task of localizing and classifying objects that were not encountered during training, overcoming the limitations of traditional detectors restricted to a fixed set of classes. Recent advances in visual language models, such as CLIP (Radford et al.,2021), have spurred advances in OVD by establishing semantic connections between image regions and text labels, even when the labels describe complex properties. This flexibility has opened up applications in fields such as autonomous driving (Ma et al.,2022), extended reality (Lu et al.,2023), and robotics (Liang et al., 2023;Huang et al.,2023;Jatavallabhula et al., 2023). Given its need for flexible taxonomies, OVD allows object labels to go beyond simple categorical terms, thus enabling, in principle, the use of any natural language freeform text to populate the vocabulary. This paradigm shift allows incorporating fine-grained details such as color, pattern, and material into the labels, unveiling fine-grained recognition tasks requiring the distinction between, for example, a brown bear and a polar bear or a dark brown wooden lamp and a gray metal lamp. This task, which we call Fine-Grained Open-Vocabulary Detection (FG-OVD), pushes the boundaries of object detection by requiring models to discern complex and subtle differences among the objects. Currently, traditional open-vocabulary object detection (Lin et al.,2014;Gupta et al.,2019) benchmarks are limited to class-level labels, failing to assess models’ capabilities with more detailed textual descriptions. Despite some attempts to address fine-grained understanding in object detection (Ramanathan et al.,2023;Bravo et al., 2023), a comprehensive investigation of how effective open-ended vocabulary detectors are in discerning fine-grained properties of objects remains largely unexplored. This gap becomes critical in some emerging problems, such as episodic memory in egocentric videos, where the ability to retrieve object instances based on specific attributes is required. To address this challenge, we present a suite of benchmarks tailored for FG-OVD tasks. This suite systematically assesses the ability of a model to detect, distinguish, and describe fine-grained object attributes using eight annotated image sets, a dedicated evaluation protocol, and specific metrics. Each benchmark instance includes an image with object locations described by a detailrich caption and negative examples that test the model’s ability to identify the correct attributes. This paper extends our previous CVPR Highlight paper (Bianchi et al.,2024), in which we employed such benchmarks to analyze the strengths and weaknesses of advanced OVD methods, providing insight into their limitations in capturing fine-grained details. In this work, we further advance the research in FG-OVD by proposing a solution to mitigate the critically low performance obtained by current state-of-the-art models. Specifically, we provide a weakly labeled training set constructed by employing a similar pipeline to the one used to generate the test sets of the benchmarks and designed to fine-tune OVD models to make them more sensitive to fine-grained descriptions. Furthermore, employing the proposed FGOVD training set, we introduce and deeply study NoctOWL (Not only coarse-text Open-World Localizator), the first open-vocabulary detector designed for tackling FG-OVD by employing a simple yet clever adaptation strategy. NoctOWL is designed to capture subtle, fine-grained object attributes by adapting the classification head to be more discriminative to details while remaining resilient to coarse-grained concepts. This is obtained by adjusting the small linear classification layer on the introduced training set and incorporating a distillation loss that keeps the model able to understand coarse-grained concepts. With this model, we show that it is relatively simple to bridge the gap between fineand coarsegrained detection performance, by also mitigating the risk of overfitting on the employed training split. Evaluating this model provides deeper insight into the trade-offs models face between fine-grained specificity and broad generalization. In summary, the paper makes the following key contributions: •We release a comprehensive benchmark suite to test models’ ability to recognize fine-grained objects’ properties — such as color, material, and pattern — and to analyze the limitations of state-of-the-art detectors. 2 •We leverage an LLM to generate a weakly labeled training set specifically designed for fine-grained open-vocabulary detection. •We present NoctOWL, a distillation-based baseline model that enhances FG-OVD performance while improving the trade-off between fineand coarse-grained detection. •We evaluate pre-trained open-vocabulary detectors and our proposed model on our novel benchmark suite, showing the ability of NoctOWL to effectively tackle the FG-OVD task with respect to the probed state-of-the-art methods. We provide code for reproducing the results and checkpoints of the trained models at https: //lorebianchi98.github.io/FG-OVD/. 2 Related work Open-vocabulary detection (OVD) aims to enhance detection capabilities beyond the limited set of labeled base classes used during training. Its goal is to identify novel classes, which may include free-form text, during inference. An early effort in open-vocabulary object detection, such as Bansal et al. (2018), replaced the final classification layer with linguistic embeddings such as GloVe (Pennington et al.,2014) to represent class names. The advent of visionlinguistic models, including CLIP (Radford et al., 2021) and ALIGN (Jia et al.,2021), trained on large image-text datasets, dramatically improved the semantic alignment between visual and textual data. These advances in cross-modal alignment have been included in several open vocabulary detectors, e.g., by introducing strategies for replacing the final class features with text embeddings (Zhou et al.,2022;Minderer et al.,2022; Feng et al.,2022;Kuo et al.,2023;Minderer et al., 2023), by further refining the ROI-Align head (He et al.,2017) by exploiting pre-trained visionlanguage backbones (Gu et al.,2022;Du et al., 2022;Zhou et al.,2022;Kuo et al.,2023;Minderer et al.,2023), or by adapting CLIP directly for open vocabulary detection (Zhong et al.,2022; Minderer et al.,2022,2023). More recent studies (Li et al.,2022;Liu et al., 2023;Kamath et al.,2021;Zhang et al.,2022) have investigated the integration of open-ended vocabulary detection with tasks such as Referring Expression Comprehension (REC) (Yu et al., 2016;Mao et al.,2016) and Phrase Grounding (PG) (Mu et al.,2021;Plummer et al.,2015). Although both focus on a single possibly complex phrase, the former focuses on identifying a single object within the image. At the same time, the latter aims to identify all entities mentioned in the text. However, there are many noticeable differences between the above-mentioned tasks and FG-OVD. In fact, (i) models trained for these tasks are generally not evaluated for their ability to handle difficult distinctions, and (ii) REC and PG assume that the given sentence unambiguously refers to an object in the image, reducing the need for sophisticated discriminative skills. Although some REC and PG models, such as GroundingDino (Liu et al.,2023) and GLIP (Zhang et al.,2022;Li et al.,2022), can be used as open vocabulary detectors, they often have limitations when they need to distinguish subtle fine-grained differences. Concerning benchmarks for open-vocabulary detection, the COCO (Lin et al.,2014) and LVIS (Gupta et al.,2019) datasets are widely recognized for evaluating the localization and classification capabilities of object detectors. Initially used to evaluate closed-set detectors, the COCO dataset was later adapted for zero-shot (Bansal et al.,2018) and open-vocabulary (Gu et al.,2022; Zhong et al.,2022;Liu et al.,2023) detection. In this context, COCO includes 48 basic categories for training and 17 new categories for testing. In contrast, LVIS offers a more extensive and diverse set of object categories organized by frequency of appearance (common, frequent, and rare). Several studies (Gu et al.,2022;Zhong et al., 2022;Kuo et al.,2023;Liu et al.,2023) have used the frequent and common categories as the basis, while the rare categories have been used for testing. Although these benchmarks are useful for evaluating open-vocabulary zero-shot detection, they do not specifically measure the model’s ability to identify specific attributes of objects. OVAD (Bravo et al.,2023) and VAW (Pham et al.,2021) are the datasets closest to our objectives. Although they focus on object attributes and propose negative samples to challenge existing detectors, our proposed method differs consistently. In fact, (i) these already existing benchmarks mainly evaluate attribute detectors, which 3 Object & Parts Attributes class: bench - part: back - part: seat - part: leg - part: arm color: dark green color: brown, material: wood color: brown, material wood material: metal material: metal LLM Attribute Substitution Positive captions Negative captions ✔A dark green bench with a brown wooden back and seat and metal arms and legs. ❌A dark yellow bench with a brown wooden back and seat, supported by plastic arms and legs. … ❌A white bench with a black metal back and seat, supported by metal arms and legs. Positive Caption Negative Captions A Manual Curation FG-OVD Train set (weakly labeled) FG-OVD Benchmarks (difficultyand attribute-based) ··· FG-OVD Dataset A dark red pillow made of fabric A dark green pillow made of fabric … … A dark red pillow made of wood Sample: image + boxes + vocabulary (pos & up to 10 negs) B C A green wooden bench A green plastic bench A green rattan bench A green leather bench A green glass bench A green fabric bench ❌ OWLv2 scores DNoctOWL ✔ Ours Fig. 1:Fine-grained Open-Vocabulary Detection (FG-OVD).A. We generate fine-grained captions of objects based on their parts and corresponding attributes. We obtain object-specific vocabularies comprised of one positive caption — generated by an LLM based on a semi-structured description — and several negative captions built via attribute substitution. B. From the obtained weakly-supervised dataset, we manually curate a suite of benchmarks (FG-OVD Benchmarks)by manipulating negative sets according to their difficulty levels or the types of attributes altered — categorized as Difficultybased and Attribute-based benchmarks — to probe the ability of state-of-the-art open-vocabulary detectors to discern detailed object properties. We also compile and release a weakly-labeled train and validation set (FG-OVD Train Sets). C. We use the provided resources to get insights into the capabilities of existing detectors and their limitations. D. We train, analyze, and release NoctOWL, a model with improved performance in fine-grained domains. usually include a head dedicated to attribute inference in addition to object classes; (ii) they rely on structured annotations instead of natural language descriptions for each object, limiting the evaluation of state-of-the-art vision-language models; (iii) they do not include specially designed negative examples, limiting an in-depth analysis of the limitations of current detectors. The PACO (Ramanathan et al.,2023) dataset was a source of inspiration for our work. It is built on the basis of COCO and includes bounding box annotations enriched with structured descriptions in JSON format containing information about attributes and parts of objects. PACO also incorporates external, categorized properties such as colors, materials, and patterns. Our benchmark suite builds on PACO by generating detailed natural language descriptions from these structured annotations using large language models (LLMs). This approach aligns with the growing trend of leveraging LLMs to produce high-quality, diverse, targeted annotations (Momeni et al.,2023). 3 Methodology We propose a dedicated benchmark suite to assess how well open-vocabulary object detectors identify fine-grained object characteristics. In subsection 3.1, we formally define the problem and introduce a structured evaluation framework. Next, in subsection 3.2, we introduce the dataset specifically designed to evaluate the ability of openvocabulary detectors to distinguish fine-grained object attributes, along with a training set aimed at enhancing their performance. Finally, in subsection 3.3, we provide a comprehensive overview of 4 NoctOWL, our baseline model for FG-OVD, and its training procedure. 3.1 Evaluation Protocol Consider an image I∈RW×H×Cand a vocabulary V={cj}T j=1, comprising Ttextual descriptions of target objects. An open-vocabulary object detector ψprocesses Iand Vto generate a set of mdetections D={di}m i=1, where each detection di= (bi,si) consists of a bounding box bi∈R4 and confidence scores si∈RT. Each entry si,j in sireflects the likelihood of the detected object corresponding to the vocabulary entry cj1. For every detection, the vocabulary item with the highest confidence score is chosen as the predicted label. In evaluation scenarios, such as zero-shot or open-vocabulary detection, the vocabulary Vis predefined and consistent across the test set. Detections Dare evaluated by comparing them with the m′ground truth annotations O= {oi}m′ i=1, where each ground truth object oi= (˜ bi, yi) includes the correct bounding box ˜ bi∈R4 and its correct label yi∈ V. Performance is measured using standard object detection metrics, such as mean Average Precision (mAP). Dynamic Vocabularies for FG-OVD To evaluate FG-OVD, we propose a refined approach that uses a customized dictionary Vifor each ground truth object oi. Unlike conventional evaluation methods that rely on a fixed vocabulary shared across all objects, this approach introduces object-specific vocabularies, offering fine-grained control over the selection of both positive and negative samples for each object. Specifically, each ground truth object oiis associated with a detailed textual description cpos i, referred to as the positive caption, which provides an accurate textual representation of the visual characteristics of the object. In addition to the positive caption, the object oiis linked to a set of negative captions {cneg i,1, cneg i,2, . . . , cneg i,N }. These negative captions describe concepts or objects that are semantically different from the positive caption for the given object. For instance, taking a positive caption like ”a glass lamp emitting yellow 1Vocabulary entries are often generated by expanding class labels into descriptive captions, e.g., the label dog might become a photo of a dog. ✔ A metal handbag in grey color ❌ A leather handbag in grey color ❌ A metal handbag in blue color ❌ A wool handbag in pink color ✔A light grey stone bench ❌ A yellow stone bench ❌ A light grey wood bench ❌ A light orange leather bench Fig. 2:Examples of Dynamic Vocabularies: The image Ifeatures three objects, each associated with a vocabulary. The positive captions cpos (marked with ✓) —A light grey stone bench and A metal handbag in grey color — are juxtaposed with three negative captions cneg (indicated by ✗). The open-vocabulary detector is applied to Itwo times, once for each distinct vocabulary. light” as an example, a negative caption may be formulated as ”a glass lamp emitting blue light”. Formally, with the introduction of objectspecific vocabularies, each ground truth object is redefined as oi= (˜ bi,Vi), where ˜ bi∈R4 denotes the bounding box of the object, and Vi= {cpos i, cneg i,1, . . . , cneg i,N }represents the object-specific vocabulary, which includes the sole positive caption together with all the negative ones. Notice that in this new scenario, Viis no longer filled with prompts derived from categorical labels as in classical OVD, but with detailed textual descriptions of objects, which completes the transformation of classical labels into pure natural language. During inference, the detector is tasked with selecting the most appropriate label for each detected object from its corresponding vocabulary Vi. For a detection to be considered correct, the predicted label must match the positive caption cpos iassociated with the ground truth object. This approach, which requires the comparison between the positive caption and carefully crafted negatives for each object, challenges open-vocabulary detectors to distinguish the correct object from closely related and semantically distinct alternatives. We report an example of our ground truth arrangement in Figure 2. 5 Metrics To evaluate the predictions, we rely on two complementary metrics: the COCO mean Average Precision (mAP) and the Median Rank. Together, they provide a robust evaluation of both object localization accuracy and the ability to assign the correct caption across varying levels of detection confidence. The mAP metric offers a detailed performance analysis by measuring how well the detector aligns its predictions with ground truth annotations, considering both spatial overlap and correct label assignment. It focuses on the highest-confidence prediction for each object, providing a comprehensive overview of the detection quality. In addition to mAP, we also employ the Median Rank metric to assess the detector’s confidence in ranking the correct caption among all possible choices in the vocabulary for each object. Specifically, let d= (b,s) represent a positive prediction — i.e., which has an Intersection over Union (IoU) of at least 0.5 with a ground truth object oi2. To compute the Median Rank, we sort the confidence scores sin descending order. We then determine the rank of the score associated with the correct caption within Vi. The final metric is the median of these ranks, aggregated across all objects in the evaluation dataset. While mAP focuses exclusively on the highestscoring prediction, the Median Rank provides deeper insights into the detector’s ability to differentiate the correct label from competing options in the vocabulary. Notice that since we are prompting open-vocabulary detectors with natural language instead of standard label-based classes, we had to adjust some inherently class-based post-processing steps for these metrics to work correctly. This is the case for the class-based nonmaximum suppression, for which we release more details in Appendix C. 3.2 Dataset In this work, we introduce both novel test sets equipped with the introduced metrics and 2In cases where multiple predictions meet the IoU criterion for the same ground truth object, the one with the highest confidence score is selected. proper evaluation protocols (FG-OVD Benchmarks) and a weakly-labeled training set (FGOVD Train Sets) designed to enhance fine-grained open-vocabulary detection (FG-OVD). Evaluating open-vocabulary object detectors using the protocol described in Section 3.1 necessitates a dataset specifically designed to handle per-object vocabularies and able to emphasize the quality and diversity of negative examples for each object. This proposed suite comprises eight distinct evaluation scenarios, divided into two primary categories: Difficulty-based and Attribute-based benchmarks. Difficulty-based benchmarks focus on evaluating the detector’s performance by manipulating the complexity of the negative captions, shifting from easy to very hard negatives. Differently, Attribute-based benchmarks enable targeted assessments by isolating specific attribute types, allowing for a detailed evaluation of a detector’s ability to recognize and distinguish particular object characteristics. In addition to the test sets, we also provide a weakly labeled training dataset. Instead of reannotating images from scratch, we choose to use PACO (Ramanathan et al., 2023) as the underlying data source. PACO comes equipped with bounding boxes annotated with a structured description of the underlying object and its attributes. However, these annotations are not immediately suitable for our scenario, as we need a free-form text formulation. To create natural language descriptions of object attributes, we employ a large language model (LLM), transforming structured attribute annotations coming from PACO into rich, descriptive captions. The test sets undergo manual revision to ensure accuracy and high-quality annotations. The following sections detail the preprocessing of PACO annotations, the transformation of structured attributes into natural language descriptions, and the derivation of positive and negative captions to construct both the benchmark suite and training dataset. Refining Structured Annotations In order to obtain detailed natural language descriptions for each object, we first preprocess and clean the structured annotations from the PACO dataset. PACO covers 75 object categories, encompassing 456 object-part categories and 55 attributes across image and video datasets. 6 Table 1:Statistics of the FG-OVD Benchmarks and Train sets. For each benchmark configuration, we report the number of images (Imgs), the number of annotated objects (Objs), the objects-to-image ratio (Objs/Img), the number of positive captions, the average number of positive captions per image, negative captions per positive caption, and objects per positive caption. Name Negative Set Strategy Imgs Objs Obj/Img ✓Caps ✓/Img ✗/✓Objs/✓ FG-OVD Benchmarks Hard Random attribute subst. (×1) 1707 3545 2.1 2349 1.4 9.9 1.5 Normal Random attribute subst. (×2) 1537 2968 1.9 2034 1.3 10.0 1.5 Easy Random attribute subst. (×3) 853 1299 1.5 971 1.1 10.0 1.3 Trivial Random captions 1707 3545 2.1 2349 1.4 9.9 1.5 Color Color attribute subst. 1599 3119 2.0 2126 1.3 10.0 1.5 Material Material attribute subst. 1577 3193 2.0 2128 1.3 10.0 1.5 Pattern Pattern attribute subst. 321 467 1.5 337 1.0 7.4 1.4 Transparency Transparency attribute subst. 230 409 1.8 238 1.0 2.2 1.7 FG-OVD Train Sets Traning set Random attribute subst. (×1) 15366 18041 1.2 18041 1.2 10.0 1.0 Validation set Random attribute subst. (×1) 757 908 1.2 908 1.2 10.0 1.0 (a) Trivial ✓A brown wooden chair. ✗A red cup with a pink plastic rim ✗A pink dog with a black and dotted ear. (b) Easy ✓A brown woven rattan basket. ✗A light green perforated fabric basket. ✗A black dotted leather basket. (c) Medium ✓A knife with a black plastic handle and a dark grey metal blade. ✗A knife with a grey stone handle and a dark grey metal blade. ✗A knife with a light pink plastic handle and a light yellow metal blade. (d) Hard ✓A lamp with a white plastic shade and a grey metal pipe. ✗A lamp with a white velvet shade and a grey metal pipe. ✗A lamp with a white plastic shade and a dark pink metal pipe. (e) Color ✓A blue hat. ✗A orange hat. ✗A yellow hat. (f) Material ✓A red plastic plate. ✗A red metal plate. ✗A red ceramic plate. (g) Pattern ✓A dark pink striped pillow made of fabric. ✗A dark pink floral pillow made of fabric. ✗A dark pink dotted pillow made of fabric. (h) Transparency ✓A transparent glass. ✗A translucent glass. ✗A opaque glass. Fig. 3:Samples from FG-OVD Benchmarks. Each benchmark tests different properties by crafting negative captions via attribute substitution. Substituted attributes are underlined in the captions. To improve caption clarity, we remove generic attributes such as plain pattern and opaque transparency — common across most objects — to prevent unnatural descriptions (e.g., A plain opaque black dog). However, these attributes are retained to generate negative captions. Redundant attributes from object parts are also eliminated to streamline descriptions, with missing details merged into the main object’s attributes. For instance, complex descriptions like A car with a 7 Table 2: Different attribute types and their possible values. Type Possible Values Colors black light blue blue dark blue light brown brown dark brown light green green dark green light grey grey dark grey light orange orange dark orange light pink pink dark pink light purple purple dark purple light red red dark red white light yellow yellow dark yellow Materials text stone wood rattan fabric crochet wool leather velvet metal paper plastic glass ceramic Patterns plain striped dotted checkered woven studded perforated floral logo Transp. opaque translucent transparent black hood, black roof, black fender, and black bumper are simplified to A black car. Also, parts without attributes are discarded. Since the PACO dataset provides attributes for only some objects and lacks scene-level attribute consistency, there is the risk of generating captions that unintentionally describe multiple objects, leading to incorrect evaluations. To mitigate this, captions are initially propagated to all objects of the same class, and any mismatches are manually revised to ensure consistency and accuracy. For the attribute-based tests, we primarily adopt attribute values from the PACO dataset, encompassing four types: 29 colors,14 materials, 8 patterns, and 3 transparency modes, as better detailed in Table 2. These attributes provide a rich and varied testing ground to evaluate detectors across a wide range of visual properties. Positive Caption Generation To generate precise textual descriptions of objects, we assume that each object is characterized by at least one attribute and may contain multiple parts, each with its own attributes. Since open-vocabulary object detectors require textual inputs, we leverage this structured definition to create fine-grained descriptions that incorporate both object-level and part-level details. To achieve this, we employ a Large Language Model (LLM), specifically OpenAssistantLLAMA-30B (K¨opf et al.,2023), guided by prompt engineering and in-context learning Brown et al. (2020). Specifically, the model is presented with structured object descriptions extracted from PACO along with carefully crafted natural language examples to establish a consistent captioning style (see Figure 4). The LLM then generates new captions based on structured queries describing the attributes and relationships of unseen objects. To ensure high-quality captions, we implement an iterative prompting mechanism to refine outputs when they fail to meet predefined criteria, such as omitting key attributes or exceeding length constraints. If a generated caption does not satisfy our criteria, we issue a series of targeted follow-up queries (see Listing 1), prompting the LLM to correct specific errors. While this methodology could be applied iteratively until all captions are optimal, we found that a single refinement step was sufficient to achieve high-quality results, eliminating the need for further iterations. Captions that still failed to meet the predefined criteria after the follow-up questions were discarded from the dataset to ensure consistency and accuracy. Following generation, the captions comprising FG-OVD Benchmarks undergo a manual review to ensure consistency and accuracy, reduce annotation errors, and improve alignment with the structured attributes of the dataset. Negative Captions Generation To comprehensively evaluate open-vocabulary object detection models, we propose a diverse set of benchmarks incorporating vocabularies with challenging negative captions derived from the generated positive captions. Our goal is to create negatives that are semantically distinct while maintaining structural consistency with their corresponding positive captions. To achieve this, we employ attribute substitution, where specific attributes in the original caption are replaced while preserving the overall sentence structure. This approach ensures controlled variations while avoiding the syntactic inconsistencies or hallucinations that could arise from modifying the structured object descriptions and re-querying the LLM. 8 Query & First Generation {"object": "lamp", “parts": [{"name": "bulb", "color": ["white"], "material": ["glass"], "transparency": ["translucent"]}, {"name": "finial", "color": ["brown"], "material": ["plastic"]}]} USER "A lamp with a white glass bulb and a brown plastic finial." LLM Could you specify that the transparency of the bulb is translucent? USER "A lamp with a white glass translucent bulb and a brown plastic finial." LLM Corrective Question (Optional) I will give you a json describing an object. I want you to give me a one natural language caption which describes the object. These are four examples: 1: {"object": "scissors", "color": ["blue"], "parts": [{"name": "blade", "color": ["light_grey"], "material": ["metal"]}, {“name": "handle", "material": ["plastic"]}]} "A blue scissors with a light grey metal blade and a handle made of plastic" 2: {"object": "pillow", "color": ["blue","red"], "material": ["fabric"], "pattern": ["striped"]} "A striped blue and red pillow made of fabric" 3: {"object": "dog", “parts": [{"name": "eye", "color": ["brown"]}, {"name": "body", "color": ["black", "white"]}]} "A dog with black eyes and a black and white body" 4: {"object": "blender", "parts": [{"name": "switch", "color": ["white"], "material": ["plastic"]}, {"name": "base", "color": ["dark_red", "light_grey"], "material": ["metal"]}]} "A blender with a white plastic switch and a dark red and light grey metal base." Use only one sentence, and avoid using subjective adjectives and information not present in the json. Be verbose and pedantic, using all the attributes in the json. Are you ready? USER Yes i am ready! Give me your json file. LLM In-context examples Warm-Up Question Fig. 4:Caption Generation and Correction. We query the LLM with a JSON description outlining the object’s parts and attributes and provide four in-context samples to mitigate hallucination risks. A warm-up question separates the query from the in-context examples and prevents confusing the attributes of the examples with those of the query object. If the initial caption does not meet predefined criteria, we ask follow-up questions (outlined in Listing 1) to address identified issues. The negative captions are designed to test the model’s ability to handle variations in both attribute types and quantities, providing a robust assessment of its resilience to fine-grained distinctions. Our benchmarks are categorized into two main types: (i) Difficulty-based benchmarks (Trivial, Easy, Medium, Hard). These benchmarks assess the ability of detectors to handle progressively more challenging negative captions. In the Trivial setting, negative captions are randomly selected from other objects, while in Easy,Medium, and Hard, negative captions are generated by replacing 3, 2, and 1 attributes, respectively. As fewer attributes are changed, the distinction between captions becomes subtler, making the task increasingly difficult (see Figure 3, top row). (ii) Attribute-based benchmarks (Color, Material, Transparency, Pattern). These benchmarks focus on a detector’s ability to distinguish specific attributes. Negative captions are generated by modifying only one attribute per caption, depending on the benchmark category, allowing for a targeted evaluation of the model’s sensitivity to each attribute type (see Figure 3, bottom row). The benchmark statistics are detailed in Table 1, providing insights into the distribution and complexity of the generated datasets. FG-OVD Train Sets Following the same procedure for the positive caption generation, we generated synthetic training and validation datasets from the PACO-LVIS training and validation sets, respectively. Differently from the FG-OVD Benchmark generation procedure, we did not perform a manual check on these sets due to the high number of samples and the more relaxed quality guarantees. However, to avoid injecting annotation errors, we applied a more conservative policy: the attributes assigned to one object are not propagated to another with unknown attributes in the same image. Therefore, for these data splits we restricted caption generation to instances with a single object of a specific category. This precaution prevents captions intended for one object from inadvertently describing another that lacks ground truth annotations, therefore limiting the presence of wrongly annotated samples. We opted for generating datasets that adhere to the Hard standard, thus including negative captions that vary from the positive ones by a single attribute only. Additionally, we limited our selection to objects that enabled us to generate 10 negative captions. These parameters were chosen to train models under the most challenging 9 0.15 0.20 0.25 0.30 0.35 0.40 0.10 0.15 LVIS-Rare mAP OWL (B/16) 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.15 0.20 0.25 OWLv2 (B/16) 0.15 0.20 0.25 0.30 0.35 0.40 0.45 FGOVD mAP 0.15 0.20 0.25 0.30 LVIS-Rare mAP OWL (L/14) 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 FGOVD mAP 0.25 0.30 0.35 OWLv2 (L/14) Model Pre-trained Fine-tune Ours Fig. 7:Effect of Coarse-grained Distillation loss term during training. Xand Y-axis report mAP on FG-OVD and LVIS-Rare, representing respectively, fineand coarse-grained performance (upperright is better). Lines represent the evolution of performance during training with and without the coarse-grained loss term. The blue dot indicates the initial performance before training. Table 5:Results (mAP) segmented by object sizes (S=Small, M=Medium, and L=Large) on the FG-OVD Benchmarks. Difficulty-based sets (Trivial, Easy, Medium, and Hard) use N= 5 negative captions, while Attribute-based ones (Color, Material, Transparency, and Pattern sets) use N= 2. Difficulty-based (N= 5) Attribute-based (N= 2) Trivial Easy Medium Hard Color Material Pattern Transparency S M L S M L S M L S M L S M L S M L S M L S M L OWL (B/16) 15.0 44.6 60.1 11.6 29.4 44.3 12.5 33.0 43.1 11.0 21.7 28.3 15.6 39.1 48.2 11.2 29.7 42.5 2.7 24.8 28.3 17.0 31.3 37.5 OWL (L/14) 33.0 56.0 67.9 19.3 39.3 42.9 18.3 34.4 39.9 13.8 22.7 27.0 24.1 37.1 45.7 22.0 39.0 47.0 11.2 34.6 35.4 20.1 26.2 30.7 OWLv2 (B/16) 26.3 48.0 54.6 15.9 33.4 44.0 19.5 34.2 38.8 14.8 21.9 26.4 24.5 42.1 44.6 14.9 27.1 37.9 18.0 19.8 20.5 20.2 26.6 29.7 OWLv2 (L/14) 32.0 55.8 65.8 13.8 34.3 48.6 24.2 34.6 43.9 14.0 21.9 26.2 32.2 47.1 53.9 17.1 30.9 40.6 11.2 24.4 22.8 3.9 8.5 15.8 Detic 39.4 71.1 75.0 19.9 19.3 19.1 11.6 20.2 18.4 10.5 11.5 12.2 16.1 23.2 21.1 20.1 39.4 42.2 27.6 33.5 30.5 0.0 28.9 40.0 ViLD 32.5 63.2 61.8 20.8 43.0 44.8 21.6 40.0 38.5 15.0 24.0 23.2 26.3 49.6 44.7 16.7 37.6 39.9 13.0 28.9 24.0 13.9 30.9 34.0 GDino 19.1 58.5 72.4 7.7 30.4 31.9 6.6 28.2 29.8 5.1 17.0 17.0 12.1 39.4 45.9 7.0 28.0 34.4 7.6 26.9 34.7 19.7 22.8 27.6 CORA 12.0 32.9 46.4 6.0 18.5 26.5 7.6 19.1 25.7 4.2 13.2 17.3 9.4 24.8 32.5 7.7 17.0 26.8 4.2 21.9 28.2 10.8 25.8 34.9 NoctOWL (B/16) 16.3 36.8 49.8 15.9 42.7 55.6 13.8 39.2 57.7 14.6 42.5 57.8 17.7 41.2 55.9 15.9 42.9 56.5 21.5 47.7 62.6 11.5 42.6 53.2 NoctOWL (L/14) 25.7 46.8 56.2 28.7 53.1 63.2 30.6 51.5 63.5 30.5 55.0 66.6 29.9 50.6 61.9 29.6 54.8 65.9 35.7 54.9 60.9 17.9 46.7 55.0 NoctOWLv2 (B/16) 24.4 41.9 48.0 24.5 46.4 54.6 23.4 47.2 59.2 24.6 47.9 55.1 27.0 46.5 53.9 24.6 48.2 55.0 30.7 48.2 57.3 19.8 40.2 46.6 NoctOWLv2 (L/14) 28.8 49.8 56.8 30.1 55.8 66.2 24.5 53.9 66.2 29.4 56.0 66.3 29.3 54.1 64.5 30.9 55.9 65.3 22.3 51.4 63.6 11.7 46.6 53.4 Choice of Fine-Grained Loss Function We compare the impact of training NoctOWL with a cross-entropy (CE) loss versus the hingebased triplet (Tri) loss. As shown in Table 6, the triplet loss generally demonstrates slightly better mAP on FG-OVD benchmarks, suggesting its enhanced capability in capturing fine-grained distinctions. On the LVIS benchmarks, the triplet loss outperforms the CE loss across all configurations, demonstrating its strength in preserving coarsegrained semantic information. These results confirm that triplet loss is a better choice for balancing fine-grained accuracy with coarse-grained generalization. 5 Conclusions In this paper, we introduced a comprehensive evaluation protocol and benchmark suite for FineGrained Open Vocabulary Detection (FG-OVD), assessing the fine-grained discriminative power of open-vocabulary detectors. Our protocol challenges models with rich, attribute-based captions 16 short (0-6) medium (7-10) long (11-14) longer (15+) 0.0 0.5 1.0 mAP Trivial short (0-6) medium (7-10) long (11-14) longer (15+) 0.0 0.5 Easy short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 Medium short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 Hard short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 mAP Color short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 Material short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 Pattern short (0-6) medium (7-10) long (11-14) longer (15+) 0.00 0.25 0.50 Transparency OWL (L/14) OWLv2 (L/14) Detic ViLD GDino CORA NoctOWL (L/14) NoctOWLv2 (L/14) Fig. 8:Effect of the caption length: We illustrate the mAP of detectors across Difficulty-based (N = 5) and Attribute-based (N=2) benchmarks, with varying caption lengths for objects. Captions are categorized into four groups based on their average word count inside the corresponding vocabulary: short (6 or fewer words), medium (7-10 words), long (11-14 words), and longer (15 or more words). OWLbased detectors are excluded from the longer group due to their inability to process captions exceeding 16 tokens. Table 6:Effect of the choice of the fine-grained loss term LCG on LVIS rare and Difficulty-based (N= 5) and Attribute-based (N= 2) FG-OVD Benchmarks. FG-OVD Benchmarks LVIS Difficulty-based Attribute-based Detector Loss Rare Triv. Easy Med. Hard Color Material Pattern Transp. OWL B/16 CE 10.4 49.8 44.3 45.3 44.2 44.2 45.3 44.3 49.8 OWL B/16 Tri 11.6 53.6 46.1 46.0 44.7 44.7 46.0 46.1 53.6 OWL L/14 CE 24.9 56.6 51.3 56.4 52.0 52.0 56.4 51.3 56.6 OWL L/14 Tri 26.0 57.2 49.8 56.9 53.1 53.1 56.9 49.8 57.2 OWLv2 B/16 CE 14.9 48.3 42.8 47.0 45.8 45.8 47.0 42.8 48.3 OWLv2 B/16 Tri 17.5 50.2 42.2 48.2 46.8 46.8 48.2 42.2 50.2 OWLv2 L/14 CE 25.3 54.5 50.2 55.1 54.0 54.0 55.1 50.2 54.5 OWLv2 L/14 Tri 27.2 55.9 49.2 57.0 55.6 55.6 57.0 49.2 55.9 and provides meaningful metrics to evaluate their performance. To create these benchmarks, we leveraged structured object descriptions and used an LLM to generate diverse, high-quality captions. By systematically modifying attributes, we established a range of difficulty levels, enabling a thorough analysis of model weaknesses. Our experiments revealed a significant gap in detectors’ ability to capture fine-grained object properties, with even the latest models struggling in complex scenarios. We also introduce a weaklylabeled FG-OVD training set and release NoctOWL, an OWL-based model specifically devised for FG-OVD. Our results demonstrate that adapting detectors on the fine-grained domain significantly improves detection performance, particularly in hard-negative settings, while distillation techniques help mitigate catastrophic forgetting of coarse-grained concepts. In the future, we aim to develop improved pretraining strategies for vision-language backbones in open-vocabulary object detection models. We 17 envision more balanced image-text representations that seamlessly integrate both fineand coarse-grained features without relying on a specially curated dataset tailored to a fixed set of attribute types. Acknowledgments This work was partially supported by the project SUN – Social and hUman ceNtered XR supported by Horizon Europe Research & Innovation Programme (GA 101092612), and by the European Union — Next Generation EU (FAIR PE00000013, ITSERR B53C22001770006, PNRR PRIN MUCES P2022BW7CW - CUP: B53D23026090001, and EKEEL P20227PEPK). Appendix A Model Architectural Details In our experiments, we evaluated the following architectural configurations for each detector: •OWL-based models (OWL-ViT and OWLv2): We tested configurations using ViT B/16 and ViT L/14 as backbones. •ViLD: The model was configured with a ResNet-152 backbone and a distillation weight of 0.1. •Detic: We used the Swin-B backbone with ImageNet-21K pretraining, specifically the Detic LCOCOI21k CLIP SwinB 896b32 4x ft4x max-size configuration. •GroundingDINO: The GroundingDINO-T variant was used, featuring a Swin-T backbone. •CORA: The model was evaluated with a ResNet-50x4 backbone. Since the captions in our benchmarks are written in natural language, all models were tested without any additional pre-appended prompts. The only exception was CORA, which inherently utilizes an internal prompt ensemble of 80 different prompts. To assess the impact of this ensemble, we also evaluated CORA with the prompt ensemble disabled, processing input captions without modifications. As shown in Figure A1, the results indicate that performance remains largely unchanged, suggesting that the prompt ensemble does not significantly affect the model’s overall effectiveness. OWL Subset Since OWL-based detectors can only process sentences with a maximum length of 16 words, we reevaluated all detectors on a filtered version of the benchmarks, where captions exceeding this limit were removed. The updated benchmark statistics reflecting this constraint are provided in Table A1. The corresponding results for all models on this subset are reported in Table A2,Table A3, and Figure A2. Notably, the removal of longer captions had a minimal impact on overall performance, with each detector maintaining the same general trends observed in the full benchmark. This suggests that key information is typically conveyed early in the captions, making the omitted portion of longer sentences largely redundant. Appendix B Handling Grounding Dino GroundingDino differs from other detectors in its design, as it is primarily intended for referring expression comprehension (REC) and does not support full caption vocabularies as input. Unlike other models, which process an entire set of captions at once, GroundingDino requires a single caption per inference. While its authors suggest that certain prompts can automatically segment textual expressions (e.g., using the ”.” character), we observed inconsistencies: single-word labels are handled correctly, but complex sentences are sometimes split incorrectly. To adapt GroundingDino for open-vocabulary detection, , we perform inference for each caption in the vocabulary individually. Each prediction is represented as di= (bi, hi, ti), where birefers to the bounding box coordinates, hirepresents the score assigned to the caption ti, and tiis the caption itself. This approach differs from other open-vocabulary object detectors, where each prediction includes a score array si, with each element reflecting the score for its corresponding caption. However, this distinction does not influence the mAP of the detector, as it only takes into account the predicted label for each bounding box. On the other hand, this difference does affect the rank metric, since the score array sifor the vocabulary entries is required for ranking. 18 0.0 0.2 0.4 0.6 0.8 mAP Hard Medium Easy Trivial 012345678910 0.0 0.2 0.4 0.6 0.8 mAP Color 012345678910 Material 012345678910 Pattern 012345678910 Transparency Number of negative captions N Detector CORA CORA w/o prompt ens. Fig. A1:Effect of the number of the prompt ensemble on CORA Table A1:Benchmark based on OWL-compatible captions: Statistics of the benchmarks based on OWL-subset benchmark for each different negative set comprising the number of images (Imgs), the number of annotated objects (Objs), objects-to-image ratio (Objs/Img), positive captions, positive captions per image, negative captions per positive caption, and objects per positive caption. Name Negative Set Strategy Imgs Objs Obj/Img ✓Caps ✓/Img ✗/✓Objs/✓ Hard Random attribute subst. (×1) 1390 2903 2.1 1816 1.3 9.9 1.6 Normal Random attribute subst. (×2) 1187 2293 1.9 1483 1.2 10.0 1.5 Easy Random attribute subst. (×3) 417 657 1.6 445 1.1 10.0 1.5 Trivial Random captions 1389 2888 2.1 1810 1.3 10.0 1.6 Color Color attribute subst. 1269 2485 2.0 1595 1.3 10.0 1.6 Material Material attribute subst. 1277 2611 2.0 1639 1.3 10.0 1.6 Transparency Transparency attribute subst. 177 323 1.8 180 1.0 2.0 1.8 Pattern Pattern attribute subst. 188 294 1.6 193 1.0 7.2 1.5 For rank computation, where we rank all possible vocabulary items for each object, we considered all predictions djgenerated for the ground truth object oi, without applying the typical classagnostic NMS used in our evaluation. Instead, we nullified the confidence scores of predictions that did not overlap with the ground truth, i.e., hj←0 if IoU(oi, dj)<0.5. The ranking score siis determined by selecting, for each vocabulary element of oi, the prediction score corresponding to the caption with the highest confidence. The median rank is then calculated using the same procedure applied to the other detectors. Appendix C Post-processing Adjustments Since we now rely exclusively on natural language descriptions instead of fixed categorical labels and we are dealing with a per-object vocabulary, some standard inference-time operations used in classical object detection require careful reconsideration. The most critical post-processing operation is class-aware non-maximum suppression (NMS), which is traditionally employed to remove redundant predictions of the same class at the same location while allowing predictions from different classes to coexist. However, in our scenario where all vocabulary entries are mutually exclusive, and there is no longer a valid notion of overlap among labels, class-aware NMS would basically perform no filtering at all. This has an important effect on COCO mAP, which does not penalize predictions from incorrect classes. In our scenario, this means that if an incorrect prediction with higher confidence overlaps with the correct one, it does not result in any penalty. In other words, even a single 19 Table A2:Benchmark based on OWLcompatible captions: mAP on Difficulty-based benchmarks (N= 5). Backbone Trivial Easy Med. Hard OWL B/16 53.9 38.4 39.8 26.2 OWL L/14 65.1 44.0 39.3 26.5 OWLv2 B/16 52.9 40.0 38.5 25.3 OWLv2 L/14 63.2 42.8 41.2 25.4 NoctOWL B/16 46.6 44.4 45.6 40.0 NoctOWL L/14 57.4 54.2 54.8 48.6 NoctOWLv2 B/16 48.3 49.1 47.1 42.1 NoctOWLv2 L/14 57.5 55.5 57.2 50.2 Detic 68.1 (-1.6) 22.3 (+3.7) 20.9 (+2.3) 12.3 (+0.8) ViLD 54.8 (-1.8) 44.0 (+4.1) 38.2 (+2.1) 22.8 (+0.7) GDino 62.6 (-0.1) 35.1 (+5.0) 32.0 (+4.1) 18.7 (+2.1) CORA 33.3 (-1.8) 23.2 (+2.8) 21.8 (+1.8) 13.4 (-0.4) Table A3:Benchmark based on OWLcompatible captions: mAP on Attribute-based benchmarks (N= 2). Backbone Color Mat. Pat. Transp. OWL B/16 45.3 37.3 26.6 34.1 OWL L/14 43.8 44.9 36.0 29.2 OWLv2 B/16 45.1 33.5 19.2 28.5 OWLv2 L/14 53.3 36.9 23.3 12.2 NoctOWL B/16 44.7 46.0 46.1 53.6 NoctOWL L/14 53.1 56.9 49.8 57.2 NoctOWLv2 B/16 46.8 48.2 42.2 50.2 NoctOWLv2 L/14 55.6 57.0 49.2 55.9 Detic 23.2 (+1.7) 38.8 31.1 (+1.0) 21.8 (-2.8) ViLD 43.9 (+0.7) 34.7 (-0.2) 25.6 (+1.1) 27.6 (-2.5) GDino 43.2 (+2.2) 30.9 (+0.7) 31.1 (-0.1) 26.9 (+1.5) CORA 24.3 (-0.7) 17.3 (-2.0) 16.9 (-5.1) 28.4 (+0.5) low-confidence correct prediction, from the mAP perspective, masks all the more significant errors. To avoid this problem, we instead employ the class-agnostic NMS procedure. In this way, a higher-confidence incorrect prediction will suppress a lower-confidence correct one, regardless of the class label, avoiding the masking problem arising in the class-aware NMS. This adjustment forces metrics like mAP to reflect errors appropriately, aligning the evaluation process with the requirements of our framework. Appendix D Additional Benchmark Samples We show additional samples from our benchmarks in Figures D3 (Color),D4 (Material),D5 (Pattern),D6 (Transparency),D7 (Trivial),D8 (Easy), D9 (Medium), and D10 (Hard). References Bianchi, L., Carrara, F., Messina, N., Gennaro, C., Falchi, F.: The devil is in the finegrained details: Evaluating open-vocabulary object detectors for fine-grained understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22520–22529 (2024) Bianchi, L., Carrara, F., Messina, N., Falchi, F.: Is clip the main roadblock for finegrained open-world perception? arXiv preprint arXiv:2404.03539 (2024) Bravo, M.A., Mittal, S., Ging, S., Brox, T.: Openvocabulary attribute detection. In: CVPR, pp. 7041–7050 (2023) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) Bansal, A., Sikka, K., Sharma, G., Chellappa, R., Divakaran, A.: Zero-shot object detection. In: ECCV, pp. 384–400 (2018) Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., Li, G.: Learning to prompt for open-vocabulary object detection with vision-language model. In: 20 0.0 0.2 0.4 0.6 0.8 mAP Hard Medium Easy Trivial 012345678910 0.0 0.2 0.4 0.6 0.8 mAP Color 012345678910 Material 012345678910 Pattern 012345678910 Transparency Number of negative captions N Detector OWL OWLv2 Detic ViLD GDino CORA NoctOWL NoctOWLv2 1 2 3 4 5 6 7 8 9 Rank Hard Medium Easy Trivial 12345678910 1 2 3 4 5 6 7 8 9 Rank Color 12345678910 Material 12345678910 Pattern 12345678910 Transparency Number of negative captions N Detector OWL OWLv2 Detic ViLD GDino CORA NoctOWL NoctOWLv2 Fig. A2:Benchmark based on OWL-compatible captions: Effect of the number of negative captions. CVPR, pp. 14084–14093 (2022) Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., Ma, L.: Promptdet: Towards openvocabulary detection using uncurated images. In: ECCV, pp. 701–717 (2022). Springer Gupta, A., Dollar, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. In: CVPR, pp. 5356–5364 (2019) Gu, X., Lin, T., Kuo, W., Cui, Y.: Openvocabulary object detection via vision and language knowledge distillation. In: ICLR (2022). https://openreview.net/forum?id= lL3lnMbR4WU He, K., Gkioxari, G., Doll´ar, P., Girshick, R.: Mask R-CNN. In: ICCV, pp. 2961–2969 (2017) Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., Fei-Fei, L.: VoxPoser: Composable 3d value maps for robotic manipulation with language models. In: Tan, J., Toussaint, M., Darvish, K. (eds.) Proceedings of The 7th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 229, pp. 540–562 (2023). https://proceedings.mlr.press/ v229/huang23b.html Jatavallabhula, K.M., Kuwajerwala, A., Gu, Q., Omama, M., Iyer, G., Saryazdi, S., Chen, T., Maalouf, A., Li, S., Keetha, N.V., Tewari, A., Tenenbaum, J.B., Melo, C.M., Krishna, K.M., Paull, L., Shkurti, F., Torralba, A.: Conceptfusion: Open-set multimodal 3d mapping. In: Bekris, K.E., Hauser, K., Herbert, S.L., Yu, J. (eds.) Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023 (2023). https://doi.org/10.15607/ RSS.2023.XIX.066 .https://doi.org/10.15607/ RSS.2023.XIX.066 21 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 Fig. D3: More samples from the Color benchmark. Legend: OWL (L/14) , OWLv2 (L/14) , Detic , ViLD , GDino , Cora NoctOWL (L/14) , NoctOWLv2 (L/14) , Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML, pp. 4904–4916 (2021). PMLR Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., Angelova, A.: F-VLM: Open-vocabulary object detection upon frozen vision and language models. In: ICLR (2023) K¨opf, A., Kilcher, Y., R¨utte, D., Anagnostidis, S., 22 Tam, Z.R., Stevens, K., Barhoum, A., Nguyen, D., Stanley, O., Nagyfi, R., et al.: OpenAssistant Conversations - democratizing large language model alignment. In: NeurIPS (2023) Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: MDETR-modulated detection for end-to-end multi-modal understanding. In: ICCV, pp. 1780–1790 (2021) Li, Z., Hoiem, D.: Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40(12), 2935–2947 (2017) Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., Zeng, A.: Code as policies: Language model programs for embodied control. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493–9500 (2023). IEEE Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV, pp. 740–755 (2014). Springer Lu, Y., Xu, C., Wei, X., Xie, X., Tomizuka, M., Keutzer, K., Zhang, S.: Open-vocabulary pointcloud object detection without 3d annotation. In: CVPR, pp. 1190–1199 (2023) Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding DIN: Marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023) Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al.: Grounded languageimage pre-training. In: CVPR, pp. 10965–10975 (2022) Momeni, L., Caron, M., Nagrani, A., Zisserman, A., Schmid, C.: Verbs in action: Improving verb understanding in video-language models. In: ICCV, pp. 15579–15591 (2023) Minderer, M., Gritsenko, A., Houlsby, N.: Scaling open-vocabulary object detection. In: NeurIPS (2023) Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al.: Simple open-vocabulary object detection. In: ECCV, pp. 728–755 (2022). Springer Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: CVPR, pp. 11–20 (2016) Mu, Z., Tang, S., Tan, J., Yu, Q., Zhuang, Y.: Disentangled motif-aware graph learning for phrase grounding. In: AAAI, vol. 35, pp. 13587–13594 (2021) Ma, Z., Yang, Y., Wang, G., Xu, X., Shen, H.T., Zhang, M.: Rethinking open-world object detection in autonomous driving scenarios. In: ACM MM, pp. 1279–1288 (2022) Pham, K., Kafle, K., Lin, Z., Ding, Z., Cohen, S., Tran, Q., Shrivastava, A.: Learning to predict visual attributes in the wild. In: CVPR, pp. 13018–13028 (2021) Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014) Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV, pp. 2641–2649 (2015) Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML, pp. 8748–8763 (2021). PMLR Ramanathan, V., Kalia, A., Petrovic, V., Wen, Y., Zheng, B., Guo, B., Wang, R., Marquez, A., Kovvuri, R., Kadian, A., Mousavi, A., Song, Y., Dubey, A., Mahajan, D.: PACO: parts and attributes of common objects. In: CVPR, pp. 7141–7151 (2023) 23 Wu, X., Zhu, F., Zhao, R., Li, H.: CORA: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In: CVPR, pp. 7031–7040 (2023) Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV, pp. 69–85 (2016). Springer Zhou, X., Girdhar, R., Joulin, A., Kr¨ahenb¨uhl, P., Misra, I.: Detecting twenty-thousand classes using image-level supervision. In: ECCV, pp. 350–368 (2022). Springer Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022) Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: RegionCLIP: Region-based language-image pretraining. In: CVPR, pp. 16793–16803 (2022) Zhang, H., Zhang, P., Hu, X., Chen, Y.-C., Li, L., Dai, X., Wang, L., Yuan, L., Hwang, J.- N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. In: NeurIPS, pp. 36067–36080 (2022) 24 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 0.0 0.5 A black plastic helmet A dark purple plastic helmet A light red plastic helmet × OWL (L/14) 0.0 0.1 × OWLv2 (L/14) 0.0 0.5 × Detic 0.00 0.25 × ViLD 0.0 0.5 GDino 0.00 0.05 CORA 0.0 0.5 NoctOWL (L/14) 0.00 0.25 NoctOWLv2 (L/14) 0.0 0.5 A black ceramic mug A dark orange ceramic mug A green ceramic mug × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.5 0.0 0.5 0.0 0.5 A black ceramic bowl A light orange ceramic bowl A grey ceramic bowl 0.00 0.25 0 1 × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.0 0.5 0.00 0.25 A light brown paper bag A orange paper bag A light purple paper bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.2 0.0 0.1 0.0 0.5 0.0 0.5 A white towel made of fabric A white towel made of paper A white towel made of crochet × 0.0 0.1 0.0 0.5 × 0.00 0.25 0.0 0.5 0.0 0.1 0.00 0.25 0.0 0.5 0.00 0.25 A white ceramic plate A white fabric plate A white paper plate × 0.0 0.2 × 0.0 0.5 0.00 0.25 × 0.0 0.5 × 0.0 0.1 × 0.0 0.5 0.0 0.5 0.00 0.25 A light brown paper bag A light brown fabric bag A light brown velvet bag 0.0 0.5 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.2 0.0 0.1 0.0 0.5 0.00 0.25 A white ceramic bowl A white glass bowl A white plastic bowl × 0.0 0.1 0.0 0.5 0.00 0.25 0.00 0.25 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A brown wool hat with a woven pattern A brown wool hat with a dotted pattern A brown wool hat with a plain pattern × 0.00 0.25 × 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.2 0.00 0.25 A grey floral patterned pillow made of fabric A grey woven patterned pillow made of fabric A grey studded patterned pillow made of fabric × 0.0 0.1 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.0 0.1 × 0.00 0.05 × 0.0 0.2 × 0.00 0.25 A dark blue ceramic vase with an orange and light blue floral pattern A dark blue ceramic vase with an orange and light blue checkered pattern A dark blue ceramic vase with an orange and light blue perforated pattern × 0.0 0.5 0 1 0.00 0.25 0.0 0.5 0.00 0.25 0.00 0.05 × 0.00 0.05 0.00 0.25 A blue plastic studded rimmed bowl A blue plastic striped rimmed bowl A blue plastic dotted rimmed bowl × 0.0 0.5 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.0 0.5 × 0.00 0.25 A transparent glass lamp with a light yellow bulb A translucent glass lamp with a light yellow bulb A opaque glass lamp with a light yellow bulb × 0.00 0.05 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 0.0 0.1 × 0.00 0.05 0.0 0.5 A transparent glass vase A opaque glass vase A translucent glass vase 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 0.0 0.5 0.00 0.25 A orange plastic glass with a translucent body A orange plastic glass with a opaque body A orange plastic glass with a transparent body × 0.0 0.2 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.0 0.1 × 0.0 0.1 × 0.0 0.2 × 0.00 0.25 A light grey plastic transparent soap dispenser A light grey plastic opaque soap dispenser A light grey plastic translucent soap dispenser × 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 × 0.0 0.5 0.00 0.05 × 0.00 0.25 0.000 0.025 A brown dog with black eyes A light brown metal handle knife with no blade A dark orange broom made of plastic A blue pan for cooking A dark pink slipper made of wool A laptop computer, the back is made of black wool 0.00 0.05 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.25 0.000 0.025 0.00 0.05 0.00 0.25 A black shoe with a white outsole A car with a grey metal roof and a transparent metal windshield A grey metal bicycle with green wheels A dark brown and white wallet with a striped white flap A handbag made of leather A dark grey glass wrench with a black handle 0.00 0.05 0.0 0.2 0.00 0.25 0.0 0.5 0.00 0.05 × 0.0 0.1 0.0 0.1 0.00 0.25 A handbag in black, orange and brown colors A ladder with a green metal rail A light grey tissue text made of paper A black television set made of leather, with a grey plastic bottom A hat made of fabric in yellow, blue and light grey colors A dark blue wooden crate with brown side panels 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.00 0.05 × 0.00 0.05 0.00 0.05 0.00 0.05 A purple metal framed mirror A car with a white exterior A grey and grey handbag made of fabric A guitar with a light brown text body and a metal fingerboard A dark grey pipe with a orange plastic nozzle stem A brown ceramic bowl with a body made of ceramic material 0.000 0.005 × 0.0 0.5 0.00 0.25 0.00 0.25 0.00 0.02 0.00 0.01 0.000 0.005 × 0.0 0.1 A light yellow plastic cup with a text pattern A black velvet cup with a perforated pattern A blue crochet cup with a checkered pattern A dark pink fabric cup with a checkered pattern A dark brown leather cup with a dotted pattern A light pink velvet cup with a woven pattern 0.0 0.1 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.05 0.0 0.1 0.0 0.1 A wood table with a brown top and grey metal legs A wood table with a green top and dark brown wool legs A crochet table with a brown top and light blue plastic legs A velvet table with a brown top and purple plastic legs A ceramic table with a brown top and blue crochet legs A wood table with a blue top and dark red paper legs 0.0 0.2 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.0 0.1 0.0 0.1 0.0 0.1 0.0 0.1 A black remote control with grey buttons and a plastic back A dark yellow remote control with dark purple buttons and a glass back A light brown remote control with dark blue buttons and a velvet back A yellow remote control with light blue buttons and a wood back A orange remote control with light green buttons and a text back A dark red remote control with light brown buttons and a wool back 0.0 0.5 × 0 1 × 0.00 0.25 0.0 0.5 0.0 0.1 0.0 0.1 0.00 0.25 0.0 0.1 A perforated bench with a brown wooden seat A studded bench with a red crochet seat A floral bench with a light pink plastic seat A dotted bench with a light red stone seat A plain bench with a dark yellow fabric seat A checkered bench with a light orange stone seat 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.000 0.025 0.0 0.1 0.0 0.5 A book with a light grey cover made of paper A book with a dark pink cover made of rattan A book with a dark green cover made of glass A book with a dark green cover made of metal A book with a light red cover made of metal A book with a dark red cover made of glass 0.0 0.2 × 0.0 0.5 0.00 0.25 0.0 0.5 0.0 0.5 0.00 0.25 0.00 0.25 0.0 0.5 A light grey metal bench A red wood bench A dark purple velvet bench A dark red rattan bench A black wood bench A purple fabric bench × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.0 0.1 × 0.00 0.25 × 0.0 0.5 A knife with a grey metal blade A knife with a light green wool blade A knife with a dark pink velvet blade A knife with a dark purple paper blade A knife with a dark orange wool blade A knife with a light red plastic blade × 0.000 0.025 0.0 0.5 × 0.00 0.25 × 0.0 0.5 0.00 0.05 × 0.0 0.1 0.00 0.25 0.0 0.5 A mirror with a brown wooden frame A mirror with a orange rattan frame A mirror with a light grey fabric frame A mirror with a blue leather frame A mirror with a black metal frame A mirror with a light yellow glass frame 0.00 0.25 × 0.0 0.5 × 0.00 0.25 × 0.0 0.5 × 0.000 0.025 × 0.0 0.1 0.0 0.1 0.00 0.05 A light grey bicycle with metal handlebars and black wheels A light grey bicycle with wood handlebars and black wheels A light grey bicycle with leather handlebars and black wheels A light orange bicycle with metal handlebars and black wheels A light grey bicycle with stone handlebars and black wheels A dark yellow bicycle with metal handlebars and black wheels 0.00 0.25 0.0 0.5 × 0.00 0.25 × 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.0 0.2 0.0 0.1 A remote control with a white plastic back A remote control with a white paper back A remote control with a light purple plastic back A remote control with a dark blue plastic back A remote control with a white rattan back A remote control with a white leather back × 0.00 0.25 0.0 0.5 × 0.00 0.25 0.00 0.25 × 0.00 0.25 × 0.00 0.05 0.00 0.25 0.00 0.25 A grey trash can made of plastic A grey trash can made of paper A red trash can made of plastic A dark purple trash can made of plastic A grey trash can made of fabric A grey trash can made of leather 0.0 0.2 × 0.0 0.5 × 0.00 0.25 × 0.00 0.25 0.00 0.25 × 0.0 0.1 0.00 0.25 0.00 0.25 A black laptop with a blue glass screen A black laptop with a blue wool screen A black laptop with a blue text screen A light pink laptop with a blue glass screen A black laptop with a blue fabric screen A black laptop with a blue metal screen × 0.00 0.25 × 0.0 0.5 × 0.00 0.25 0.00 0.05 × 0.00 0.25 0.00 0.05 0.0 0.2 Fig. D4: More samples from the Material benchmark. Legend: OWL (L/14) , OWLv2 (L/14) , Detic , ViLD , GDino , Cora NoctOWL (L/14) , NoctOWLv2 (L/14) , 25