Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach

Draft Version Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach Yoav Evron1, Michal Bar-Asher Siegal2, Michael Fire1 Abstract The recent Artificial Intelligence (AI) revolution has opened transformative possibilities for the humanities, particularly in unlocking the visual-artistic content embedded in historical illuminated manuscripts. While digital archives now offer unprecedented access to these materials, the ability to systematically locate, extract, and analyze illustrations at scale remains a major challenge. We present a general and scalable AI-based pipeline for large-scale visual analysis of illuminated manuscripts. The framework integrates modern deep-learning models for page-level illustration detection, illustration extraction, and multimodal description, enabling scholars to search, cluster, and study visual materials and artistic trends across entire corpora. We demonstrate the applicability of this approach on large heterogeneous collections, including the Vatican Library and richly illuminated manuscripts such as the Bible of Borso d’Este. The system reveals meaningful visual patterns and cross-manuscript relationships by embedding illustrations into a shared representation space and analyzing their similarity structure (see figure ??). By harnessing recent advances in computer vision and vision-language models, our framework enables new forms of large-scale visual scholarship in historical studies, art history, and cultural heritage making it possible to explore iconography, stylistic trends, and cultural connections in ways that were previously impractical. Keywords Historical Document Analysis — Deep Learning — Computer Vision — Artificial Intelligence — Illustrations — Manuscripts 1Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Be’er Sheva, Israel 2The Goldstein-Goren Department of Jewish Thought, Ben-Gurion University of the Negev, Be’er Sheva, Israel 1. Introduction Manuscripts from Late Antiquity, the Middle Ages, and the Early Modern period serve as a unique window into human society’s cultural, intellectual, and social worlds during pivotal eras of its history [ 1 ]. These manuscripts are not merely physical objects composed of ink and parchment or paper; they are vibrant reflections of their time’s knowledge, beliefs, religions, politics, and arts. Through these texts, we can explore the processes of idea creation, dissemination, and reception, often revealing the individuals behind them. Manuscripts illuminate patterns of life and the challenges of past societies, providing a foundation for understanding the origins of modern culture [2,3]. Illustrations within manuscripts are more than just decorative elements; they are a vital component of the text that enhances its meaning, accessibility, and cultural value [ 4 , 5 ]. In illuminated manuscripts, illustrations often serve liturgical, pedagogical, or commemorative purposes, reinforcing the sacred or intellectual nature of the text [6]. In many cases, illustrations serve as visual storytelling tools, bridging the gap between literacy and comprehension for audiences of varying educational backgrounds. They provide context to complex narratives, clarify textual ambiguities, and offer symbolic interpretations that deepen the reader’s engagement with the material [ 3 ]. Additionally, illustrations reflect the aesthetic preferences, artistic trends, and technological capabilities of the societies in which they were created. They are often beautiful or grotesque, surprising or remarkably realistic [ 4 ]. They can showcase a deep knowledge of plants and animals [ 7 ], while also displaying whimsy, humor, theology, heresy, and the artist’s exceptional talent. They also reveal cultural exchanges, as artistic motifs and techniques have often traveled across regions and traditions, enriching the visual language of different cultures [ 5 ]. For scholars of religion, art history, and history in general, these visual elements provide insights into social hierarchies, religious practices, and even material culture, offering a multidimensional view of the past. However, the sheer scale of digital archives presents a significant challenge: manually identifying and cataloging illustrated pages within millions of scanned documents is an overwhelming and often impractical endeavor [ 8 , 9 ]. The exponential growth in digitized content, encompassing manuscripts, books, and historical documents from diverse cultural and temporal contexts, has outpaced traditional methods of scholarly analysis [ 10 ]. Sifting through these extensive collections to locate pages with visual illustrations demands substantial time, 1 Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 2/17 labor, and expertise, often making it infeasible for individual researchers or collaborative teams. Institutions like the Library of Congress, 1 British Library 2 , and the Biblioth ` eque nationale de France (BnF) 3 have released tens of thousands of high-resolution scans, covering diverse periods, styles, and cultural contexts. This diversity poses challenges for visual pattern recognition [ 11 ], but it also provides fertile ground for computational learning. The scale and variety of this visual data now make it possible to apply computer vision and deep learning techniques to manuscript analysis. Building on this opportunity, our study presents a scalable and generalizable approach for identifying illustrated pages within digitized manuscripts. While the challenge of locating illustrations in digitized manuscripts has long been recognized [ 9 , 12 ], existing solutions often fall short when applied at scale. Previous approaches have relied on image segmentation techniques that attempt to label each pixel in an image as belonging to categories such as text, decoration, or marginalia [ 13 ]. While segmentation techniques can be precise in theory, they often require substantial pre-processing (such as binarization and de-skewing) and post-processing (such as filtering or assembling bounding regions) to produce usable results [ 9 ]. These methods also tend to involve long processing times per page [ 14 ], since segmentation models must generate predictions at the level of individual pixels rather than only bounding boxes (i.e., millions of pixel predictions compared to tens of box predictions per page). Consequently, in many practical pipelines, object-detection is employed as a preprocessing stage for subsequent segmentation tasks [ 15 ]. As a result, such approaches are generally unsuitable for use as off-theshelf tools in high-volume manuscript digitization projects. Our work builds on this foundation but shifts the focus toward lightweight, generalizable, and scalable deep-learning-based detection methods. Motivated by the need to process millions of scanned pages, we developed a pipeline designed to identify, extract, and describe visual content in large-scale historical collections. In the first stage, a convolutional neural network classifies each page as either “illustrated” or “non-illustrated” depending on the presence of illustration in that page. This allows us to discard the majority of empty pages or those that contain only text, and focus only on the relevant visual material. In the second stage, an object detection model locates and crops illustrations, including ornate initials, marginalia, and full miniatures. Lastly, we use vision-language models to generate rich human-readable textual descriptions of each detected illustration. These captions are stored alongside the cropped images in a database, enabling keyword-based or semantic search across extensive collections. This integrated framework 1 Library of Congress Digital Collections, accessed March 2025, https: //www.loc.gov/collections/ 2 British Library, Digitised Manuscripts, accessed March 2025, https: //www.bl.uk/manuscripts 3 Biblioth ` eque nationale de France, Gallica Digital Library, accessed March 2025, https://gallica.bnf.fr allows scholars to perform queries that were previously impossible to execute manually, for example, searching for phrases such as “winged horse” or “angel holding a sword” and instantly retrieving visually relevant fragments from millions of pages. The illustrations can be further embedded into a shared representation space and linked into an illustration-similarity graph. This additional layer provides a corpus-level view of the visual landscape, surfacing cross-manuscript relationships and clusters of images that share stylistic, iconographic, or compositional features. This graph-based perspective reveals patterns that remain invisible when pages are examined in isolation, opening new pathways for large-scale, data-driven art-historical and cultural-heritage research. While the proposed framework is designed as a general and modular pipeline, in this study we implemented it using a specific set of algorithms. The page-level classification stage employs an EfficientNet-based[ 16 ] convolutional network fine-tuned on over 20,000 manually labeled pages and taken from diverse collections spanning centuries, regions, and artistic traditions. For the object detection stage, we used a YOLO architecture[ 17 ] trained on more than 1,500 annotated examples sourced from various Vatican manuscripts and related materials. Finally, for the captioning stage, we utilized the LLaVA (Large Language and Vision Assistan) vision-language model[ 18 ] to generate detailed textual descriptions for each extracted illustration. We emphasize that the framework itself is modular by design, and as new algorithms emerge, each stage can be independently improved or replaced to enhance overall performance. In our current implementation, the classification model achieved a ROC-AUC of 0.95, demonstrating strong overall discrimination between illustrated and non-illustrated pages. At the optimal decision threshold, it reached a precision of 78.6%, recall of 74.6%, an F1-score of 76.5%, and an overall accuracy of 95.1% on the held-out test set. To evaluate our method in a realistic, large-scale setting, we applied the whole pipeline to more than 10,000 items, including over five million scanned manuscript pages from the digitized collection of the Vatican Library. 4 These collections span diverse periods, regions, and scripts [ 19 ]. The dataset includes complex layouts, varying resolutions, and frequent image artifacts such as stains, tears, or marginal notes-characteristics that pose significant challenges for computational analysis [20,21]. Our classification model filtered out over 90% of pages classified as text-only, allowing the object detection and captioning stages to focus on a smaller, more relevant subset. Overall, we identified and extracted more than 350,000 unique illustrations. The average processing time was under 0.06 seconds per page, enabling end-to-end analysis of the entire Vatican corpus within days. The illustrations were automatically described using the AI image-captioning model and indexed in a searchable database. Qualitative inspection revealed a 4 Vatican Apostolic Library, Vatican Digital Library, accessed March 2025, https://digi.vatlib.it/ Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 3/17 wide variety of visual motifs, from elaborate miniatures to marginalia and decorated initials. Our framework not only automates a historically laborintensive task, but also enables a new mode of scholarship: for the first time, researchers can systematically compare, retrieve, and analyze visual motifs across vast manuscript collections across time, geography, and artistic traditions. This paradigm shift opens the door to new discoveries in art history and cultural studies, making it possible to detect patterns, symbolic connections, and stylistic trends that have remained hidden in plain sight for centuries. Our main contributions are as follows: • We release an open scalable deep learning pipeline for identifying, extracting, and describing visual content in digitized historical manuscripts. • We demonstrate the effectiveness of our two-stage approach across noisy and heterogeneous manuscript pages, which performs reliably across noisy, irregular manuscript pages, with inference time of under 0.1 seconds per page, 5 enabling efficient processing of millions of pages. • We deploy the extracted visual content in a searchable platform that enables humanities scholars to retrieve illustrations based on semantic queries - available upon request. • Our pipeline enables new types of cultural and iconographic inquiries, for instance, tracing the evolution of symbolic motifs, identifying shared visual programs across distant manuscripts and artists, or discovering latent visual connections previously inaccessible to human review. The remainder of this paper is organized as follows: Section 2provides an overview of related work in illustration detection and manuscript image analysis. Section 3describes our methodology, including dataset construction, model selection, and pipeline implementation. Section 4presents the evaluation setup and results, including qualitative examples and runtime benchmarks. Section 5outlines potential applications and limitations. Finally, Section 6summarizes the main findings and discusses future directions for improving illustration retrieval and understanding in historical archives. 2. Related Work In this section, we review existing approaches to the analysis of historical manuscript. In Section 2.1, we outline the largescale digitization of historical collections. In Section 2.2, we examine methods for illustration detection, ranging from traditional techniques such as Optical Character Recognition (OCR) and Page Layout Analysis (PLA) to recent computer 5 Experiments were conducted on an Intel Core i7-1355U (13th Gen), 16 GB RAM, Intel Iris Xe GPU. vision approaches, including image classification and object detection. In section 2.3 we provide an overview of advances image captioning techniques and vision-language models and their emerging applications to historical collections. Finally, Section 2.4 discusses image-embedding methods and graphbased similarity representations. 2.1 Digitization In recent years, major libraries have substantially expanded their digitization initiatives, making an unprecedented amount of collections of historical manuscripts accessible to scholars [ 10 ]. Institutions such as the British Library, the Biblioth ` eque nationale de France, and the Library of Congress now offer significant portions of their historical archives online, spanning across diverse time periods, and languages. Gallica, the digital library of the Biblioth ` eque nationale de France, alone offers access to millions of high-resolution pages [ 22 ] while the British Library’s “Digitised Manuscripts” collection features over 8,000 items [ 23 ]. This digitization surge has highlighted the lack of scalable tools for analyzing digitized books in general, and visual elements such as illustrations, which remain understudied compared to text [24]. This abundance of material has significantly reshaped humanities research [ 25 ]. For instance, these manuscripts reveal the cultural dynamics of Jewish communities in their interactions with surrounding societies and how distinct Jewish traditions were formed and shaped during periods of both prosperity and persecution [ 26 ]. Beyond their cultural and historical value, manuscripts constitute the primary source material for constructing diachronic corpora, enabling linguists to investigate orthographic and grammatical changes over time. For example, large-scale historical corpora spanning more than 1,400 years in Arabic have been built from digitized manuscripts, supporting automated periodization and morphological analysis [ 27 ]. In art history, digitized manuscripts enable comparative studies like the depiction of dragons, angels, or botanical illustrations across regions and time periods [ 28 ]. Another important line of research has examined medical drawings in medieval manuscripts, revealing how anatomical sketches and disease depictions shaped the transmission of medical knowledge across Europe and the Middle East [29,30,31]. Large-scale digitization necessitates reliable image storage and retrieval solutions. For this purpose, many institutions have adopted the International Image Interoperability Framework (IIIF) [ 32 ]. IIIF provides standardized APIs to deliver, annotate, and display high-resolution images across repositories, facilitating interoperability and consistent metadata management. For example, the Vatican Library uses IIIF to provide access to millions of digitized pages from tens of thousands of manuscripts [33]. 2.2 Illustration Detection in Historical Manuscripts A significant portion of historical document analysis techniques focus on textual extraction, particularly through Optical Character Recognition (OCR) and Handwritten Text Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 4/17 Recognition (HTR) [ 34 , 35 , 36 ]. While textual content is effectively being extracted in these approaches, non-textual elements such as illustrations or marginalia are often ignored. Tools such as Transkribus [ 37 ] and eScriptorium [ 38 ], explicitly developed for historical manuscripts, employ machine learning-based models to recognize handwritten text and perform region or line segmentation. However, these systems are primarily optimized for textual content, and are not designed to detect or analyze visual elements such as illustrations or decorated initials at large scale [ 9 , 39 ]. Moreover, while these platforms perform well on Latin-script manuscripts, applying them to Semitic languages that are written right-to-left poses distinct challenges due to different writing directions, diacritics, and script structures. Another common approach is Page Layout Analysis (PLA), which segments pages into distinct content regions (e.g., headings, paragraphs, images) [ 40 , 41 , 42 ]. Although effective for comprehensive document understanding, traditional segmentation based methods such as docExtractor [ 13 ], often rely on pixel-level analysis and require complex preand post-processing pipelines [ 43 , 44 ]. These techniques can be computationally intensive and may not scale efficiently to large manuscript collections [45]. While layout segmentation tools occasionally identify illustration regions, few methods explicitly aim to extract illustrations as discrete visual units suitable for standalone analysis. This distinction is critical, especially when illustrations are embedded within ornate initials, marginalia, or complex text-visual arrangements. Traditional layout methods often treat such elements as secondary byproducts rather than primary targets [ 39 ]. Moreover, detecting illustrations in historical manuscripts poses unique challenges due to wide stylistic variation, irregular spatial organization, and frequent physical degradation (e.g., stains, bleed-through, or torn edges) [20,21]. Recent years have witnessed rapid progress in deep learning for image classification and object detection [ 46 ]. In classification, convolutional neural networks (CNNs) have evolved from early architectures like AlexNet [ 47 ] and VGG [ 48 ] to more efficient and accurate designs such as EfficientNet [ 16 ] and ResNet [ 49 ], which achieve state-of-the-art results [ 50 ]. These models are particularly effective for identifying abstract visual categories under varying conditions and have been widely adopted in domains requiring robust generalization, including historical document classification. For instance, CNN-based classifiers have been used to distinguish between pages in incunabula containing text, tables, pictures, titles, and handwriting [ 51 ]. Other studies have trained classifiers to detect Arabic manuscripts authors [52]. Parallel advancements in object detection models have produced architectures that combine high accuracy with realtime performance. Frameworks such as YOLO (“You Only Look Once”) [ 17 ], have demonstrated strong performance across diverse detection tasks [ 53 ], including challenging domains like digitized archives [ 54 ]. For example, YOLO models have been successfully applied to detect text lines and text characters in Ottoman manuscripts [ 55 ] and birch-bark manuscripts [ 54 ]. Their ability to process images quickly - often in several milliseconds per image [ 56 ] makes them ideal for large-scale archival applications. 2.3 Image Captioning and Vision-Language Models Recent breakthroughs in vision-language modeling have transformed the ability to describe and interpret visual content using pretrained image-to-text models [ 57 , 58 ]. Modern image captioning systems integrate visual encoders, often based on vision transformers (ViTs) with large language models (LLMs), enabling the generation of fluent and rich captions for images. Architectures such as BLIP (Bootstrapped LanguageImage Pretraining) [ 59 ], CLIP [ 60 ], and LLaVA [ 18 ], exemplify this approach by pairing an advanced image encoder with a language model that can either generate text or follow instructions in natural language. In particular, BLIP introduced a framework that combines image-text matching with caption generation through a twotower transformer design. CLIP showed how contrastive training on large sets of image–text pairs can bring visual and textual features into a shared space, which makes zeroshot classification and cross-modal retrieval possible. LLaVA builds on CLIP’s image encoder together with a LLaMA language model, resulting in strong performance on open-ended tasks like visual question answering and instruction following. While these models were primarily trained on modern images, several recent works have begun exploring their application to historical content [ 61 , 62 , 63 ]. For example, Thomas and Testini (2024) investigated the automated identification and analysis of image captions in a large corpus of historical book illustrations [ 61 ]. Cetinic (2021) proposed methods for generating and evaluating iconographic image captions tailored to artworks [ 62 ]. Similarly, Gupta et al. (2020) explored approaches for adapting image captioning models to art-historical datasets [63]. 2.4 Image Embedding and Graph-Based Similarity Image embedding techniques transform visual inputs into compact vector representations that capture semantic and stylistic features[ 49 ]. This dimensionality reduction significantly reduces storage requirements and computation time, while preserving the most relevant visual information, making embeddings particularly suitable for large-scale analysis and retrieval, and are particularly powerful for identifying relationships between images[ 64 ]. Modern image-embedding methods rely primarily on deep neural architectures such as convolutional networks (e.g., ResNet) [ 49 ] and vision transformers (ViT) [ 65 ]. More recent multimodal models, including CLIP [ 60 ] and ALIGN [ 66 ], have also demonstrated remarkable performance across a wide range of visual understanding tasks. In the context of cultural heritage, image embeddings have been commonly used. For example, Garcia et al. proposed an embedding framework for art-historical images, learning visual representations that integrate metadata Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 5/17 such as artist, period, and school in order to improve tasks like author attribution, style classification, and cross-modal retrieval [ 67 ]. Similarly, Springstein et al. developed the iART system, a large-scale art-historical image search engine that relies on deep learning embeddings to cluster artworks, support similarity-based exploration, and facilitate comparative analysis across collections and periods [68]. Several works have proposed constructing similarity graphs in which nodes represent embedded images and edges encode visual proximity [ 69 ]. These graphs support community detection, motif tracking, and corpus-level exploration of visual themes, enabling researchers to trace stylistic shifts and uncover connections across manuscripts. For example, Wu et al. introduced the Deep Graph Embedding framework, which constructs large-scale similarity graphs from deep features and uses them for improved image classification and retrieval [ ? ]. Similarly, Wan et al. used a graph-based re-ranking method for image retrieval, modeling the global structure of visual similarity through a kNN graph to significantly improve retrieval precision [?]. 3. Methods Our research addresses the challenge of efficiently retrieving illustrations from digitized historical manuscripts using computational methods that can scale to millions of pages in practical time. To meet this challenge, we developed a modular pipeline composed of three main stages: (1) Image Extraction, where a classification deep learning model filters out text-only pages and an object detection deep learning model identifies and crops illustration regions; (2) Image Captioning, where a vision-language model produces rich textual descriptions for each cropped illustration ; and (3) Search and Retrieval, where both illustrations and captions are stored in a searchable database, enabling scholars to perform keyword-based queries through a web interface. In addition, we construct image-similarity graphs from the extracted illustrations, enabling corpus-level analysis and the discovery of recurring visual patterns and stylistic trends across manuscripts. While the framework is designed to be general and modular, capable of integrating different models or algorithms at each stage, in this study we implemented it using a specific configuration: an EfficientNet-based classifier for pagelevel filtering, a YOLO-based detector for illustration localization, and the LLaVA model for caption generation. This concrete implementation serves as a proof of concept, while the pipeline itself remains adaptable to future advances. This section outlines the datasets, processing steps, methodologies, and evaluation procedures used to build our system. In Section 3.1, we describe the manuscript page images that serve as the dataset of our study. Sections 3.2 and 3.3 then detail the illustration extraction process, including data preprocessing, annotation, model training, and the metrics used for evaluation in both images classification and object detection stages. Section 3.4 discusses how we applied multimodal vision-language models for caption generation. Section 3.5 presents how the resulting illustrations and captions form the backbone of the retrieval system, enabling scholars to efficiently locate visual material through keyword searches. Finally, Section 3.6 describes how we construct illustration-similarity graphs from the extracted images, supporting corpus-level analysis and the discovery of visual patterns across manuscripts. 3.1 Data Acquisition We collected digitized pages from the Vatican Library’s public IIIF API [ 70 ]. We likewise incorporated the Bible of Borso d’Este 6 . These platforms provide high-resolution color scans of historical documents. Our project focused on two document types: medieval manuscripts - handwritten documents created before the invention of printing; and incunabula, printed books produced before the year 1501. Both types offer rich visual content and span diverse historical periods, regions, and artistic traditions. The Vatican Digital Library consists of 96 collections of manuscripts and incunabula from different regions, periods, languages, and artists; together, these collections comprise 28,814 items encompassing 9,664,009 pages [ 70 ]. 7 The Bible of Borso d’Este is a 15th-century Italian Renaissance masterpiece, renowned for its breathtaking illuminations, intricate decorative borders, and finely painted marginal scenes [ 71 ]. Often described as the “Mona Lisa” of illuminated manuscripts, this codex exemplifies the pinnacle of artistic achievement in Renaissance book production. 3.2 Illustration Presence Classification 3.2.1 Data Preparation and Labeling We constructed the training dataset for the classification stage through a two-step process. In the first step, we randomly sampled 1,000 images from a diverse range of manuscript and incunabula and manually labeled each page as either ”art” or ”no-art”, depending on the presence of illustrations. To ensure that our models would generalize well across volumes from different geographic regions, artists and time periods, we randomly sampled pages from the full range of manuscript and incunabula volumes available via the IIIF API [ 70 ], intentionally selecting pages from different volumes, time periods, and creators. This strategy aimed to expose our models to the broadest possible range of layouts and illustration styles to reflect the heterogeneity of historical sources. Since the images came from different manuscripts, we applied a uniform naming convention and directory structure to track each image’s source volume, page number, and manuscript metadata throughout the pipeline to ensure that subsequent search and retrieval steps would work seamlessly. Using this initial 1,000-image dataset, we fine-tuned a preliminary classifier, which was then employed to predict labels for an additional randomly selected set of 20,000 pages. 6 Accessed November 2025 via the Italian Digital Library, https:// edl.cultura.gov.it/media/schedaopen?id=3015932 7 The reported numbers are accurate as of January 2025, when the data were collected. Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 6/17 In the second step, these automatically generated predictions were carefully reviewed and manually corrected to ensure label accuracy and consistency. This semi-automated approach provided a practical balance: it enabled a substantial expansion of the labeled dataset while reducing the time and effort required for exhaustive manual annotation. As a result of this pipeline, we constructed a robust dataset of approximately 20,000 images, each reliably labeled as either ”art” (illustrated page) or ”no-art” (non-illustrated page). This dataset served as the foundation for training and evaluating our subsequent classification model and object detection one. 3.2.2 Class Imbalance Illustrated pages constituted a significant minority in the full corpus. While only 1,173 of the 20,000 images contained illustrations (5.8%), the remainder consisted of blank pages or textonly pages. To mitigate this extreme imbalance, we retained all illustrated pages while randomly discarding 9,000 nonillustrated pages. This yielded a working dataset of 11,000 images. This downsampling preserved sufficient variation in the negative class (”no art”) while improving the positive (”art”) ratio to 10.6%. While this did not yield a perfectly balanced dataset, it substantially reduced the original skew (from 5.8% positives to 10.6%), which we deemed sufficient for stable training. More aggressive balancing techniques-such as oversampling or synthetic data generation-were intentionally avoided, as these can introduce artifacts and distort the natural distribution of the corpus. Instead, we prioritized maintaining the natural diversity of the original corpus while reducing imbalance to a level that the model could effectively learn from. Similar downsampling strategies had been also employed in prior work to manage highly skewed datasets [ 72 ]. Although hybrid approaches that combine oversampling and downsampling are often reported as yielding the most robust results [ 73 ], we opted for downsampling alone in order to avoid introducing synthetic artifacts and to preserve the natural distribution of the corpus. 3.2.3 Evaluation Metrics We evaluated our classification model on a held-out test set of 1,105 images (around 10% of the labeled data). Following standard practice, we report Accuracy, Precision, Recall, F1-score, AUC, and PR-AUC as performance metrics. In addition, given the class imbalance inherent to our dataset, F1score provides a more reliable measure of overall performance than accuracy alone. Since the vast majority of pages do not contain illustrations, false positives would overwhelm downstream applications with irrelevant material such as stained or purely textual pages. For this reason, we place particular emphasis on Recall, ensuring that illustrated pages are reliably identified for subsequent processing. The dataset was split into 70% training, 20% validation, and 10% testing. This proportion reflects standard practice in deep learning [ 74 , 75 , 76 ], allocating the majority of data to training while keeping substantial, disjoint partitions for hyperparameter tuning and final evaluation. After this separation, the training set contained 821 illustrated pages and 6,908 non-illustrated pages. 3.2.4 EfficientNet Fine-Tuning For the classification stage, we fine-tuned an EfficientNet-B0 model [ 77 ], selected for its favorable trade-off between accuracy and computational efficiency on high-resolution image classification tasks [ 77 , 78 ]. Leveraging a pre-trained model allowed us to transfer features learned from large-scale natural image data (ImageNet [ 79 ]) to our domain of manuscript pages, where labeled data is comparatively scarce. Although more recent large-scale image datasets have been introduced, ImageNet remains widely adopted, as its visual diversity has been shown to generalize effectively across domains, including non-natural imagery. Given that our dataset of manuscript pages is visually distinct from everyday photographs, initializing the model with ImageNet-pretrained weights provided a stable starting point for effective adaptation to our domain. Training followed a two-stage fine-tuning protocol. In the first stage, we froze the convolutional backbone and trained only the final classification layer for 10 epochs, using binary cross-entropy loss and the ADAM optimizer with a learning rate of 10−3 that was chosen empirically, following common transfer-learning practice [ 80 ]. In the second stage, we unfroze the top portion of the network (the last 20 layers, corresponding to the final two blocks in EfficientNet-B0) and continued fine-tuning the entire network with a reduced learning rate of 10−5 that was also chosen empirically, following common transfer-learning practice [ 80 ]. Early stopping based on validation loss was employed to mitigate overfitting. To improve generalization, we applied standard data augmentations (random horizontal flips and rotations up to 20°) using TorchVision. 8 Input images were resized to 224 ×224 pixels - the standard input resolution for EfficientNet-B0 to ensure direct compatibility with the pre-trained model. The images were then normalized using the mean and standard deviation of the ImageNet dataset. The final model was trained on 11,000 labeled pages and evaluated on a held-out test set (10% of the data). Once reliable performance was established, we used the model to automatically classify over five million manuscript pages from more than 10,000 volumes. This large-scale application demonstrates the practicality of our pipeline: by pre-filtering over five million pages from more than 10,000 manuscripts, the system efficiently excluded non-illustrated pages helping reduce the computational cost of illustrations detection and captioning, which highlight the feasibility of applying our method beyond research prototypes to real-world, libraries collections. 3.3 Illustration Localization - Object Detection After identifying the pages that contain illustrations, the next step was to pinpoint the exact regions of these illustrations. In 8Implemented via torchvision.transforms. Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 7/17 many cases, illustrations occupy only small portions of a page otherwise filled with text, making it inefficient to store entire pages and lowering the accuracy of subsequent captioning. To address this, we fine-tuned YOLOv11 [ 17 , 81 ], chosen for its balance between detection accuracy and computational efficiency, which makes it well suited for processing millions of manuscript pages. While bounding-box localization is sufficiently precise, it’s far more practical than pixel-level segmentation for large-scale datasets [82,83]. 3.3.1 Data Preparation and Annotation We constructed a dedicated annotation dataset consisting of 1,800 pages randomly sampled from those previously flagged as illustrated during the classification stage. In selecting these pages, we prioritized diversity across artists, regions, and styles to maximize the model’s ability to generalize. Each image was manually annotated with bounding boxes using the open-source tool LabelImg [ 84 ], marking all instances of visual content, including marginalia, decorative initials, and embedded miniatures. To enable the search and retrieval system to also return sub-illustrations - for example, a main character or an animal within a larger scene - we annotated not only the overall illustration but also its major subcomponents (e.g., a specific character or figure within the drawing). Three pages containing only noise were excluded. 3.3.2 Evaluation The final dataset was split into 70% training, 20% validation, and 10% testing, following common practice in object detection tasks. Annotations were saved in YOLO format, with bounding box coordinates recorded relative to image dimensions. Evaluation was based on mean Average Precision (mAP), precision, and recall, providing complementary insights into localization accuracy and detection robustness, following common practice in object detection tasks. 3.3.3 YOLOv11 Fine-Tuning We fine-tuned the lightweight YOLOv11n architecture, initialized with pre-trained weights, for 50 epochs with a batch size of 16 on a dedicated GPU server. 9 Default data augmentation strategies provided by the Ultralytics [ 85 ] implementation were used, including color perturbations, translation, scaling, flipping, and mosaic augmentation. Training followed the default Ultralytics [ 85 ] configuration, which employed the AdamW optimizer with automatic hyperparameter tuning, early stopping to prevent overfitting, and mixed-precision optimization. Once trained, the detector was deployed on the full corpus of pages identified as containing illustrations. For each page, the model extracted cropped regions corresponding to illustration bounding boxes, producing isolated visual segments that have been stored in organized by volume, collection and library for efficient search and retrieval system. To contextualize these results, we compared our classifier with docExtractor [ 13 ], a segmentation-based system. 9The server is equipped with an NVIDIA RTX 6000 Ada, 48GB. For an optimal comparison, we converted the pixel-level segmentation outputs of docExtractor into bounding boxes. Specifically, we applied their algorithm to the 179 test images to obtain binary masks of illustration regions, then performed morphological closing to merge nearby areas and removed small noisy regions. Bounding boxes were then extracted from the cleaned masks and evaluated against the YOLO ground truth labels. After experimenting with different values (3, 5, and 10 pixels) for the morphological closing parameter, we found that a 10-pixel tolerance yielded the most consistent results. This procedure allowed us to evaluate docExtractor using the same classification and detection metrics as our pipeline. 3.4 Image Captioning To enable meaningful retrieval and exploration of the illustrations, we generated a textual caption for each cropped image using a vision-language model. Unlike classification or detection tasks that focus on identifying the presence or location of objects, image captioning addresses the semantic content describing what the image portrays [ 57 , 86 ]. Captions were expected to capture key visual features such as objects, emotions and colors (e.g., “a crowned figure holding a scepter,” “a floral border,” or “a medieval battle scene”). We evaluated several state-of-the-art models, including Florence-2[ 87 ], BLIP [ 59 ], and LLaVA [ 18 ], using them off the shelf without additional fine-tuning. To compare their qualitative performance, we conducted an empirical manual evaluation over a representative set of 100 manuscript illustrations. While we did not apply quantitative metrics (e.g., BLEU or CIDEr) at this stage, the manual inspection revealed noticeable differences in output quality. We observed that LLaVA tended to generate captions that were more accurate, linguistically richer, and better able to reflect the stylistic diversity of the illustrations. Based on this empirical judgment, we selected LLaVA for further development. The generated captions were then stored alongside the associated cropped illustrations and metadata. 3.5 Search and Retrieval In the final stage, the cropped images, their captions, and manuscript metadata were integrated into a searchable database and made accessible through a lightweight web interface. Each record contained the cropped illustration, its automatically generated caption, manuscript identifier, page number, and an IIIF URL linking back to the full digitized volume. By integrating classification, detection, and captioning into a unified modular framework, our system enables scalable semantic search and retrieval of historical visual content. These capabilities provide humanities scholars with direct verbal access to illustrations otherwise hidden within large-scale manuscript collections, opening new possibilities for studying visual culture across time, style, and geography. Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 8/17 Figure 1. Our pipeline for transforming a vast collection of scanned historical document pages into a searchable system for artwork and illustrations. 3.6 Illustration Similarity Graph On top of the search and retrieval layer, we generated visual embeddings - numerical representations that capture the stylistic and semantic content of each illustration using CLIP [ 60 ]. We measured similarity between pairs of embeddings using cosine similarity and, for each image, connected it to its fifty nearest neighbors in the embedding space. This procedure produced an illustration-similarity graph in which each node corresponds to an extracted image and edges connect visually related pairs. This graph offers a corpus-level perspective on visual relationships across the collection. Analyzing its structure reveals coherent communities of related images—clusters that reflect shared stylistic features, iconographic motifs, or thematic elements. The graph serves two complementary purposes. First, it enables interactive exploratory browsing: starting from any illustration, scholars can navigate to visually similar images and trace chains of related motifs both within and across manuscripts. Second, applying graph clustering and community detection methods allows the identification of groups of images that share common visual patterns, such as similar compositions, color palettes, or page-layout positions. These communities support higher-level inquiries, for example, tracing the evolution of animal imagery across manuscripts from different periods. 4. Results To evaluate our system’s performance, we tested it on a dedicated dataset of digitized collections from the Vatican Library and the Bible of Borso d’Este. The Vatican Library offers an exceptionally diverse corpus, spanning a wide temporal range, numerous artistic and scribal traditions, and a rich variety of materials, making it an ideal source for training and evaluating our algorithms on heterogeneous data. In contrast, Bible of Borso d’Este provide a compact yet illustration-rich item, providing a focused test case that allow us to assess the system’s ability to detect and analyze richly decorated pages within a more constrained corpus. Our objectives were threefold: (1) accurately distinguishing illustrated pages from text-only ones; (2) detecting and cropping illustrations within pages; and (3) generating meaningful textual descriptions. In this section, we present the results of each stage of the pipeline, followed by an analysis of overall throughput and scalability, including the downstream construction of an illustration similarity graph, which leverages the extracted embeddings to reveal corpus-level visual structure. 4.1 Classification: Illustration Presence We first evaluated our page-level classification model (EfficientNet-B0) on a held-out test set of 1,105 images that were not used during training or validation. The set contained 118 illustrated pages (”art”) and 987 text-only pages (”noart”), mirroring the real distribution while ensuring a balanced evaluation of 10.6%. The model demonstrated strong overall discrimination performance, achieving a precision of 78.6%, recall of 74.6%, F1-score of 76.5%, and an accuracy of 95.1%. The area under the ROC curve (ROC-AUC) reached 0.95, and the area under the precision-recall curve (PR-AUC) was 0.82, indicating a Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 9/17 high ability to separate illustrated from non-illustrated pages despite the natural class imbalance. Notably, when adopting a lower decision threshold (0.2) that prioritizes recall over precision for the ”art” class, recall increased to 83%, capturing a substantially larger share of illustrated pages, at the cost of a modest rise in false positives. For comparison, a naive baseline that predicts all pages as “non-illustrated” would achieve an apparent accuracy of 89.3%, reflecting the dominance of text-only pages in the dataset. However, such a model would completely fail to identify any illustrated pages (precision = 0, recall = 0, F1 = 0), providing no practical utility. In contrast, our model achieves both high accuracy (95.1%) and strong discrimination (precision = 78.6 %, recall = 74.6 %, F1 = 76.5 %). 4.2 Detection: Illustration Localization For illustration localization, the YOLOv11n model was evaluated on a test set of 179 images containing bounding boxes for all visual regions. Results are presented in Table 1. Table 1. Illustration detection performance on 179 test images. Model [email protected] [email protected]:0.95 Precision Recall YOLOv11n 75.6% 51.2% 55.3% 78.7% Our test dataset consisted of 179 images containing 404 manually labeled illustrations. Of these, 318 were successfully detected by our fine-tuned model, while 86 were missed, yielding a relatively high recall of 0.79. In addition, our model identified 257 illustrations that had not been manually labeled as such, resulting in a precision of 0.55 - meaning that slightly more than half of the detected illustrations were indeed annotated ground-truth illustrations. Through the comparison with docExtractor , a segmentation-based approach, their system detected 134 of the 404 ground-truth illustrations while producing 608 false positives. In terms of efficiency, our model processed each page in 0.06 seconds, compared with an average of 51 seconds for docExtractor on the same hardware. These measurements refer only to the segmentation-mask stage and exclude preand post-processing overhead. 4.3 Captioning: Generating Descriptions After illustrations were cropped, we applied LLaVA for caption generation. A qualitative evaluation of 100 randomly sampled illustrations showed that more than 75% of generated captions accurately described the illustrations and captured the main semantic elements (objects, figures, actions). However, the model struggled with detailed illustrations, sometimes missing finer elements. It also tended to misinterpret decorative initials and occasionally identified abstract shapes as animals or people. Nevertheless, in most cases the generated captions were sufficiently accurate to enable meaningful keyword-based Figure 2. Examples of detected illustrations using our fine-tuned YOLOv11n. Bounding boxes highlight illustration regions. Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 16/17 [81] R. Khanam and M. Hussain, “Yolov11: An overview of the key architectural enhancements,” arXiv preprint arXiv:2410.17725, 2024. [82] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll ´ ar, “Microsoft coco: Common objects in context,” 2015. [Online]. Available: https://arxiv.org/abs/1405.0312 [83] X. Zhang, H. Li, F. Meng, Z. Song, and L. Xu, “Segmenting beyond the bounding box for instance segmentation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 2, pp. 704–714, 2021. [84] Tzutalin, “Labelimg,” https://github.com/tzutalin/ labelImg, 2015, git code. [85] G. Jocher, J. Qiu, and A. Chaurasia, “Ultralytics YOLO,” https://github.com/ultralytics/ultralytics, Jan. 2023, version 8.0.0, License: AGPL-3.0. [86] R. Bernardi, R. Cakici, D. Elliott, A. Erdem, E. Erdem, N. Ikizler-Cinbis, F. Keller, A. Muscat, and B. Plank, “Automatic description generation from images: A survey of models, datasets, and evaluation measures,” Journal of Artificial Intelligence Research, vol. 55, pp. 409–442, 2016. [87] Y. Yao, J. Zhang, J. Yang, P. Zhang, X. Xu, X. Wang, J. Han, J. Chen, M. Zhou, J. Bian et al., “Florence-2: Advancing a unified representation for a variety of vision tasks,” arXiv preprint arXiv:2407.10793, 2024. [Online]. Available: https://arxiv.org/abs/2407.10793 [88] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” 2023. [Online]. Available: https://arxiv.org/abs/2308.12966 [89] M. AI, “Meta llama 4: The future of multimodal ai,” SSRN, Tech. Rep., 2025. [90] OpenAI, “Chatgpt: Optimizing language models for dialogue,” https://openai.com/research/chatgpt, 2023, accessed: 2025-09-10. Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach — 17/17 (a) Global illustration similarity network with highlighted regions corresponding to two selected communities. (b) Zoomed-in view of a community dominated by decorated initials. (c) Zoomed-in view of a community composed of general painted scenes. Figure 4. Similarity network of illustrations from the Bible of Borso d’Este. (a) The full network layout, where two regions are marked to indicate communities selected for closer inspection. (b) A zoomed-in cluster centered on decorated initials. (c) A zoomed-in cluster consisting primarily of narrative and ornamental painted scenes.