Agent-Based Structuring of Multimodal Pipe Organ Resources: Scalable Methodologies for Heterogeneous Data Ingestion and Knowledge Integration

Agent-Based Structuring of Multimodal Pipe Organ Resources: Scalable Methodologies for Heterogeneous Data Ingestion and Knowledge Integration Working Paper [v1] 2025 Dominik Ukolov Digital Humanities (Image/Object), Friedrich-Schiller-University Jena Research Center DIGITAL ORGANOLOGY, Leipzig University [email protected] Working Paper [v1] submitted on 21 November, 2025. DOI [v1]: 10.5281/zenodo.17643049 Abstract The computational modeling of complex musical heritage faces significant ontological challenges, particularly regarding the pipe organ as a distributed and evolving Hyperobject that resists static cataloging. Conventional documentation methods are currently constrained by the Ingestion Bottleneck, a critical disparity between the volume of unstructured historical data and available human curation resources. Developed within the scope of the MODAVIS project, this work presents a specialized agentic ingestion framework designed to address these limitations. A Store-First, Structure-Later architecture is introduced, rigorously distinguishing between the legally persistent Continuant and its transient material Manifestations. By implementing a confidence-driven escalation loop, the system optimizes computational resources by routing normalization tasks from heuristic parsers to Large Language Models only when specific uncertainty thresholds are met. Furthermore, an agentic protocol is defined that generates strict Intermediate Representations rather than raw SQL, thereby securing database integrity while populating a dynamic and high-fidelity knowledge graph. Keywords Digitization ·Musical Instrument ·Pipe Organ ·Augmentation ·Recording ·Software Agentic Structuring of Pipe Organ Resources Working Paper [v1] 1 Introduction Developed within the context of the ongoing doctoral project MODAVIS (Multimodal Organ Data Analysis and Virtualization Systems), this research addresses the computational challenges inherent in modeling complex musical instrument heritage. The pipe organ occupies a singular position in the taxonomy of this object class. Unlike the discrete, portable artifacts typically cataloged in museum databases (such as violins or woodwinds), the pipe organ constitutes a “Hyperobject” in the ecological sense defined by Morton [1]: an entity that is distributed across time and space, massively complex, and intimately entangled with its architectural environment. Permanently affixed to a host structure yet subject to ceaseless metamorphosis, a single instrument may originate in the 17th century, undergo a re-pitching (pitch standard modification) in the 18th, receive pneumatic action in the 19th, and be converted to electro-pneumatic transmission in the 20th. Consequently, the digital representation of such an object cannot function as a static census record; it must model a non-linear lineage of material and acoustic states. 1.1 The Organ as a Hyperobject: Epistemological Challenges This continuous transformation presents the classic Ship of Theseus paradox within the domain of cultural heritage. If an organ retains its façade but possesses windchests, pipework, and action replaced over three centuries, does it persist as the same entity? Current documentation models, such as the National Pipe Organ Register (NPOR) [2] or Musical Instrument Museums Online (MIMO) [3], typically resolve this ambiguity by creating a new record only upon total destruction or by appending unstructured text notes to a static entry. This “snapshot” approach fails to capture the ontological reality of the instrument, collapsing centuries of evolution into a flat description that resists computational analysis. A central challenge for digital organology is to rigorously distinguish between the Continuant (the enduring legal or conceptual identity) and its multiple Manifestations (temporal, material states), enabling precise provenance tracking across centuries of modification. 1.1.1 Data Silos and Fragmentation The current landscape of organological data is characterized by severe fragmentation, where critical information is scattered across isolated silos lacking semantic interoperability: • Textual Archives: Web-accessible representations of pipe organ catalogues and archives often contain rich historical data but exist as disparate, unstructured, or semi-structured resources. • Acoustical Repositories: Virtual instruments and open-source alternatives such as GrandOrgue [4] provide audio recordings and physical models of pipe behavior but frequently lack deep historical context regarding the specific state of the instrument at the moment of capture, which is solved by the Sonus model [5]. • Relational Encyclopedias: Platforms such as musiXplora [6] offer distinct entity linking for persons and locations but often lack the API availability or structured data interfaces required for interoperable communication. This separation severs the critical link between the material history of an object (e.g., a change in wind pressure) and its sonic output (e.g., a change in timbre), hindering multimodal research. 1.1.2 The Ingestion Bottleneck The primary barrier to unifying these resources is the prohibitive cost of data ingestion. Populating a highly normalized schema (one distinguishing between a Juridical Person, an Organizational Configuration, and a Physical Manifestation) requires expert-level domain knowledge. Manually structuring thousands of 1 Agentic Structuring of Pipe Organ Resources Working Paper [v1] heterogeneous, multilingual, and ambiguous historical records is computationally and financially unscalable. This is termed the Ingestion Bottleneck: the widening gap between the volume of available digital heritage data and the capacity of human curators to transform it into structured semantic knowledge. To address this, incoming data streams (often JSONs pre-structured via various source-specific approaches) must be integrated into the MODAVIS Data Model. This highly normalized relational structure serves as the backbone for subsequent knowledge graph generation. 1.2 Related Work: Automated Ontology Population While rule-based Extract-Transform-Load (ETL) pipelines have historically dominated cultural heritage data processing, recent approaches have employed Large Language Models (LLMs) for zero-shot Named Entity Recognition (NER) and text-to-knowledge-graph (KG) pipelines [7], such as ATR4CH [8, 9]. Within the broader domain of database interfaces, recent surveys on LLM-based Text-to-SQL highlight “Decomposition” ( C1 ) as a critical methodology for handling complex queries [10]. Similarly, frameworks like DBCOPILOT have addressed the “Semantic Mismatch” between natural language and domain-specific schemas through specialized schema routing [11]. MODAVIS extends these approaches by introducing a state-aware, ontologyconstrained agentic loop tailored to long-lived, evolving cultural artifacts. 1.3 The “Store-First, Structure-Later” Paradigm To overcome the Ingestion Bottleneck, an Extract-Load-Transform (ELT) paradigm is adopted, a pattern increasingly utilized in data lakes and knowledge-graph construction, tailored specifically for the semantic enrichment of cultural heritage objects. 1.3.1 Architecture Shift In standard ETL workflows, data is cleaned and structured prior to database entry. This necessitates rigid input validation that often rejects ambiguous historical data (e.g., “built circa 1750s”). In contrast, the proposed architecture implements a “Store-First” philosophy: all incoming data, regardless of structure, is immediately serialized as immutable JSONB blobs within a dedicated ingestor.context_store table. To optimize storage efficiency for high-volume ingestion, the system leverages database-native compression. The entries are cryptographically hashed for integrity, aligning with data-lake patterns that enable scalable, schema-later enrichment. 1.3.2 Post-Ingestion Processing Normalization is decoupled from acquisition and performed as an asynchronous post-ingestion process. This facilitates a Gather & Process strategy, wherein complex structuring tasks are delegated to autonomous agents that map raw inputs to the normalized MODAVIS Data Model. Crucially, this architecture supports iterative refinement: as LLMs improve, they can re-process the immutable raw data stored in the context_store table to generate higher-fidelity relations without requiring re-acquisition of the source material. This effectively future-proofs the database against advancements in Artificial Intelligence (AI), allowing the semantic depth of the knowledge graph to evolve over time. 1.4 Paper Contributions This work presents a significant part of MODAVIS [12], a framework integrating these principles into a production-ready research environment. The specific contributions of this paper are: 1. A Scalable Agentic Pipeline: A cost-efficient, confidence-driven 6-step escalation loop for data normalization is introduced. This pipeline minimizes computational costs by utilizing heuristic 2 Agentic Structuring of Pipe Organ Resources Working Paper [v1] parsing and Small Language Models (SLMs) for Tier 1 tasks, escalating to resource-intensive LLMs and extrinsic searches only when internal confidence thresholds (θ) are not met. 2. Agentic Intermediate Representation (IR) Protocol: A novel methodology is demonstrated wherein agents interact with the database schema via a deterministic safety layer. Agents are constrained to generate strictly typed JSON objects (Intermediate Representations) validated against Pydantic models [13]. These objects are subsequently transduced into SQL transactions by a deterministic engine, ensuring referential integrity and preventing hallucinated schema violations. 2 Ontological Targets: Defining the Agent’s Goal To automate the structuring of organological data, agents must be directed toward a precise ontological target. The MODAVIS data model rejects the flat, attribute-based schemas characterizing traditional inventories in favor of a high-fidelity, realist ontology capable of modeling identity through time. 2.1 Distinguishing Continuants and Manifestations The primary classification task for the ingestion agents is to map input entities to specific Continuants—persistent identities that maintain their essence despite changes in state. The specific spatiotemporal state of a continuant is subsequently derived as a Manifestation. 2.1.1 Material Continuants and Object Identity AMaterial Continuant corresponds to the ‘object’ type within the types.continuant schema. It represents the enduring physical identity of an entity, serving as the anchor for all temporal attributes. For example, ‘The Arp Schnitger Organ at St. Jacobi’ is a continuant that persists from 1693 to the present. The goal of the agent is to identify this persistent entity and link all incoming data to it, such as descriptions of an 1836 restoration by Johann Gottlieb Wolfsteller. The restoration event itself is not the object, but a causal dependency that initiates a transition to a new Transformative Manifestation of that continuant. This strictly separates the identity of the instrument from its mutable physical properties. 2.1.2 Juridical Continuants and Organizational Identity A novel contribution of this schema is the extension of the ‘Ship of Theseus’ solution to agents and organizations. The system distinguishes between the Juridical Person (the legal entity, type agent) and its Organizational Configuration (its temporal state). For instance, if a text states that a pipe organ was ‘acquired by the Royal Museum in 1960’, the agent must link the attribution to the persistent Juridical Continuant of the museum. Because the names and legal forms of organizations and persons may change over time, the label ‘Royal Museum’ is treated as a time-bound property of the specific manifestation operative at that timestamp. This allows the database to resolve the correct historical name (e.g., ‘Royal Museum’ vs. ‘National Institute’) dynamically based on the query date, effectively obviating the need for brittle hardcoded string comparisons. 2.1.3 The Expression Layer for Musical Assets For musical resources, the ontology utilizes a specialized hierarchy. While sharing conceptual parallels with IFLA LRM (and its predecessor FRBR) [14] as used in music librarianship (specifically regarding the separation of Work and Expression), MODAVIS implements a distinct data model optimized for data provenance and complex manifestation lineage [5]. Agents processing audio data must distinguish between these layers to ensure accurate cataloging: 3 Agentic Structuring of Pipe Organ Resources Working Paper [v1] • The Musical Work (Continuant): The abstract composition that maintains its identity through different representations (e.g., J.S. Bach’s Toccata in D). • The Performance (Activity): The specific temporal event is an activity of type Expression (e.g., the recording session on May 10, 1995), related to the musical work. • The Recording (Manifestation): As the recording session produces a recording that realizes the expression of a musical work, this output is modeled as a Manifestation of that Expression. This manifestation can be further classified as a digital and constitutive manifestation, and may, through subsequent processing workflows (e.g., mastering, format conversion, pressing), give rise to physical carriers and additional Derivative Manifestations derived from the constitutive manifestation. • The Copy (Instance): This recording (manifestation) of the musical work (continuant) can be copied, thus resulting in an Instance of the Manifestation. This may be the digital object or carrier (e.g., the specific WAV file or CD). This separation enables agents to correctly classify ‘remastered’ tracks not as new performances, but as Derivative Manifestations linked to the original Expression via a processing protocol. In practice, highly specialized, domain-informed agents are deployed, e.g. agents tailored to organ-restoration documentation or musical work to person relationships. 2.2 The Epistemic Object: Constructing Claims A key constraint of the agentic system is that it does not write absolute ‘truth’ into the database. Instead, it generates Epistemic Objects that explicitly encode claim-source pairs, creating an Epistemic Dependency. Every relation created by an agent, whether it asserts a build date or an attribution, is therefore linked to one or more sources. This allows conflicting claims to coexist as distinct, provenance-qualified assertions. 2.2.1 Traceability and Locality Agents are required to populate the ingestor.context_provenance table for every assertion. This table links the generated relation back to a specific Locality within the source material (e.g., a character range in a text description or a bounding box in an image). This ensures that the ‘reasoning’ of the agent is permanently auditable, allowing human curators to verify the exact sentence that led to a specific classification. This architectural decision aligns with the concept of ‘meta-algorithmic curation’ proposed by Arantes [15], which posits that algorithms must be rendered visible and open to critique rather than accepted as neutral arbiters. By reifying the ‘claim’ as a database entity, the system avoids the ‘violence of abstraction’ [16] often inherent in datafication, where nuanced historical realities are flattened into static database fields. This ensures that the database functions not merely as a repository of facts, but as a contested space of ‘data solidarity’ [17], where conflicting narratives are preserved rather than algorithmically resolved into a false consensus. 2.3 Enforcing Semantic Specificity: The Anti-Generic Constraint To ensure long-term data integrity, the schema enforces a strict ‘Anti-Generic’ constraint. Rather than writing to generic, undifferentiated ‘Event’ tables, agents are required to map actions to ontologically distinct primitives in the dynamics schema. These dynamics describe the transition between manifestations: •State Change: Must be written to dynamics.transition. •Location Change: Must be written to dynamics.displacement. •Ownership Transfer: Must be written to dynamics.attribution. 4 Agentic Structuring of Pipe Organ Resources Working Paper [v1] These activities can be linked via relations with associated types, such as causal or sequential. This constraint prevents the Category Error common in legacy databases, where distinct phenomena like moving an organ (Spatiality) and rebuilding an organ (Properties/Materiality) are generically represented as ‘history events’. 3 System Architecture: The Agentic Ecosystem The MODAVIS Ingestor’s normalization architecture is designed as a loosely coupled ecosystem of autonomous agents orchestrated around a shared, immutable data core. This design incorporates explicit versioning and rigorous monitoring to mitigate common distributed system failure modes, including agent drift, model-version skew, and conflicting schema proposals. 3.1 The Immutable Context Store The central component of the system is the ingestor.context_store table. It functions as a strict “WriteOnce, Read-Many” (WORM) repository for all incoming raw data. Regardless of the source format (PDF scan, JSON API response, or scraped HTML), data is immediately serialized into a binary JSON (JSONB) representation, LZ4-compressed by PostgreSQL 14+ to reduce storage footprint and I/O overhead while preserving efficient random access during query execution, and then persisted here. • Cryptographic Deduplication: A canonical SHA-256 hash of the bytestream ensures that identical source files are processed exactly once, even if re-ingested from disparate endpoints. • Idempotent Re-Processing: Because the raw context is preserved immutably, the system supports non-destructive iteration. As agent models advance (e.g., transitioning from GPT-5.1 to superior future architectures), the system can re-scan the ingestor.context_store to extract higher-fidelity relations without necessitating the online availability of the original source servers. 3.2 Dynamic Service Discovery via the ‘agents’ Schema Agents within MODAVIS are not hardcoded into the application logic but are defined dynamically within the persistent storage layer. The agents schema functions as a live service registry, decoupling tool definitions from execution logic. • The Agent Registry: This table encapsulates the operational profile of each agent, including its specific model version (e.g., Qwen3-4B-Instruct-2507 plus its source), its designated competencies (e.g., name_normalization,geocoding), and its current health status. • Capability Querying: The Orchestrator Agent (OrchA) does not invoke sub-agents via static API endpoints. Instead, it queries the database (e.g., SELECT * FROM agents.capabilities WHERE task='name_normalization' ) to resolve the optimal tool for a given sub-task at runtime, enabling hot-swapping of models. • Computational Provenance: To ensure algorithmic accountability, every SQL transaction generated by the system includes the specific agent_id and prompt_version used. This allows researchers to trace not merely where data originated (source provenance), but how it was interpreted (computational lineage). 3.3 Concurrency Control and Transaction Isolation The multi-agent architecture requires explicit concurrency control to prevent race conditions when multiple agents process related contexts simultaneously. A hybrid approach utilizing PostgreSQL 15 features was implemented to manage both task distribution and data consistency. 5 Agentic Structuring of Pipe Organ Resources Working Paper [v1] To manage the ingestion queue, agents utilize SELECT …FOR UPDATE SKIP LOCKED . This clause allows concurrent agents to claim distinct unprocessed records without blocking one another, effectively turning the relational database into a high-throughput message broker. For the write operations to the target knowledge graph, Serializable Snapshot Isolation (SSI) was implemented. As defined by Cahill et al. [18], SSI modifies standard snapshot isolation to automatically detect and prevent serialization anomalies, such as write-skew, at runtime without requiring the strict pessimistic locking that degrades performance. Each agent operates within a transaction that commits only after validating referential integrity. This approach aligns with modern Multi-Version Concurrency Control (MVCC) paradigms [19], ensuring that read-heavy operations, such as context retrieval, never block write operations, thereby maximizing throughput in multi-core environments. Conflicting updates to the same Continuant are detected via row-level versioning; when a serialization failure is detected, the transaction is rolled back and re-queued for processing against the updated state. 4 Methodology: The Confidence-Driven Escalation Loop Populating the knowledge graph exclusively with frontier-class LLMs would be computationally inefficient and financially prohibitive. Consequently, MODAVIS employs a confidence-driven escalation loop designed to route normalization tasks to the most economical computational agent capable of resolving them with high certainty. 4.1 The Orchestration Agent (OrchA) The architectural control plane is managed by the Orchestration Agent (OrchA). This meta-agent evaluates the complexity of each incoming input segment and governs the transition between processing tiers. For generative classification tasks, uncertainty is quantified using a Self-Consistency protocol [ Wang2023 ]. The system generates the classification k times (with k= 3 for standard operations). A Consistency Score (Sc) is derived as the ratio of the dominant semantic output count to k. To ensure the integrity of the target schema, a strict unanimity constraint ( θ= 1.0 ) is enforced. Any generation yielding Sc<1.0 is rejected. While larger sampling ( k≥10 ) would allow for majority voting convergence ( Sc≥0.7 ), the computational latency of such an approach is deemed unacceptable for highthroughput ingestion. Therefore, a deterministic rejection policy is applied: ambiguous results that fail the unanimity check are immediately escalated to the Human-in-the-Loop (HITL) queue. 4.2 Tier 1: Deterministic and Heuristic Processing The initial tier addresses unambiguous, structured data requiring no semantic inference, utilizing computationally inexpensive symbolic methods. • Step 1: Structural Parsing: Regular expressions (RegEx) and invariant parsing rules are applied to standardized data types, such as ISO-8601 temporal strings or decimal geographic coordinates. • Step 2: Approximate String Matching with Guardrails: Incoming entity references are compared against the existing Controlled Vocabulary using Levenshtein distance and Trigram similarity metrics. To prevent incorrect linkage of homonyms (e.g., distinguishing ‘Franz Caspar Schnitger’ from ‘Arp Schnitger’), a Uniqueness Constraint is enforced. If a single candidate matches within a strict tolerance threshold ( >95 %) and aligns with the rudimentary temporal bounds of the context, the relation is established. Ambiguous matches (e.g., a generic ‘Schnitger’ with multiple dynastic candidates) are deliberately failed by this tier and escalated to Tier 2 for semantic disambiguation. 6 Agentic Structuring of Pipe Organ Resources Working Paper [v1] 4.3 Tier 2: Local Semantic Association This tier resolves ambiguity by leveraging the immediate context available within the serialized input object. • Step 3: Zero-Shot Classification (Small Language Models): Efficient, low-parameter models (SLMs) are deployed for fundamental tasks such as Entity Class Disambiguation. This would include distinguishing toponyms from anthroponyms (e.g., classifying ‘Lincoln’ as a Locality versus ‘Lincoln’ as the builder Henry Cephas Lincoln) or resolving technical polysemy (e.g., distinguishing ‘Manual’ as a keyboard division versus an adjective). • Step 4: RAG-Assisted Disambiguation: If zero-shot classification proves insufficient, the agent initiates a retrieval phase. An asynchronous vectorizer, which maintains embeddings for all ingestor.context_store entries, enables dense vector search. This retrieves relevant historical context beyond exact keyword matches to support decision-making. • Step 5: Deep Inference (Frontier Models): For complex narrative descriptions containing implicit causality, high-capacity models are utilized. These agents infer latent relationships, such as deducing a ‘teacher-student’ lineage between organ builders based on stylistic descriptors present in the text. 4.4 Tier 3: Extrinsic Knowledge Generation The final tier is activated only when internal context is structurally deficient. • Step 6: The Extrinsic Generator Agent (ExGenA): In cases where critical metadata is absent (e.g., the location of a defunct historical settlement), ExGenA is authorized to execute controlled retrieval operations. Unlike preceding agents which structure existing data, ExGenA performs generative augmentation. Consequently, its outputs are tagged with a lower confidence score and prioritized for subsequent human verification. 5 Execution: From Unstructured Text to Strict Schema This section details the operational protocols agents use to transform unstructured textual descriptions into validated, strictly typed database transactions via the Intermediate Representation (IR) layer. 5.1 Deconstructing the ‘Description’ Field The primary challenge is converting a narrative block (e.g., ‘Built by the Royal Court Builder Silbermann in 1714 for the St. George Church’) into atomic facts without hallucinating schema violations. The process begins with Contextual Decomposition, where the text is segmented into candidate claims. The agent then follows an “Execute or Delegate” protocol: •Execute: If the agent identifies a simple fact (e.g., Date: 1714), it does not generate raw SQL. Instead, it instantiates a strictly typed JSON Intermediate Representation (IR) conforming to the temporal.temporality Pydantic model. This isolates the database from direct LLM injection risks. • Delegate: If the agent encounters a complex entity (e.g., ‘Hoforgelm. Fr. Ladegast’), it pauses execution and generates a Delegation Request to the specialized Name Normalization Agent, passing the raw string and local context. 7 Agentic Structuring of Pipe Organ Resources Working Paper [v1] 5.2 Logic Gate: The Dynamics Decision Tree To ensure ontological correctness, the Relation Agent (RelA) operates under a strict decision tree prompt derived from the dynamics schema. This prevents the population of generic event tables by forcing specific classifications: • IF the qualitative state of the object changed → Generate dynamics.transition (e.g., re-pitching). •IF the physical spatiality changed →Generate dynamics.displacement (e.g., relocation). • IF a component was added or removed → Generate dynamics.assembly or dynamics.disassembly (e.g., adding a stop). • IF ownership or custody transferred → Generate dynamics.attribution (e.g., sale or inheritance). For complex historical events like a ‘Restoration’, the agent is trained to recognize this as a Composite Transaction. Rather than a single entry, it generates a linked sequence of Disassembly, Transition, and Assembly IRs to capture the full granularity of the intervention. 5.3 Conflict Typology and Resolution Strategies The automated ingestion of heterogeneous organological data inevitably introduces ‘instance-level heterogeneities’ as contradictory attribute values provided by distinct sources for the same entity [20]. Within the MODAVIS architecture, a distinction is made between Source Conflicts (contradictions inherent in the historical record) and Ingestion Conflicts (inconsistencies arising from agent processing or concurrency). 5.3.1 Epistemic Preservation vs. Resolution Traditional data curation often employs a strategy of ‘Conflict Resolution’, aiming to mediate contradicting values to produce a single ‘golden record’ [20]. However, given the Hyperobject nature of the pipe organ, prematurely resolving valid historical disagreements (e.g., differing stop-counts in two survey years) results in data loss. MODAVIS adopts a strategy of ‘Conflict Preservation’ via the creation of Epistemic Objects. Rather than overwriting a record, agents append new dynamics.attribution or temporal.temporality claims linked to their specific ingestor.context_provenance . This aligns with the view that conflict resolution in digital preservation must be domain-specific and expert-guided rather than algorithmically forced at the point of entry. 5.3.2 Anomaly Detection in Agent Outputs While historical contradictions are preserved, hallucinations or parsing errors by agents must be rejected. To automate data quality validation without rigid rule-sets, the system adapts the ‘novelty detection’ approach proposed by Redyuk et al. [21]. By computing descriptive statistics on context_store batches (e.g., the distribution of build years relative to builder lifespans), the Orchestrator Agent (OrchA) identifies output vectors that significantly deviate from the learned characteristics of ‘acceptable’ data. If an agent proposes a dynamics.transition date that is statistically incongruent with the associated Continuant’s established timeline, the Confidence Metric ( θ ) is penalized, triggering the HITL escalation loop. This allows the system to self-adapt to temporal changes in data characteristics without manual constraint redefinition. 5.3.3 Eventual Consistency via Deterministic Resolution When multiple agents process related contexts simultaneously (e.g., normalizing the same builder name from different documents), the system faces race conditions regarding the canonical state of the Juridical Person. To manage this without strict locking that hinders throughput, Deterministic Conflict Resolution 8