PATOF – From the PAst TO the Future: Legacy Data in Small and Medium-Scale "PUNCH" Experiments – a Blueprint for PUNCH and Other Disciplines

PATOF - Deliverable D5 Ding-Ze Hu May 2025 Abstract The purpose of this report is to audit the aims and objectives of the task package and the results achieved by the case studies, as well as to clarify the list of relevant topics for further development. This report describes the aims and objectives of the task package and the results delivered. 1 Introduction Data from the “PUNCH” (Particles, Universe, NuClei & Hadrons) disciplines particle / astroparticle / hadron & nuclear physics, and astronomy are valuable and often allow for new scientific insights not expected during an experiment’s lifetime. The PUNCH disciplines are very experienced in data management, largely due to early digital data acquisition systems, high data rates & volumes to be managed, and globally distributed user communities. The experiment and strategy of metadata management from PUNCH will be utilized in the PATOF project and we will go beyond. As the DESY library, we provide a “cookbook” capturing the conceptual methodology for making individual experiment-specific data FAIR and describing a “FAIR Metadata Factory”, i.e. a process to create a naturally evolved metadata schema by extending the last version of the DataCite metadata schema without discarding the original individual metadata concepts. 1.1 Metadata Factory Metadata generation is a crucial task for enhancing the FAIRness in scientific data from concrete experiments. In this project we capture the conceptual methodology for making individual experiment-specific metadata schemas FAIR and describing a ”FAIR Metadata Factory”, i.e. a process to create a naturally evolved metadata schema by extending the latest DataCite metadata schema without discarding the original individual metadata concepts. The result is a schema that contains FAIR experiment-specific extensions on top of the schema that is already defined by DataCite. 1 1.1.1 Design of the metadata schema in Metadata Factory In order to generate XML metadata records with well-defined and wellstructured schema, the metadata schema defined by DataCite is being used in the metadata generation process in the Metadata Factory. As for the schema designed for the metadata factory has two layers, the top-layer metadata schema with common metadata fields follows along the schema defined by DataCite, which is already used by many research institutes within Germany for metadata modeling for their research data. The second layer, which encompasses experiment-specific metadata, is currently being designed and refined based on data and feedback provided by domain experts from concrete experimental contexts. These contributions are helping to ensure that the resulting structures accurately reflect real use cases and can accommodate the diversity of data formats, practices, and requirements observed across different research environments. As the metadata schema defined by DataCite is well-established, also DataCite as a DOI register agency for research data sets and it is used by a large number of research institutions in Germany. We decide to utilize the metadata schema from DataCite for the ALPS II use case. As for the metadata schema being designed, there are two types of schema for metadata records: One type is for DOI minting, which contains the bare minimum information that is needed when minting a DOI for this data set. On the other hand, another type of metadata record contains a lot more information than just bare minimum. Both types of metadata records are linked together. The following is the metadata fields in the preliminary metadata schema that is being designed from one of the experiment use cases, ALPS II: •identifier •creators •titles •subjects •contributors •publisher •publicationYear •resourceType •DESY ALPSII instrument –device –UUID –deviceType –deviceName –manufacturer –firmwares –hardwares –dates –network 2 –formfactor –relatedItems –installation –specifics •DESY ALPSII software –UUID –softwareType –softwareName –contributors ∗contributor ·givenName ·familyName ·nameIdentifier ·affiliation –version –operationSystem –dates –relatedItems –installation –specifics 1.1.2 XML (Extensible Markup Language) for metadata record For this use case, we create metadata records using XML (Extensible Markup Language), which is one of the most common formats for representing structured metadata. The way metadata is used typically implies that it will be shared and interpreted beyond its original creator community, requiring a clear and consistent structure. Extensible Markup Language is the underlying standard that enables such interoperability, providing a flexible yet well-defined framework for describing, exchanging, and validating metadata across diverse systems and domains. 1.1.3 XSD (XML Schema Definition) for defining metadata To ensure that XML metadata records are structured consistently and can be validated automatically, an accompanying XML Schema Definition (XSD) file will be developed. The XSD specifies the elements, attributes, and relationships that define the structure and content of the metadata, effectively serving as a blueprint for data creators and users alike. By providing this formal schema, it becomes possible to check the correctness of XML files, maintain interoperability across different systems, and enable automated processing of metadata within and beyond the originating community. 3 •Usage of metadata schema from the latest version of DataCite metadata schema: As the metadata schema defined by DataCite is wellestablished, also DataCite as a DOI register agency for research data sets and it is used by a large number of research institutions in Germany. We decide to utilize the metadata schema from DataCite for the ALPS II use case. •Two types of metadata records: Two types of metadata records are created for a data set. One type is for DOI minting, which contains the bare minimum information that is needed when minting a DOI for this data set. On the other hand, another type of metadata record contains a lot more information than just bare minimum. Both types of metadata records are linked together. •Each data fields in the metadata record that is used for DOI minting –identifier –creator –title –publisher –publicationYear –resourceType •Each data fields in the metadata records that contains a lot more information –subject –contributors –relatedItems –collaborations –DESY ALPSII instruments hardware –DESY ALPSII software sofware 1.2 Metadata cookbook Building upon the Metadata Factory, the metadata cookbook consolidates conceptual, technical, and procedural knowledge into a coherent reference framework. It translates the experience gained from schema design, workflow definition, and software implementation into a structured and reusable form. The cookbook serves as both a record of the methods employed and a practical resource that supports consistent metadata generation across different experimental domains. The metadata cookbook serves as both a conceptual framework and a practical guide for the creation, extension, and management of metadata within the Metadata Factory. It consolidates the knowledge and experience gained through the development of metadata schemas, workflows, and software components, translating them into a structured and reproducible methodology. The cookbook aims to support researchers, data managers, and developers in generating consistent, interoperable, and FAIR-compliant metadata across a variety of experimental contexts. 4 At its core, the cookbook outlines a systematic process for producing metadata records that integrate two complementary layers: the top-layer, based on the well-established DataCite metadata schema, and the secondlayer, which contains experiment-specific extensions derived from collaboration with domain experts. The top-layer ensures alignment with widely accepted metadata standards and supports DOI registration, while the second-layer captures the contextual richness of individual experiments, including instruments, software, and data acquisition details. Together, these layers enable both interoperability across disciplines and precision within specific scientific domains. Conceptually, the cookbook describes not only the structure of metadata but also the reasoning behind its design. It provides an overview of how common metadata elements can be reused, adapted, and extended in a controlled manner. Practically, it offers templates, examples, and guidance for generating XML metadata records validated by corresponding XSD schemas. These examples illustrate how metadata fields can be populated, linked, and verified to ensure internal consistency and compliance with established community practices. In addition to serving as documentation of the metadata schema, the cookbook functions as a living resource that can evolve with the needs of the community. It facilitates the exchange of knowledge between research groups by providing a shared language and framework for describing experimental data. The intention is that the cookbook can be applied beyond the initial use cases, guiding new communities in adopting or extending the Metadata Factory approach while maintaining interoperability with existing infrastructures. Ultimately, the metadata cookbook bridges the conceptual and operational dimensions of metadata management. It captures the rationale, the process, and the implementation details required to generate meaningful, structured metadata that is both machine-readable and scientifically relevant. By combining methodological clarity with practical tools, the cookbook supports a sustainable path toward more coherent and reusable metadata practices across diverse research environments. 1.2.1 Conceptual and Practical Guide for Metadata Generation This section outlines the practical steps and conceptual principles that guide the generation of metadata within the Metadata Factory framework. The process combines standardized top-layer metadata fields with experiment-specific extensions to ensure both interoperability and contextual accuracy. The approach aims to enable the reproducible generation of metadata records that are compliant with DataCite standards while remaining adaptable to the unique needs of individual research communities. Step 1: Establishing the Core Metadata Layer The first step in the metadata generation process is to define the top-layer metadata fields that are based on the DataCite schema. These elements form the backbone of all metadata records and include core identifiers such as 5 identifier,creators,titles,publisher, and publicationYear. These fields ensure that every record can be recognized, cited, and integrated into existing data catalogues such as SciCat or institutional repositories. The cookbook provides templates and XML snippets that demonstrate how these fields are structured and how they correspond to DataCitecompliant elements. This ensures that generated metadata records maintain a consistent baseline and can be seamlessly incorporated into DOI registration workflows. Step 2: Incorporating Experiment-Specific Metadata The second layer of the metadata schema captures domainor experimentspecific details. These include fields describing instruments, software, acquisition settings, and experimental parameters that provide the necessary scientific context. For example, within the ALPS II use case, metadata extensions such as DESY ALPSII instrument and DESY ALPSII software have been defined to document detailed hardware and software information. In practice, these fields are designed collaboratively with domain experts, ensuring that the vocabulary, structure, and granularity accurately reflect real experimental conditions. The cookbook illustrates how new fields can be introduced through XML Schema Definition (XSD) extensions, while still maintaining compatibility with the DataCite core. Step 3: Schema Definition and Validation For each set of metadata records, a corresponding XSD file defines the allowed structure, relationships, and data types. The cookbook includes examples of XSD structures, showing how experiment-specific elements can be declared and linked to existing DataCite components. Validation tools are introduced to verify that XML metadata records conform to the schema, ensuring quality and interoperability. Researchers and data managers can validate XML records using standard schema validation tools or through automated pipelines integrated into the Metadata Factory workflow. This validation process enhances the consistency and reliability of metadata produced across different use cases. Step 4: Workflow Integration and Automation The metadata generation process will be designed to integrate with broader data management workflows. Within the Metadata Factory, automation scripts or modules can populate XML templates using data provided from local databases, instruments, or user inputs. The cookbook describes how these components interact, providing a practical overview of the data flow from source information to validated metadata record. Examples in the cookbook illustrate how workflow steps can be modularized, allowing new experimental setups or metadata schemas to be incorporated without major structural changes. This supports scalability and flexibility across different research contexts. 6 Step 5: Sharing and Reuse Once generated and validated, metadata records can be deposited into shared catalogues such as SciCat or institutional repositories, where they can be searched, reused, and linked to publications or datasets. The cookbook provides guidance on aligning metadata output with repository requirements and DataCite DOI registration formats. This alignment promotes wider visibility and facilitates integration into existing research infrastructures. Summary In summary, the metadata cookbook provides a structured yet adaptable framework for generating, validating, and integrating metadata. It guides users from conceptual schema design to practical implementation, ensuring that metadata records remain FAIR, interoperable, and scientifically meaningful. The approach combines a stable foundation in the DataCite standard with the flexibility to incorporate experimentspecific details, offering a reproducible and extensible model for metadata creation across diverse scientific disciplines. 7