Application of the FDO standard developed in D3.5 to example climate-simulation datasets using an automated approach - Milestone M3.5

Centre of Excellence in Simulation of Weather and Climate in Europe Phase 3 Application of the FDO standard developed in D3.5 to example climate-simulation datasets using an automated approach Milestone M3.5 Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093054. ESiWACE3 Milestone M3.5 Contact details Project Office: [email protected] Website: www.esiwace.eu YouTube:https://www.youtube.com/@esiwace880 X: https://X.com/esiwace LinkedIn: https://www.linkedin.com/company/esiwace3 ESiWACE is on Zenodo, the Open Access repository for scientific results https://zenodo.org/communities/esiwace Disclaimer: Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European High Performance Computing Joint Undertaking (JU). Neither the European Union nor the granting authority can be held responsible for them. 2 ESiWACE3 Milestone M3.5 Lead Beneficiary: Deutsches Klimarechenzentrum GmbH (DKRZ): Marco Kuluke, Karsten Peters-von Gehlen , Ivonne Anders, Beate Krüss Type: Report Dissemination level: Public Delivery date: 27.08.2025 Means of verifications: Software conceived for FDO creation and resulting FDOs published and available online. Achieved: Yes/ No If not achieved, indicate forecast achievement date: Insert dd/mm/yyyy Comments: In case of comments, please add them here below 3 ESiWACE3 Milestone M3.5 1. Introduction This milestone demonstrates the application of the FAIR Digital Object (FDO) standard, as examined in Deliverable D3.5, to example climate-simulation datasets. Building upon the strategic framework established in Milestone M3.1, we now validate the FDO concept by applying it to real-world datasets through a machine-actionable, automated approach. The implementation focuses on adopting FAIR principles by leveraging Persistent Identifiers (PIDs), Data Type Registries (DTR), and automated cataloging standards such as Spatio Temporal Asset Catalogs (STAC). 2. Objective The objective of this work is to demonstrate the practical application of the domain-specific FDO standard. This is achieved by (1) transforming climate simulation data into machine-actionable FDOs, (2) bridging data management systems through the integration of STAC-based catalogs with DTR definitions, and (3) showcasing interoperability through automated workflows. 3. Methodology In the following sections, we distinguish between the workflows for creating and using FDOs. The datasets1 employed, such as CMIP6 from CEDA and ERA5 from Copernicus, are provided as STAC catalogs and can be accessed using standardised STAC operations. 3.1 Creating FDOs The creation process involved converting these STAC-based datasets into machine-actionable FDOs by applying standardized profiles and persistent identifiers: Step 1: Define the FDO Profile and Data Type in the Data Type Registry An FDO Profile is a specification that defines the structure, semantics, and constraints for describing an FDO. It serves as a template by specifying mandatory and optional attributes, allowed value types, and relationships to ensure interoperability across systems. An FDO Record is an actual instance that conforms to such a profile. It contains the PID, metadata, and references for a specific digital object, providing a concrete, machine-actionable description of that object. 1 https://api.stac.ceda.ac.uk https://cds.climate.copernicus.eu/api/catalogue/v1/ 4 ESiWACE3 Milestone M3.5 Before creating an FDO Record, the corresponding FDO Profile must be defined. This has been carried out in alignment with the FDO community and registered in a DTR, where it is linked to a PID.2 Additionally, the data type itself (STAC) has been defined and registered in a DTR.3 Step 2: Create PIDs Each catalog was assigned a unique PID (i.e. 21.T14995/bc915619-6220-4c01-aa64-0047c1026bae for the CEDA STAC catalog of CMIP6 and 21.T14995/e4ee4e6e-84c4-47b0-ab5c-b70efaab7808 for Copernicus STAC catalog for ERA5) to ensure persistent referencing and reliable resolution. These PIDs enable consistent global open access and machine-actionable linkage across systems. The FDO record (see Fig. 1) contains key information, including references to the FDO Profile (FDO_Profile_Ref) and the data type (FDO_Type_Ref), both of which were defined in the previous step. Hence, the FDO record can be validated against its associated profile reference, while the corresponding STAC catalog can be verified using the type reference to ensure compliance with the defined specifications. This ensures that each PID is semantically enriched and can be interpreted unambiguously by automated workflows. Figure 1: Example of a STAC catalog PID handle resolved via https://proxy.handle.net/. The resolution provides a structured handle record containing metadata and references, including links to the FDO Profile (FDO_Profile_Ref) and the data type (FDO_Type_Ref), ensuring machine-actionable interoperability and persistent identification. 3 https://typeregistry.lab.pidconsortium.net/#objects/21.T11969/68398d5511a138742b47 2 https://typeregistry.lab.pidconsortium.net/#objects/21.T11969/2b3dbd8491ca0edc4dc1 5 ESiWACE3 Milestone M3.5 3.2 Using FDOs Once the FDO has been created and assigned its corresponding profile, data type, and PID, it becomes a machine-actionable digital object that can be integrated into various workflows. In the following section, we illustrate how these FDOs are utilized to enable consistent data access, automated processing, and interoperability across systems: Step 1: Accessing and validating the FDO Record The usage of FDOs leverages their structured metadata and persistent identifiers to support automated discovery, retrieval, and processing. By referencing the FDO profile and data type, clients and services can interpret the object in a standardized manner, ensuring interoperability and reproducibility across different systems and workflows. The FDO is accessed through its PID and validated by verifying that its syntax complies with the FDO profile and data type defined in the DTR. This validation step ensures consistency, semantic correctness, and interoperability across different systems and workflows, as illustrated in Figure 2. Figure 2: Access and validation of an FDO record via its PID. The process involves retrieving the FDO, verifying that its syntax and structure conform to the corresponding FDO profile and data type defined in the DTR. Step 2: Catalog Navigation and Data Discovery Automated agents can interpret and evaluate the FDO using the profile metadata (Fig. 3). This capability enables a range of operations, including catalog navigation, data discovery, and filtering based on standardized attributes. At present, these functionalities are implemented using the Python library PySTAC4. In the future, a fully FDO-native solution is envisioned, which would be programming-language agnostic and allow seamless integration of automated workflows across diverse computational environments. 4 https://pystac.readthedocs.io/en/stable/ 6 ESiWACE3 Milestone M3.5 Figure 3: Querying an FDO and displaying its information. This enables data discovery operations, supporting machine-actionable workflows. Step 3: Data Retrieval The queried data can then be retrieved for further use, as illustrated in Figure 4. This retrieval process allows automated workflows or users to access the actual dataset referenced by the STAC catalog FDO, enabling subsequent analysis, processing, or integration into downstream applications while maintaining traceability and consistency. Figure 4: Retrieval of data associated with an FDO. Once the data has been retrieved, the researcher can begin processing it. Leveraging the structured metadata and standardized formats provided by the FDO, analyses can be performed consistently and reproducibly. This enables downstream workflows such as data transformation, statistical evaluation, model input preparation, or visualization, all while maintaining traceability to the original source and ensuring interoperability across different tools and systems. Step 4: Individual Data Processing Researchers can now process the retrieved data according to their specific requirements. Figure 5 illustrates the evolution of the Earth's surface temperature, integrating information from multiple datasets made available as FDOs. This example demonstrates how standardized, 7 ESiWACE3 Milestone M3.5 machine-actionable digital objects facilitate the combination and analysis of heterogeneous data sources. Figure 5: Visualization of Earth's surface temperature using climate FDOs, illustrating the integration of multiple datasets with standardized metadata and persistent identifiers for reproducible analysis. The illustration is taken from the example notebook mentioned in 4.1. 4. Results 4.1 Transforming Climate Data Catalogs into Machine-Actionable FDOs Several STAC climate data catalogs have been transformed into FDOs, demonstrating key capabilities: (1) standardization of geospatial climate data, (2) seamless, machine-actionable catalog navigation and dataset retrieval, and (3) integration with external FDO profiles to enable advanced operations.5 To support these functionalities, a prototypical FDO manager has been developed for verification, discovery, and automated access to climate model FDOs. Implemented as a Python module, this manager has been demonstrated through a Jupyter Notebook6 that automatically retrieves and analyzes climate model outputs represented as FDOs. These results highlight the potential of FDO-based approaches to enable interoperable, automated, and reproducible workflows for climate data analysis. The use of STAC catalogs as a foundation for FDO transformation significantly enhances data discovery 6 Kulüke, M., & Krüss, B. (2025). MarcoKulueke/Application-of-the-FDO-standard: Application of the FDO Standard (v0.1). Zenodo. https://doi.org/10.5281/zenodo.16813009 5 Kulüke, M., Peters-von Gehlen, K., & Anders, I. (2025, June 11). FDO Operations Implementation Example - Record Typing. Zenodo. https://doi.org/10.5281/zenodo.15926971 8 ESiWACE3 Milestone M3.5 and retrieval by providing standardized, queryable structures, thereby facilitating efficient access to relevant datasets. 4.2 Integration with IPFS (Optional Extension) As a parallel test case, climate data from the ORCESTRA campaign was indexed in IPFS and assigned a CID, demonstrating a supplement to the mandatory FDO metadata fields. These climate data sets were harvested into the DKRZ IPFS node and linked via catalogs.7 5. Benefits & Impact This approach enables the automation of workflows for accessing, analyzing, and linking datasets, while enhancing interoperability across data spaces through standardized FDO profiles and metadata. It also supports cross-domain usability by defining data types and operations in shared DTRs. 6. Challenges Encountered The lack of universally accepted FDO specifications required iterative refinement throughout the implementation process. Metadata enrichment was occasionally constrained by the limitations of existing catalog formats, and full integration with external DTR systems remains an area of ongoing development. 7. Conclusion This work demonstrates the feasibility and benefits of applying the domain-specific FDO standard to climate data management. By transforming STAC-based climate data catalogs into machine-actionable FDOs, we have shown how standardized profiles, persistent identifiers, and shared registries can enable automated workflows, enhance interoperability across data spaces, and support reproducibility in scientific analysis. The development of a prototypical FDO manager and its implementation in a Jupyter Notebook highlight the practical potential of this approach for real-world use cases. While challenges remain, such as refining FDO specifications, addressing limitations in catalog metadata, and completing integration with external DTR systems, the results underline the promise of FDO-based strategies for building scalable, cross-domain, and interoperable data infrastructures. 7 Kulüke, M., Peters-von Gehlen, K., and Anders, I.: Advancing Climate Data Use by Leveraging the Synergy of the Fair Digital Object Standard and the InterPlanetary File System Protocol, EGU General Assembly 2025, Vienna, Austria, 27 Apr–2 May 2025, EGU25-5788, https://doi.org/10.5194/egusphere-egu25-5788, 2025. 9