D7.1 Data Infastructure Setup

1 Document information Grant agreement 101137656 Project title Underpinning infrastructure for the efficient and flexible analysis of large climate datasets Project acronym EXPECT Project start date 1 April 2024 Related work package WP7 Related task(s) 7.1, 7.2, 7.3 Lead organisation DKRZ Authors Stephan Kindermann, Daniel Fulla, … Submission date 30 September 2025 Dissemination level PU (Public) History Date Submited by Reviewed by Notes 17/07/2025 First draft, S. Kindermann, D. Fulla First draft 29/08/2025 Stephan Kindermann, Daniel Fulla, Pierre Antoine Bretonniere, Marco Puccini, Serena Lorenzini To be reviewed by Expect Theme 4 With comments to be addressed 08/08/2025 Theme4 Christian Lessig, Bernd Funke Version submitted to external reviewers 2 30/09/2024 Daniel Fulla et. al. Christian Lessig, Bernd Funke Final version with comments from reviewers addressed Disclaimer: Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them. 3 Table of content About 5 Executive summary 6 Requirements on infrastructure from partners’ feedback 7 EXPECT Infrastructure 8 Data Storage and Transfer 9 Data Storage and transfer at DKRZ 10 Data storage and transfer at CINECA 12 Data Processing 16 Data Processing at DKRZ 16 Data processing at CINECA 19 Expect Catalog (of Catalogs) 22 Data analysis workflows 24 Summary 26 Future Work 27 Appendix 28 List of tables Table 1 6 Table 2 6 List of figures Figure 1 EXPECT Infrastructure Overview Figure 2 Cloud data storage setup Figure 3 JupyterHub at DKRZ Figure 4 Multi-user Dask gateway implementation Figure 5 Dask data processing and diagnostics Figure 6 The EXPECT STAC catalog of catalogs Figure 7 Distributed data analysis support 4 About The climate system is changing rapidly and some regions have seen increases in extremes beyond what is expected from climate model simulations. To support targeted climate adaptation strategies, EXPECT will enable trustworthy assessments and predictions of regional climate change including extremes by developing a prototype operational capability for integrated attribution and prediction of climate. This ambitious goal is closely aligned with the WCRP Lighthouse Activity on Explaining and Predicting Earth System Change. EXPECT will identify and quantify the mechanisms by which physical processes govern regional climatic changes, including extremes, on inter-annual to multi-decadal time scales. It will do so by exploiting newly available climate simulations and Earth Observations (EOs), and by combining machine learning (ML) with physical methods. The research will target fundamental knowledge gaps related to atmospheric circulation and land-atmosphere interactions, which represent major limitations in current climate predictions and projections, and in particular in understanding changes in European summer extremes. To underpin the research, and benefitting the wider research community, EXPECT will develop tools to efficiently analyze a variety of large data sets in combination that are hosted in different repositories across institutions. This will facilitate the exploitation of recent investments into high-resolution climate models and E O data. EXPECT will further build data science capacity for the scientifically robust, efficient and reproducible analysis of the massive data assets, including novel ML approaches, and provide training for the climate science community and the next generation of researchers in particular. EXPECT will thus deliver significant scientific and technological advances for society and the climate science community that will last well beyond the project, in support of WCRP’s strategic objectives. 5 Executive summary This deliverable reports work done as part of work package 7 which establishes the foundational infrastructure for distributed data analytics within the EXPECT project. The goal is to enable future cross-institutional, scalable and FAIR data workflows that support diverse scientific goals of EXPECT. The objectives focus on integrating distributed compute and storage resources, enabling data ingestion from key climate datasets, and developing unified catalogues and tools for efficient, location-aware analysis. Over the first 18 months, progress has been made in gathering partner requirements, tracking dataset usage, and integrating institutional infrastructures across major European centers such as DKRZ and CINECA. A first version of processing services is available for user testing and interactive computing platforms such as JupyterHub have been deployed to support user workflows. A first version of an integrated data catalogue system (integrating externally managed catalogs like ESGF) is in place. This deliverable presents an overview of the status of the integration of infrastructure components enabling workflows that meet user needs and enable future cross-site interoperability. It also outlines planned enhancements to improve data staging, federated catalogues, and hybrid cloud-HPC processing models. These developments contribute to a growing ecosystem of European climate data infrastructure, complementing and building upon initiatives such as ESGF, Copernicus, and Pangeo. The EXPECT project’s distributed approach aligns with current EU priorities for FAIR data and scalable, collaborative research environments. The technical solutions proposed in this deliverable are based on the requirements expressed by the partners of the consortium in the context of task 7.1 whose conclusions were presented in milestone M10 “Collecting requirements for shared data infrastructure and distributed processing (architecture, services)” delivered at M12. 6 Requirements on infrastructure from partners’ feedback In M10, available on the project wiki1, the users were asked about the datasets they were using and planning to use during the project, considering their volume, where they were planning to access the data and with which tools, programming languages, and protocols. The main conclusions and how they have been taken into account in the design of the infrastructure are described in this document. The principal datasets that people are planning to use come from ESGF (CMIP6, LSFMIP), the Copernicus Datastore Store (CDS) or MARS (ERA5 or mars seasonal forecasts) and some other external High Resolution Climate models data (Climate Digital Twin DestinE, EERIE, NextGEMS). CMIP6 and ERA5, which are available in Jasmin2 and DKRZ are the most requested datasets. Users either download the data from the CDS3 and ESGF (~60%) and MARS4 (~25%) to their local machines, or take advantage of the prefetched data from institutional repositories such as CEDA, DKRZ, BSC. The data access (bandwidth and complexity) is the main driver to choose where and how to run the analysis. This clearly indicates the usefulness of proposing the CINECA computing platform in addition to the BADC and DKRZ ones already mentioned. The currently prevalent approach of downloading the data to local disk for analysis also shows the importance and future benefits in terms of performance of the infrastructure described below. Through its online processing capacities, it is expected to significantly increase performance of the analysis, avoiding the overhead of the download. To support effective collaboration between infrastructure partners (BSC, DKRZ, UREAD, CINECA), the project organizes regular coordination meetings both within and across infrastructure work packages, as well as broader project meetings such as the Theme 4 kickoff, Executive Board meetings, and the EXPECT General Assembly. Bilateral meetings between specific partners, for example, DKRZ and CINECA, are organized whenever closer alignment is required. Collaborative development can further be supported through shared code repositories hosted on the DKRZ and BSCGitLab platforms, ensuring transparent version control and facilitating streamlined contributions across all teams. 4 Meteorological Archival and Retrieval System 3 Climate Data Store 2 High-performance data analysis and storage platform developed and operated by the UK Centre for Environmental Data Analysis (CEDA) 1 https://earth.bsc.es/expect/lib/exe/fetch.php?media=wiki:content:expect_user_survey-m10.pdf 7 EXPECT Infrastructure An overview of the distributed infrastructure supporting climate analysis workflows in EXPECT is provided in Figure 1. The infrastructure is initially built around the three core data centers DKRZ, BSC and CINECA. DKRZ and CINECA provide storage and compute resources together with a range of associated data and compute services. These will be progressively integrated and extended as part of EXPECT. BSC complements these resources by hosting additional internal data servers. It also acts as a bridge to the evolving DestinE data infrastructure. The data storage, transport and compute services available at these sites which need to be integrated to enable cross-institutional workflows are summarized in the following subsections. Figure 1 EXPECT Infrastructure Overview illustrates the distributed infrastructure connecting DKRZ, CINECA, and BSC, highlighting their complementary storage, compute, and data services. To exploit DKRZ and CINECA resources as part of EXPECT workflows, scientists need to register at the sites and get access rights to resources and services: 8 To get an account at DKRZ and join the EXPECT project (project code: bk1444), users should first obtain a DKRZ user account by registering through the official portal. Once the account is approved, they should log in to the DKRZ user portal (LUV) and submit a request to join the EXPECT project via the project application interface. At CINECA, users should begin by registering in the User Database (UserDB). After completing the UserDB registration and receiving approval from CINECA, users may have access to the EXPECT project systems hosted there. With an account at either of the computer centers, users can access data storage, transfer data between sites, and use data processing capabilities. Work towards integration into a federated AAI5 infrastructure (e.g. based on the EGI AAI6 and aligned with EOSC AAI architecture efforts7) is not part of EXPECT, yet efforts are underway as part of other projects which involve EXPECT partners (especially DKRZ and BADC8/UKRI9 e.g. in RI-SCALE10). BSC provides additional infrastructure for standardized storage on disk and tapes, available exclusively to internal users only. This infrastructure supports datasets such as CMIP6, CDS seasonal forecasts, and ERA5. Additionally a dedicated workflow has been implemented to analyze DestinE data stored on the MareNostrum5 data bridge. Data Storage and Transfer Climate data analysis activities involve high volume data collections spread over distributed and different types of storage systems. On the one hand there is a need for fast, high-throughput storage access (e.g. based on disk or ssd hardware and associated high performance file systems) which are associated with (e.g. HPC) compute resources. On the other hand there is the need to temporarily share, transfer and store large amounts of data, which require automatable data transfer steps and associated data sharing systems e.g. based on cloud storage. Additionally there is the need to interact with long term archived data and thus access long term preserved data residing on e.g. tape-based storage systems. 10 https://www.riscale.eu/ 9 UK Research and Innovation 8 British Atmospheric Data Centre 7 EOSC AAI architecture task force: https://eosc.eu/advisory-groups/aai-architecture/ 6 EGI AAI: https://www.egi.eu/service/check-in-internal/ 5 Authentication and Authorization Infrastructure 9 Overview of data transfer options To govern data transfer between devices over a network, one or more standardised protocols needs to be used. These protocols specify standardised rules and formats for data transfer and vary in terms of speed, security and complexity. The most suitable protocol will be chosen depending on the specific use within the project. To move data to/from HPC systems, Cineca offers dedicated data transfer services that broadly fall into two categories: data movers and GridFTP. Data movers are dedicated, containerised nodes without interactive access that support only a limited set of commands (scp, rsync, sftp, wget, curl, rclone, AWS S3 and S3). GridFTP is also available on these nodes, but it can only be used via the globus-url-copy client, which must be run from the user's local machine. Official documentation details the data transfer services (https://docs.hpc.cineca.it/hpc/hpc_data_storage.html). Data Processing Data processing within the EXPECT project is progressing through implementations at DKRZ and CINECA, where dedicated platforms are being developed to support scalable, efficient, and reproducible workflows for climate data analysis. At DKRZ, the Levante HPC system is configured to enable both interactive and batch processing, using tools such as JupyterHub, Slurm, and the Rook Web Processing Service. CINECA has deployed a cloud-native infrastructure based on Kubernetes, JupyterHub, and Dask, allowing for flexible, distributed data processing. These platforms can already support EXPECT use cases, and further work is ongoing to expand functionality and improve interoperability. Data Processing at DKRZ DKRZ offers HPC resources tailored for climate modeling and data-intensive workflows, with CPU-only, GPU-enabled, and interactive nodes. Storage relies on a high-performance Lustre-based file system for fast I/O. Typical workflows include data 16 preparation (using Python, CDO33, NCO34), batch simulations via Slurm, and postprocessing with data reduction and visualization. Automation through workflow managers and scripting enables efficient, reproducible research. Batch jobs run on Slurm, where users submit scripts specifying resources and commands. This non-interactive mode suits large, parallel simulations using MPI35 or multi-threading. Interactive data processing (jupyterhub) DKRZ’s JupyterHub provides an interactive, web-based environment on the Levante HPC system, enabling scalable, data-near analysis of large climate datasets like CMIP6 and ERA5. Users authenticate with DKRZ credentials, select resource profiles via Slurm, and launch sessions supporting Python, R, and Julia. Containerized environments and custom kernels allow reproducible workflows and advanced analyses. For EXPECT, DKRZ’s Jupyter notebooks repository offers hands-on tutorials and use cases for model data analysis. Notebooks demonstrate optimal HPC usage, processing tools, and data visualization. Users can clone the repository directly in JupyterHub, run demo notebooks, or create tailored kernels via conda/mamba for advanced workflows. 35 Message Passing Interface 34 NetCDF Operators 33 Climate Data Operators 17 Fig 3 JupyterHub at DKRZ. The figure shows how DKRZ’s JupyterHub service connects researchers to the Levante HPC system. Users log in through Apache with LDAP authentication, after which the JupyterHub hub manages sessions and routes requests via CHP. Resource allocation is handled by the DKRZSpawner, which offers both advanced and preset options and submits jobs to Slurm. Once scheduled, sessions run on Levante compute nodes, providing interactive environments. 18 Web Processing Service Rook is a web-based climate data processing service developed under the ROOCS36 project by DKRZ (Germany), CEDA/STFC (UK) and Ouranos (Canada). It offers standardized access to climate data processing via a Web Processing Service (WPS) API, enabling distributed workflows across diverse storage systems. Built on GeoPython tools, including pywps37 (OGC WPS implementation38), clisops39 (xarray-based climate data operations), rooki (a Python client for Jupyter integration), and daops40 (data operation support), Rook facilitates processing close to the data, reducing large transfers and supporting FAIR principles. Within EXPECT, Rook enables distributed, standards-based access to climate datasets and preprocessing services and integrates with interactive environments like Jupyter notebooks. Deployments include a permanent joint DKRZ & IPSL instance, a standalone DKRZ node integrated with ESGF and The ENES Climate4Impact41 portal, and a CINECA installation optimized for S3-based HPC workflows. These run on virtual machines accessing local or cloud storage, enabling scalable and flexible processing. For example, the Copernicus Climate Data Store (CDS) uses a Rook adapter on a DKRZ VM to access CMIP6 and CORDEX datasets, complementing Copernicus’ observational focus by providing model data processing with tools like regridding to harmonize spatial resolutions. Rook integrates with federated catalogs such as ESGF, Copernicus CDS, STAC, and Pangeo standards. However, its WPS interface currently lacks full compatibility with ESGF S3 storage instances. Regridding via clisops addresses challenges in grid inconsistencies and interpolation practices, offering a reproducible, programmable interface despite some community-level issues. For hands-on use, the public WPS endpoint is rook.dkrz.de, with demo notebooks available at the rooki GitHub repository. Notably, the CMIP6 subsetting notebook 41 https://www.climate4impact.eu/c4i-frontend/ 40 https://pywps.readthedocs.io/en/latest/ 39 Climate Simulation Operations 38 Open Geospatial Consortium’s Web Processing Service specification 37 PyWPS Web Processing Service implementation: https://pywps.org/ 36 Remote Operations On Climate Simulations 19 supports time and bounding-box queries, and the time-components notebook demonstrates advanced subsetting and workflow examples. Data processing at CINECA The data processing setup involved deploying an easy-to-use, multi-user platform for processing data. The following details are from the testing and validation performed up to M18. Overview of data processing components The platform consists of several components deployed over K8s, including a user authentication mechanism, a JupyterHub interface and a dask gateway, as depicted in the following figure. Access to processing tools is currently restricted to authorised users (currently only internal staff) to ensure full access control and to avoid security breaches. Figure 4: Multi-user Dask gateway implementation The authentication has been enabled using X container/tool. The JupyterHub (vide infra) has been deployed with X container/Helm chart that allows multiple users to run their Jupyter notebooks directly on cloud infrastructure where the data is stored, enabling data-proximate computation. The third component of the data processing setup is the deployment of Dask-Gateway. This is a powerful tool for managing and scaling Dask clusters. It does this in a secure, multi-tenant environment. This makes it ideal for distributed data processing in shared infrastructure like Kubernetes or HPC systems. It enables users to launch and connect to Dask clusters dynamically on demand while giving administrators control over resource limits and authentication. By separating the user interface from back-end cluster management, Dask-Gateway simplifies complex data workflows and enables efficient parallel computing across large datasets. 20 The deployment has been tested to address the several issues that includes; ensuring that the JupyterHub service exposes the cluster service through correct API endpoints, ensuring the compatibility of Dask and Dask-Gateway versions by avoiding outdated default images, enabling resource cleanup when the user shuts down the Dask cluster or closes the notebook server, enforcing user isolation via different namespaces to prevent two users from accessing the same Dask cluster, and examining the secure execution of Dask workflows. Interactive data processing (jupyterhub) JupyterHub is a multi-user orchestration layer that can be used to spawn and manage Jupyter notebook servers on shared infrastructure. Typically backed by Docker, Kubernetes, or other spawner backends, it provides centralised authentication, user session isolation, and resource management. It also supports integration with OAuth, LDAP, and other identity providers, making it suitable for secure environments. When deployed with Kubernetes, it enables scalable, containerised notebook environments for each user with configurable resource quotas and persistent storage. It is often the de facto choice for research platforms providing interactive computing environments with reproducible workflows. JupyterHub can be effectively combined with Dask to provide computing resources and real-time performance monitoring via Dask diagnostics dashboards. When users run distributed Dask computations in their Jupyter notebooks within JupyterHub, Dask automatically launches a diagnostics dashboard providing live insights into task scheduling, memory usage and workload distribution. The following figure shows a screenshot of the typical diagnostics dashboard. Figure 5: Dask data processing and diagnostics 21 Each user can then monitor their own computations via a link to the Dask dashboard. This is typically hosted on a dedicated port and can be securely proxied through JupyterHub. This integration is particularly beneficial in data-intensive environments, where performance visibility and scalability are essential. The implementation was tested to ensure the proper functioning of the dashboard, the isolation of individual user dashboards, and the possibility of exposing the dashboards to separate windows for visualisation convenience. Web Processing service The Rook-WPS service is part of a larger suite of tools known as ROOCS (Remote Operations on Climate Simulations). ROOCS implements a data-aware paradigm, which means it enhances data storage systems with the capability to analyze, process, and aggregate data directly where it resides. This approach allows for efficient processing of ESGF data by bringing the computation to the data, rather than moving large datasets. The Rook-WPS service requires access to the same publicly exposed data made available through ESGF. Since the service is containerized, it can be hosted on a simple server or a virtual machine, as long as it can mount the volume where the data is stored. This requirement, however, created a significant challenge for CINECA. The organization has updated its ESGF software stack to a new Kubernetes-based version that primarily uses S3 storage. This led to an extensive period of configuration and testing, as CINECA attempted to either use Rook directly with the S3 data or find a suitable workaround. The only viable solution at present is to have an ESGF dataset replicated on both S3 storage (for publication via the new services) and a standard file system that can be mounted on the virtual machine hosting the Rook-WPS service. Coincidentally, the OptimESM project data, due to its own specific project requirements, already possesses this dual-storage characteristic. Therefore, the next steps will focus on the configuration and deployment of the Rook-WPS service using this particular dataset. Expect Catalog (of Catalogs) 22 Data collections which are used in EXPECT data analysis workflows are cataloged in different heterogeneous systems. However, currently a strong community effort focuses on harmonizing cataloging approaches based on Spatio-Temporal Asset Catalogs42 (STAC) and the evolving associated STAC tooling ecosystem43. Thus, the newly developed Expect Catalog is based on STAC. It integrates external STAC catalogs but also deploys a dynamic STAC backend for additional data accessible at EXPECT partner sites. The basic structure is illustrated in Figure 6. Fig 6. The EXPECT STAC catalog of catalogs To be able to use tools from the rapidly evolving STAC tool ecosystem, the EXPECT catalog of catalogs can be browsed (with the the stac-browser tool44) based on a 44 Stac-browser: https://github.com/radiantearth/stac-browser 43 STAC tools & resources: https://stacspec.org/en/about/tools-resources/ 42 STAC Spatio Temporal Asset Catalogs https://stacspec.org/en 23 centrally managed json description45. This stac-browser view is available at https://discover.dkrz.de. This catalog of catalogs currently includes the following sub-catalogs: - DKRZ / EXPECT STAC catalog: hosted at DKRZ, exposing climate model data from the following projects/efforts: - EERIE data collections - Warmworld data collections - External STAC catalogs - ESGF (east) STAC catalog - Copernicus catalog - The DestinE data lake catalog The EXPECT catalog of catalogs will be adapted, extended and refined during the project lifetime: ● The characterization of catalogs in the catalog of catalogs is currently based on metadata from the registry of research data repositories (re3data.org) effort. This characterization is quite generic and will be refined to include specific metadata useful to exploit climate model data repositories (e.g. providing information on the specific controlled vocabularies used to support data search). ● As mentioned before, many parallel efforts are ongoing to catalog climate data collections based on STAC and provide STAC APIs and catalogs. The EXPECT catalog of catalogs will follow these developments and associated ongoing harmonization efforts. For example, the STAC catalog that is currently integrated to provide access to data collections from the global ESGF data federation (e.g. CMIP6 and CORDEX) is based on the STAC catalog hosted at CEDA UK. It will be superseded by a STAC catalog which is fully synchronized with the ESGF (west) STAC catalog hosted in the US and will go operational to support CMIP7 later this year. ● There are also currently many early efforts to enable an automatic generation of static (as well as dynamic) STAC catalogs based on (e.g. temporary) data collections that are hosted on disk (and cloud) resources, associated metadata conventions, and controlled vocabularies. Different tooling support is currently under development, community wide (e.g. for 45 EXPECT catalog of catalogs: https://tinyurl.com/ExpectCatalog 24 ESGF data publication) as well as at institutional level (DKRZ disk and tape data STAC catalog generators). Currently it is too early to decide on specific tool suites which can be proposed for e.g. EXPECT data providers wanting to register data in the EXPECT data catalog. Such data registration would support climate data sharing between climate data centers (and research institutes). ● To be able to exploit the EXPECT STAC catalog to support distributed climate analytics workflows as planned in workpackage 9, specific low level data access information will be included in the EXPECT catalog. This is necessary for users to have information on how to best access the information based e.g. on xarray and dask, as well as the low-level technical details necessary to configure client side tooling. For example, the exploitation of data collections based on STAC catalog entries is currently not as user friendly as those based on static intake catalogs46, available for some data collections. The current DKRZ STAC catalog thus also includes information on data exploitation based on intake client side tooling. (see e.g. DKRZ static intake-esm data catalog47) Data analysis workflows A key objective of the work in WP7 was to build the infrastructure foundation on which future work in WP9 can build upon to support distributed data analysis workflows involving high volume climate data collections in EXPECT. In Figure 7, a simple representative workflow example is provided, illustrating the necessary integration of the different infrastructure components that are summarized in the previous sections: ● Users interact with a centralized catalog to browse and search for the available data sets they need for the planned data analysis activity . This STAC catalog provides an overview of the data accessible at EXPECT HPC sites as well as via external data providers (e.g. Copernicus and ESGF). This is illustrated by the green lines in Figure 7. 47 https://discover.dkrz.de/external/stac-dev-a722d9.gitlab-pages.dkrz.de/static/DKRZ-static-int ake-esm.json?.language=de 46 Intake: Taking the pain out of data access and distribution: https://intake.readthedocs.io/en/latest/index.html 25