D11.3 - Report on the development of scalable scientific workflows with integration of modular data-centric solutions

Report on the development of scalable scientific workflows with integration of modular data-centric solutions Deliverable ID D11.3 Work Package Reference 11 Issue 1.0 Due Date of Deliverable 31/12/2025 Submission Date 16/12/2025 Dissemination Level PU Lead Partner BSC Contributors Marvin Axness Grant Agreement No 101081383 Call ID HORIZON-CL5-2022-D1-02-02 Page 2 of 22 Prepared by Reviewed by Approved by Marvin Axness (BSC) Pierre-Antoine Bretonnière (BSC) Thomas Jung (AWI), Malcolm Roberts (AWI) Issue Date Description Author 1.0 05/12/2025 First issue of the deliverable Marvin Axness (BSC) Page 3 of 22 Index Index 3 Executive Summary 4 1. Workflow 5 1.1. Basic Purpose and Functionality 5 1.2. Jobs 6 1.2.1. Primary jobs 6 1.3. EERIE-specific jobs 7 1.3.1. Model-side specifications 7 1.3.2 Adaptations performed to MultIO and eccodes 8 1.3.3. Setup of model output 8 1.3.4 Model output storage 9 1.3.5. EERIE-specific Autosubmit data reduction jobs 9 1.3.6. Granularity of postprocessing jobs 11 1.3.7. Resource usage in postprocessing jobs 12 2. Core of EERIE Postprocessing 13 2.1. Introducing the challenge 13 2.2. Design of software 14 2.2.1. Input requirements 14 2.2.2. Flow of code 14 3. Data quality checker 16 3.1. Input requirements 17 3.2. Output and checker result report 17 3.3. Checker flow of code 17 4. Transferring of data 18 4.1. Transfer to /esarchive/ 18 4.2. Transfer to Levante 19 5. Portability 19 5.1. Across HPC systems 19 5.2. Across models 19 6. Conclusions and future perspectives 20 Annex 21 Page 4 of 22 Executive Summary This deliverable presents the integration of data-centric solutions into scalable and automated workflows for the EERIE IFS-NEMO simulations. Building on the Climate-DT workflow, an EERIE-specific IFS-NEMO model version generates NetCDF output, which is subsequently joined, CMORized, and structured according to EERIE format tables. The entire process is automated via Autosubmit, from model compilation to dedicated postprocessing tasks and quality checks, ensuring that data are ready for further analysis or ESGF dissemination with minimal manual intervention. This approach enables seamless integration of online diagnostics and postprocessing into model simulations, reduces data transfer overhead, and supports reproducibility and operational efficiency. We also explain in this deliverable how the solutions described for this model and platform is portable to other use cases. Page 5 of 22 1. Workflow 1.1. Basic Purpose and Functionality In EERIE we are taking advantage of the workflow of the Destination Earth Climate Digital Twin, which runs in different EuroHPC machines (Marenostrum 5 and LUMI, for example). The ClimateDT workflow is publicly available at https://github.com/DestinE-Climate-DT/Workflow/ . In EERIE, we are modifying some of the scripts and configurations to take into account the scientific and technical specificities of each workflow. At its core, the workflow orchestrates complex climate simulations by automating model compilation, execution, post-processing, data quality checks, and application workflows. It leverages Autosubmit4 1 to coordinate tasks and supports multiple operational modes: ● Model mode for pure climate model simulations. ● End-to-end mode to run models and applications in a streaming pipeline. ● Applications mode to simulate application processing without running a model. ● Simless mode to process existing data or manage ancillary jobs like backups and transfers. The workflow supports modular customization via YAML configuration files, allowing users to select models, applications, platforms, resolutions, and job-specific settings dynamically . 1 Manubens-Gil, D., Vegas-Regidor, J., Prodhomme, C., Mula-Valls, O., & J. Doblas-Reyes, F. (2016). Seamless management of ensemble climate prediction experiments on HPC platforms. 2016 International Conference on High Performance Computing & Simulation (HPCS), 895–900. https://doi.org/10.1109/HPCSim.2016.7568429 Page 6 of 22 1.2. Jobs 1.2.1. Primary jobs Depending on the workflow mode selected, different primary jobs will be executed. The jobs are what Autosubmit submits to the remote platforms. They are a combination of templates (bash scripts) and the configuration selected by the user. The templates are located in /workflow/templates . ● Local_setup : Performs basic checks as well as compressing the workflow project in order to be sent through the network from the local machine to the HPC. ● Synchronize : Syncs the workflow project with the remote platform. ● Remote_setup: Loads the necessary environment. ● Dvc setup : Initializes DVC repository needed for model inputs. ● Ini : prepares any necessary initial data for the climate model runs. Runs in the login node of the HPC. ● SIM : Main script used to run the different climate models Page 7 of 22 Figure 1. Example of a model simulation workflow with Autosubmit4. 1.3. EERIE-specific jobs 1.3.1. Model-side specifications To define the requirements for additional Autosubmit jobs for handling model output, it is essential to first understand the model specifications. The EERIE IFS-NEMO simulations use a fixed bundle version, DE_CY48R1.0_EERIE_20240726, which relies on modified versions of ECMWF ecCodes 2.35.1 and MultIO 2.1.9. Eccodes is an ECMWF library that handles the encoding of grib files and MultIO is the model I/O library used. These modifications were specifically developed to enable direct NetCDF output from the model. In EERIE experiments, the model resolution is TCo1279 for IFS (the atmospheric component) and eORCA12 for NEMO (the ocean component). Page 8 of 22 Simulations are executed on 125 CPU nodes (including 3 I/O nodes for IFS and 2 I/O nodes for NEMO), achieving an average performance of 1.47 simulated years per day. All runs are managed via Autosubmit in 6-month chunks, facilitating modular execution and postprocessing. 1.3.2 Adaptations performed to MultIO and eccodes When using the official tags, IFS-NEMO by default outputs GRIB messages. To facilitate, automate and speed up the analysis, and data sharing without duplicating data (and therefore save disk storage), we have enabled the direct writing of the data by the model in NetCDF. To enable direct NetCDF output, we adapted an existing ecCodes library, grib_to_netcdf , which is originally designed to handle GRIB files of any size. Furthermore, the model writes GRIB files by sequentially appending GRIB messages, where each message represents a field “quanta”, i.e., a single timestep for a specific field and level. As a result, the NetCDF pipeline produces one NetCDF file per field, timestep, and level. For example, a 6-month simulation chunk generates 6 NetCDF files for monthly surface fields, while 6-hourly pressure-level fields may result in up to ~14,000 files per field. This output structure creates a clear need for a robust postprocessing pipeline capable of combining, merging, CMORizing, and compressing the data in accordance with CF and CMOR conventions and based on the recommendations formulated in deliverable 11.2. 1.3.3. Setup of model output The configuration of IFS-NEMO model output is managed through a set of YAML files, referred to as “plans”. These plans define the sequence of operations applied to each field, including interpolation, computation of statistics (e.g., averages), and encoding. Due to the model-side modifications described above, any change in encoding also requires corresponding updates to the MultIO YAML plans. An example of a MultIO yaml plan for NetCDF output is shown in Figure A-1 in the Annex . Page 9 of 22 To simplify management of the numerous YAML plans, an in-house Python tool called Autoplans was developed. Autoplans provides a single entry point for users to configure the full set of output plans efficiently. The tool leverages Jinja2 templates, allowing YAML files to include placeholders and logic, which enables flexible, automated generation of fully customized configuration files. 1.3.4 Model output storage The total raw model output for a 6-month simulation chunk amounts to 975 GB, distributed as follows: 19 GB for monthly fields, 264 GB for daily output, 592 GB for 6-hourly output, and 102 GB for 3and 1-hourly output, all on a regular 0.25° × 0.25° latitude-longitude grid. This volume corresponds to uncompressed data stored in double-precision format. After the postprocessing steps—which include converting variables from double to float and applying NetCDF compression—the total size is typically reduced by about 60%, depending on the variable characteristics and output frequency, as further described in the following sections. 1.3.5. EERIE-specific Autosubmit data reduction jobs To manage the thousands of individual NetCDF files produced by the model and convert them to CMORand CF-compliant EERIE standards , a dedicated postprocessing infrastructure has been developed. This section provides an overview of the EERIE-specific Autosubmit jobs, with technical details described in later sections. Following the SIM job, the following postprocessing jobs have been added: ● EERIE_POSTPROC : A Python-based workflow that collects and fixes raw model output into fully CMORized NetCDF files. Page 16 of 22 certain IFS-NEMO variables to convert units from the raw model output to those specified in the CMOR tables. Variable data types are changed from double to float, spatial coordinates are renamed according to CMOR standards, and latitudes are reordered from descending to ascending. Vertical atmospheric levels are converted from hectopascal to pascal, and ocean levels are converted to depth in meters. Additional adjustments, such as renaming auxiliary variables and updating attributes, are also applied following the CMOR JSON tables. The final step in the VariableProcessing pipeline is the write function. This is the most computationally intensive step and represents the main bottleneck in the code. Various strategies were tested, including different chunking schemes, engines (netCDF4 and h5netcdf), and writing first to SSD before transferring to GPFS. The largest performance gain, however, was achieved by disabling compression. Although compression uses zlib with a deflate level of 1, compressing large files consumes significant CPU resources and can slow writing by up to seven times. As a result, compression is performed offline using optimized tools such as cdo and nccopy, allowing the main pipeline to run efficiently. 3. Data quality checker The data quality checker was developed as part of a larger Python infrastructure designed to validate and fix files generated from raw model output. The code follows a modular, class-based design, where all checks are derived from a base class called BaseCheck . Some checks are fixable, such as variable attributes, and are implemented via a derived class FixableCheck that includes the corresponding fix method. Other checks, such as AllTimestepsCheck , which verifies that all timesteps are present, are inherently non-fixable. Page 17 of 22 3.1. Input requirements The input requirements are the same as those for the postprocessing code described in Section 2.2.1. An optional flag, fix_issues , can be provided if issues detected by the checker should be automatically corrected. However, within the workflow pipeline, the primary purpose of the software is data quality validation rather than automatic correction. 3.2. Output and checker result report At the end of its execution, the checker produces a summary and detailed report listing all issues and their descriptions. If any issue is detected for a file, an error is logged, and the corresponding Autosubmit task is failed to prevent accidental data removal or propagation of invalid output. 3.3. Checker flow of code An orchestrator class, CheckRunner , manages the execution of the different checks on every file found in the EERIE-structured folder output. Similar to the postprocessing workflow described in Section 2, a series of Descriptor objects is generated to provide metadata for each NetCDF file located in the inner folders. For each FileDescriptor , the full set of checks is executed using a common class method named run . If a check identifies an issue, an Issue object is returned and tracked by a Report object. The checks are categorized into three types: file-quality, NetCDF-quality, and physical-quality checks. Page 18 of 22 File-quality checks focus on file-specific format issues, such as verifying that filenames match the expected conventions, timestamps are correct, and the model name is present in the filename. NetCDF-quality checks are more extensive, addressing issues commonly found in IFS-NEMO raw model output data. These include verifying variable types, dimensions, attributes, compression, time bounds, and other metadata requirements. For more details on the data standards that NetCDF files must follow, see EERIE Deliverable D3.2 Publication of Phase 1 simulations . Physical-quality checks ensure that the data values and units are physically consistent. Some checks, such as temperature ranges, are straightforward, while others cannot be generalized and require case-specific validation. Finally, it is worth noting that the FixManager object, which orchestrates the application of fixes to the data, is not described here. The main difference compared to the postprocessing workflow in Section 2 is that FixManager uses the lower-level Python module netCDF4 to modify datasets and rewrite them when higher-complexity fixes are required. 4. Transferring of data 4.1. Transfer to /esarchive/ The model simulations are performed on MareNostrum5, with initial data stored on its GPFS filesystem. Given the large volume of model output and storage requirements—approximately 2 TB per simulation chunk before compression—the full dataset cannot be retained on GPFS. Therefore, the data must be transferred to the permanent storage facility at BSC, located at /esarchive . Currently, these transfers are performed manually using dedicated data-transfer nodes. Page 19 of 22 4.2. Transfer to Levante Additional EERIE partners analyze the simulation data on the Levante system at DKRZ. Consequently, the simulation output is also transferred to Levante offline to provide access for collaborative analysis. 5. Portability 5.1. Across HPC systems The Autosubmit climate DT workflow has been successfully deployed on multiple HPC ecosystems, including Marenostrum 5, and LUMI 3 . The workflow orchestration itself requires a virtual machine that can access the platforms where the different parts of the workflow run. Autosubmit is installable through conda or pip, as described in the documentation 4 . The postprocessing pipeline and the data quality checker are distributed within a Singularity container, which allows for straightforward deployment across different HPC environments. The container includes Python (version 3.11) along with other required libraries like xarray (version 2025.1.2) and netCDF4 (version 1.7.2). 5.2. Across models The climateDT workflow has been used with other climate models such as IFS-FESOM and ICON. Integrating additional models is also possible and it consists of adding model-specific run scripts to the workflow jobs described in Section 1.2.1. Regarding the postprocessing pipeline described in Section 2, certain steps, such as merging single NetCDF files, are currently specific to the IFS-NEMO model version. However, the file rebuilding phase, which creates the CMOR-compliant NetCDF outputs, is model-agnostic and can be applied to any type of NetCDF output. 4 https://autosubmit.readthedocs.io/en/master/installation/index.html 3 https://lumi-supercomputer.eu/ Page 20 of 22 The data quality checker is also reusable across different models, provided that the model outputs follow the EERIE-specific output folder structure. While some checks are project-specific (because they depend on the EERIE CMOR tables or some internal conventions regarding for example the ordering of the latitudes and longitudes), the modular design allows for easy extension to accommodate new models or additional verification criteria. The code is available in the BSC gitlab and can be made accessible on demand. The recipes of the containers follow the same approach. They are currently stored in Marenostrum 5 but can be shared. 6. Conclusions and future perspectives The full automation and modularization of the workflow and data processing described in this deliverable shows, for one model, solutions to enhance and facilitate the data management by reducing the time-to-solution for the scientific analysis. Even if not described in this deliverable, the diagnostics for monitoring the simulation described in D11.2 “Report on the design and implementation of atmosphere, ocean and sea ice diagnostics, including their containerization”, including the containerized AQUA diagnostics, will also be integrated in the workflow of the coming EERIE IFS-NEMO simulation, using the data preparation and reduction described in this deliverable. In addition, the workflow orchestration can easily be adapted to other models or infrastructure. In particular, in the framework of linking the EERIE and EDITO projects, some reduced versions or configurations of the EERIE workflow could be run in the EDITO infrastructure. Page 21 of 22 Annex Figure A-1. Example of a MultIO plan for EERIE netcdf output. Page 22 of 22 Funding This project has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101081383 All UK Partners in EERIE are funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee (grant numbers 10057890, 10049639, 10040510, 10040984). ETH Zürich’s contribution to EERIE is funded by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract #22.00366.