Data Access for km-scale resolution models

What happened? Resolution exploded Snapshots from the CMIP MPI-ESM HR, and a 2.5 km resolution ICON simulation, as performed in the context of nextGEMS. With the advent of km-scale models, the grid resolution started to vastly exceed the resolutions of even the best screens. This means there is no expectation of showing a whole 2D field in a plot at full resolution. Previous workflows were used to averaging in the time dimension, but keep the full spatial resolution. With the new datasets, also averaging in the spatial dimension becomes a plausible option before performing global analyses. While for local studies, it becomes important to be able to load only a specific region of a dataset. Datasets became messy At the classic resolutions, model output for one simulation step could be more-or-less grouped into one netCDF file. With the vast amounts of data related to even just one 3D field, this approach stopped working and datasets started consisting of several different files even just for one simulation step. This put the burden on the user to figure out which files to combine for an analysis. By forming big, coherent datasets containing all variables written by the model, we take this burden off the user, and force a coherent output configuration. We provide different resolutions of this dataset in space and time to allow fast global analyses and prototyping as well as fine-grained analysis of individual features of the climate system. Data got buried Typical paths to model output look like Get an account on this computer if you can. Go to “/work/my_project/some_user/ model_version_23/experiment_42/”, and then take all files from the “output_YEAR” directories that have “3D_pl_7_” in their filenames. By listing datasets in catalogs, we make the datasets findable again. By providing them as Zarr via https, we even make them accessible. This includes the option of accessing the data without first jumping through hoops to get access to a foreign HPC, as well as providing it in small chunks, that can easily be loaded via an internet connection. Data Access for km-scale resolution models Florian Ziemen1, Tobias Kölling2, Lukas Kluft2 1German Climate Computing Center (DKRZ), 2Max Planck Institute for Meteorology Why HEALPix? Chose your grid for analysis performance Efficient subsetting of datasets requires a data layout that is locally coherent. Furthermore, a direct translation of lat/lon to grid indices speeds up subsetting. Two grids that can fulfill these requirements are the classic lat/lon grid, and the Hierarchical EqualArea isoLatitude Pixelation (HEALPix) grid (Górski et al., 2022). lat/lon is impractical for high resolutions as the longitude lines converge at the poles. HEALPix is hierarchical The HEALPix grid is hierarchical. This makes it very easy to build a whole set of resolutions from a single dataset. When increasing the resolution by one step, each cell is subdivided into four smaller cells. This is done again and again to reach the desired resolution. HEALPix indices in the nested layout. Each of the 12 primary cells of the grid is subdivided into 4 smaller cells, that are divided into 4 cells again and again and again, until the desired resolution is reached. The numbering then follows this hierarchy. HEALPix is equal area All cells in the HEALPix grid have the same size, so taking the mean of a field across multiple cells is simply taking the mean of the values. Also, all regions of the globe are covered equally well with this grid, a strong contrast to lat/lon grids, that converge at the poles and spend a lot of pixels describing a very limited area there. HEALPix is isolatitude The pixels in HEALPix are aligned along isolatitude bands, making it very easy to compute a zonal mean. HEALPix is nobody’s grid While this sounds like a bad sign, it actually can be of advantage in political decisions like common grids in model intercomparisons, or projects like DestinE. Everybody has to adjust to the new grid, and no team is the winner or the loser. Why Zarr? Zarr scales Zarr assumes a dataset to be spread over a multitude of files, but still provides a coherent view on the whole dataset. This view can be obtained by loading one simple JSON file containing the whole description and layout. With this layout, a dataset of a hundred variables and 500 TB spanning thousands to millions of files can be loaded in milliseconds by reading the JSON file. Zarr is made for networks The whole description of a dataset is contained in one JSON file, and data chunks are stored in individual small files, easing network access and caching of datasets and arbitrary subsets. https is a standard backend protocol, and others can be easily implemented as no special features are necessary. Zarr is simple The Zarr format is simple with all metadata being stored in plain JSON files, and data chunks stored in independent binary files. Thus, parallel writes can simply be performed by having different chunks written by different processes without overhead for coordination. Editing the metadata can be done in a text editor, and the metadata files can be even be version controlled. Zarr is supported As Zarr is very simple, implementations in various languages already exist and/or are under development. Zarr as an interface As Zarr is simple and versatile, in can be used as an interface for presenting datasets stored in other formats. In python, kerchunk and fsspec can be used to index HDF5 and other datasets, and present them as zarr. By combining this with a python web server, any language that supports Zarr via https can access the datasets as if they were stored as Zarr. Zarr makes logging easy When routing data access through servers, it becomes trivial to track usage patterns and gain insights that can be used to further optimize data storage and handling. Similarly, even with the pure disk-based access, the access times of the individual zarr chunks can teach us a lot about data access patterns. Why catalogs? Share data and document your work While we can usually remember the simulations we performed in the last week, sharing data requires descriptions that can be found and read by others. Catalogs are the way to collect this information in humanand machine-readable form. The hierarchy of storing data from the individual chunk to the catalog. Describe datasets Descriptions of the datasets can provide users with information relevant for analysis of simulation output or observation data. In addition, catalogs contain references to the data, possibly in various formats or locations. These references are machine-readable, so it’s easy to load data e.g. from a python script or the command line without knowing its location on disk. Group datasets Catalogs provide a hierarchy that positions a dataset within a simulation belonging to a project, etc., This makes it easy to (a) find a dataset and (b) find related products. References Krzysztof M. Górski, Benjamin D. Wandelt, Eric Hivon, Frode K. Hansen, and Anthony J. Banday: The HEALPix Primer, https://healpix.jpl.nasa.gov/pdf/intro.pdf Acknowledgments nextGEMS is funded through the European Union’s Horizon 2020 research and innovation program under the grant agreement number 101003470. This work used resources of the Deutsches Klimarechen zentrum (DKRZ) granted by its Scientific Steering Committee (WLA) under project IDs bb1153 and bm1235. Where can I learn more? Join the global hackathon What: Analyze storm resolving models When: 12-17 May 2025 Where: 11 regional nodes worldwide By: WCRP Digital Earths Lighthouse Activity More: https://digital-earths-global-hackathon.github.io/ Play with the data import intake import easygems.healpix as egh cat = intake.open_catalog("https://data.nextgems-h2020.eu/online.yaml") ds = cat["ICON.ngc4008"](zoom=7).to_dask() tas = ds["tas"].sel(time="2024-12-11") egh.healpix_show(tas, cmap="inferno") A simple surface air temperature plot based on a storm-resolving simulation plotted within seconds on a standard laptop. The outputs of the ICON nextGEMS Cycle 3 and 4 simulations are available via the nextGEMS online catalog. Further simulations from EERIE are available via https://eerie.cloud.dkrz.de Visit easy.gems.dkrz.de Here we collect descriptions of storm-resolving experiments and instructions for analyzing and plotting them. The experiments include the DYAMOND Summer and Winter intercomparisons as well as nextGEMS and EERIE experiments. Instructions cover topics like working with the intake catalogs at DKRZ, the ICON grid, the HEALPix grid, and examples of efficient plot scripts. The whole page is user-driven. You are invited to contribute your own examples or improve existing ones.