Metrics to quantify performance, energy and CO₂ footprint of CMIP7 simulations

1 Metrics to quantify performance, energy and CO₂ footprint of CMIP7 simulations. Instructions for Modelling Centres, prepared by the Energy Consumption and Carbon Footprint Task Team. Table of Contents BACKGROUND 1 TARGETED CMIP7SIMULATIONS 2 COLLECTION, TIMELINE, AND DISSEMINATION OF THE METRICS 3 TIMELINE 3 DISSEMINATION 4 THE METRICS 4 GENERAL INFORMATION 4 TIER 1(MANDATORY) 5 TIER 2(ENCOURAGED) 6 TIER 3(OPTIONAL) 6 ANNEX 9 COMPUTING THE NUMBER OF GRID POINTS 9 ENERGY PROFILING 9 Facilitating energy profiling 11 Background Climate simulations are becoming increasingly complex, with each CMIP phase introducing more sophisticated models, a greater number of experiments, and larger ensembles, leading to a continuous rise in computational requirements. These simulations rely on some of the world’s most powerful supercomputers, which consume energy to reproduce the Earth’s climate. Yet, efforts to systematically quantify the computational cost, energy use, and environmental impact of CMIP activities have so far been limited. The Energy Consumption and Carbon Footprint Task Team was established to address this gap by measuring and reporting these aspects for the upcoming CMIP phase, CMIP7. 2 The Computational Performance for Model Intercomparison Project (CPMIP) defines a standardised set of metrics for evaluating Earth System Models (ESMs) and their execution on high-performance computing (HPC) platforms. These metrics were originally described in Balaji et al. (2017) and provide a framework to characterise climate model experiments in a consistent and comparable way. Most of the CPMIP metrics relate to runtime performance (e.g., throughput, computational cost, coupling, I/O) for a given model configuration (e.g., resolution, complexity, parallelisation). For CMIP7, participating modelling centres are now asked to report energy consumption, which enables the estimation of the carbon footprint associated with running climate model experiments. Modelling centres participating in CMIP7 are asked to provide these metrics for the experiments they are running for the Assessment Fast Track (AFT) and community MIPs. Similar work was conducted for CMIP6 (M.C. Acosta et al., 2024), providing valuable lessons for this next phase. While many activities contribute to the overall CO₂ emissions of CMIP projects including running simulations on HPC platforms, storing and transferring data, ESGF replication, analysis, travelling and commuting for project meetings, and managing the life cycle of computing infrastructure, the current effort focuses only on emissions directly related to computing the simulations. Through the inclusion of the Power Usage Effectiveness (PUE) factor, the estimates will also account for the energy overheads of the HPC facility, such as cooling, networking, and power distribution. However, data storage, data transfers, and other offline processes are not included in this phase of the analysis. The emissions from computing the simulations, together with the infrastructure overheads captured via PUE, are expected to represent a substantial share of CMIP’s total energy use. These are areas where CPMIP metrics are most relevant and where modelling centres can consistently provide data. Ongoing work aims to incorporate other emission sources in future assessments. To promote consistent and widespread collection of experiment-level metrics by modelling centres in CMIP7, they have been organised into three tiers, reflecting both the effort required to obtain them and their analytical value. -Tier 1 metrics contain the minimum mandatory information required to assess the energy consumption, computational cost, and carbon footprint of the experiments. -Tier 2 metrics are strongly recommended and provide the additional information to analyse and interpret Tier 1 results, including basic model configuration and performance indicators to facilitate comparison across institutions and HPC machines. 3 -Tier 3 metrics are optional but encouraged, offering a deeper understanding of model and system behaviour through more detailed performance indicators, which help to identify bottlenecks and improve cross-platform comparability. In addition to these three-tier metrics requested for each experiment, general information will also be asked regarding the centre providing the data, the machine used, and the energy source. These additional metrics are defined at the modelling group or platform level and therefore will only need to be provided once. Following the previous approach, part of this information will be mandatory to ensure a minimum level of analysis, while other elements will be recommended or optional. Targeted CMIP7 simulations We propose to collect metrics across three categories of simulations carried out by modelling centres that participate in the CMIP7 Assessment Fast Track and community MIPs: 1. Production simulations: Runs that generated datasets for scientific analysis, directly linked to one CMIP7 experiment, whether they were published on ESGF or not. 2. Tuning simulations: Runs performed during the calibration and development of the model, not meant for direct publication. 3. Discarded simulations: Runs that were interrupted, failed, or ultimately not used due to bugs. These simulations were thought to be in production but ended up not being used because of unexpected issues after running them. By distinguishing these categories, we aim to capture the full range of computational effort invested in CMIP7, as well as to quantify their individual impact. Collection, Timeline, and Dissemination of the metrics This document introduces all metrics and specifies how they are to be collected, serving as a guideline to assist centres in collecting and reporting their data consistently. Links to an online spreadsheet will be distributed to all modelling institutes (one per institute). The spreadsheet will request the values of the metrics defined in this document. The information collected through the individual modelling institute spreadsheet will be compiled into a secure document maintained by BSC (Barcelona Supercomputing Centre) and – depending on the agreed data-sharing 4 policy – made available to participating modelling institutes. This shared dataset will serve as the foundation for subsequent analysis, intercomparison and reporting activities. A dedicated FAQ page will accompany the spreadsheet to address common questions and provide clarifications. The FAQ will be maintained by the Energy Consumption and Carbon Footprint Task Team (TT) updated as new questions arise, ensuring that responses are available across all institutes. Modelling institutes are encouraged to reach out at any time for assistance or clarification, please contact: [email protected] Timeline The initial collection phase will cover CMIP7 Assessment Fast Track (AFT) simulations. Confirmation by the modelling centres that they can access the spreadsheet and understand the information required by January 2026. Afirst round of data collection is anticipated by September 2026, aiming to support downstream activities for the IPCC AR7 process, which targets historical and scenario (MIP) data availability by Summer 2026. A second phase will extend to the broader CMIP7 MIPs, with detailed timelines to be defined once those experiments are scheduled. Due to the nature of the collection, we expect the involvement of modelling institutes to be a continuous process rather than a one-time effort, extending throughout the full CMIP7 life cycle. Dissemination The results derived from this activity will be disseminated through a peer-reviewed publication, acknowledging participating modelling groups, and public visualisations and summaries of key statistics, similar to ESGF Data Statistics (to be decided). The metrics General information 1. Modelling group (mandatory) 1.1. Name. 1.2. Location (city and country). 5 2. HPC Platform Information (mandatory) 2.1. Platform name. 2.2. Power Usage Effectiveness (PUE): A metric of a supercomputer’s energy efficiency. 3. Emission Factor (mandatory) 3.1. For the energy used by the computing centre, the carbon dioxide emissions produced per unit of a specific energy source (unit: kg CO2/MWh). Recommendation: Use grid averages from the most recent calendar year. 4. Energy Mix (encouraged) HPC site-specific energy mix, expressed as the percentage share of primary energy sources (e.g., coal, natural gas, nuclear, hydro, wind, solar). Provide at least one of the following: 4.1. A reference or official source for the HPC site’s energy mix data. 4.2. The actual energy mix values (raw percentages for each energy source). Recommendation: Use averaged values over the most recent full calendar year. If sitespecific data are not available, please indicate the geographic location of the HPC infrastructure (city, region or country). Tier 1 (mandatory) 1. Experiment information: 1.1. Experiment type: Indicate whether the simulation corresponds to a production, tuning, or discarded run. 1.2. For production experiments: Specify the community-accepted experiment name as defined in the CMIP7 experiment table ‘Experiments’ tab. 1.3. Model type:coupled, atm only, ocean only, etc. 2. Simulated Years: The number of simulated years for the experiment (unit: years) 3. Core-Hours: number of CPU core-hours consumed by the experiment, i.e. the number of cores used to run the model multiplied by the elapsed time of the run (unit: core-hours) 4. Data Output: Volume of scientific data produced by the experiment saved on the HPC disk, even if only temporarily and not permanently stored afterwards (unit: gigabytes) 6 5. Energy Consumption: Energy consumed by the experiment job (unit: megajoules). 5.1. Preferred measurement: Hardware counters (e.g., Intel RAPL or IPMI via Baseboard Management Controller (BMC) for CPUs, NVIDIA’s NVML library or ROCm-SMI for GPUs) 5.2. Requirements 5.2.1.Specify the data source and methodology used. This includes whether the value comes from direct node measurement (i.e., power meter), hardware counters, scheduler-level accounting, or estimation. 5.2.2. Report dynamic energy, i.e., energy consumed solely by the application workload. Idle (static) energy must be excluded. 5.2.3. Report total energy, i.e., energy consumed by the application plus idle energy. Prioritising hardware counters ensures available, high-resolution, workload-specific measurements that allow for fairer comparisons across different HPC sites. Where counters are not available, other methods may be used, provided the methodology is clearly documented. Tier 2 (encouraged) 1. SYPD (Simulated Years Per Day): Number of model years simulated per 24 hours of wall-clock run time (unit: years/day). 2. QSYPD (Queue SYPD): Simulated years per day including both queue time and run time (unit: years/day). 3. RSYPD (Real SYPD): Simulated years per day including queue time, run time, system interruptions, and workflow errors (unit: years/day). 4. Parallelisation: Number of CPU cores used for the simulation (unit: cores). Include the number of GPUs when applicable (e.g., 1000 cores + 20 GPUs). 5. Data Output Intensity: Data volume produced per compute hour (unit: GB/core-hour), including all output generated during the simulation e.g. scientifically valuable data, restarts, logs, temporary files, etc. Easily obtained by checking the runtime directory size before and after the run, divided by the number of core-hours consumed: 𝐷𝑎𝑡𝑎 𝐼𝑛𝑡𝑒𝑛𝑠𝑖𝑡𝑦 = 𝑂𝑢𝑡𝑝𝑢𝑡 𝑉𝑜𝑙𝑢𝑚𝑒 [𝐺𝐵] / 𝐶𝑜𝑟𝑒𝐻𝑜𝑢𝑟𝑠 7 6. Number of Grid Points: The total number of 3D grid points used in the model integrations (unit: digit). Tier 3 (optional) 1. Complexity: Number of prognostic variables (unit: digit) 1.1. For multi-component models, complexity is the sum of the prognostic variables across all components 1.2. If the value is not directly known for a component or model, it can be approximated using the restart file size and Resolution: 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 = 𝑆𝑐/𝐺𝑐/8 Where Sc = restart file size (bytes), Gc = number of grid points, and 8 = bytes per variable (assuming double precision) 2. Data Output Cost: Performance overhead due to the I/O. The measurement approach will depend on whether synchronous I/O or asynchronous I/O is used (unit: ratio). 2.1. Synchronous I/O: the application waits for each I/O operation to complete before proceeding 2.1.1. If direct I/O timers are available, calculate the ratio of I/O core-hours to the total corehours consumed by the experiment. If not available; 2.1.1.1. Re-run the same experiment with the I/O deactivated to obtain the CHSY without I/O (CHSY_without_io) 2.1.1.2. Compute Data Output Cost as: 2.1.1.3. 𝐷𝑎𝑡𝑎 𝑂𝑢𝑡𝑝𝑢𝑡 𝐶𝑜𝑠𝑡 = 1 − 𝐶𝐻𝑆𝑌_𝑤𝑖𝑡ℎ𝑜𝑢𝑡_𝑖𝑜 𝐶𝐻𝑆𝑌 2.2. Asynchronous I/O: the application can continue computing while I/O operations are performed in the background 2.2.1.Expressed as the ratio between experiment parallelisation (P, including I/O servers) and the number of dedicated I/O processes (Pio). The Data Output Cost will be computed as follows: 𝐷𝑎𝑡𝑎 𝑂𝑢𝑡𝑝𝑢𝑡 𝐶𝑜𝑠𝑡 = 𝑃𝑖𝑜 𝑃 2.3. Requirement: Always specify whether the model uses synchronous or asynchronous I/O. 8 3. Coupling Cost: Ratio of the computational overhead due to coupling and load imbalance (unit: ratio) 3.1. Concurrent coupling: If the different components run on separate cores in parallel (i.e. concurrent coupling, external coupler, Multiple Program Multiple Data), this is computed as the ratio between the core-hours spent by the coupled simulation (core_hours_coupled) over the sum of the individual components' standalone core-hours (core_hours_standalone) 𝐶𝑜𝑢𝑝𝑙𝑖𝑛𝑔 𝑐𝑜𝑠𝑡 = 1 − ∑𝐶 𝑐𝑜𝑟𝑒_ℎ𝑜𝑢𝑟𝑠_𝑠𝑡𝑎𝑛𝑑𝑎𝑙𝑜𝑛𝑒𝐶 𝑐𝑜𝑟𝑒_ℎ𝑜𝑢𝑟𝑠_𝑐𝑜𝑢𝑝𝑙𝑒𝑑 3.2. Sequential coupling: If the different components run on the same cores one after the other (i.e. sequential coupling, integrated coupling framework, Single Program Multiple Data), this is approximated as the extra parallelisation required to maintain the same throughput as in a concurrent setup (i.e., the sum of each component’s parallelisation so that all achieve the same throughput). Note: this requires knowing the scalability curve of the standalone components. 4. Memory Bloat: Ratio of actual memory size to the ideal memory use (unit: ratio) 4.1. Measure the runtime memory usage (Mr), often called ‘Resident Set Size’ (RSS) 4.2. Estimate memory devoted to instructions (Mi) as executable file size x number of processes 4.3. Approximate ideal memory size (Mo) using the combined size of all restart files (sum across components) 4.4. The Memory Bloat is then computed as follows: 𝑀𝑒𝑚𝑜𝑟𝑦 𝐵𝑙𝑜𝑎𝑡 = (𝑀𝑟 − 𝑀𝑖) 𝑀𝑜 9 Annex Computing the Number of grid points How to calculate the number of grid points depending on the grid type: 1.1. Regular latitude-longitude / regular (full) Gaussian grid: Nhoriz = 8N Ntotal = Nhoriz x Nz Where Nis the number of latitude lines between the pole and the equator Source: https://confluence.ecmwf.int/display/OIFS/1+Gaussian+grids 1.2. Reduced / Octahedral Gaussian: Nhoriz = 4N(N+9); Ntotal = Nhoriz x Nz Where Nis the number of latitude lines between the pole and the equator Source: https://confluence.ecmwf.int/display/OIFS/1+Gaussian+grids 1.3. Icosahedral grids Nhoriz = 20 x n² x 4^m Where nis the root refinement and mis the number of bisections Source: https://doi.org/10.1029/2017MS001242 Energy profiling Energy consumption can be divided into two components: 1. Static (Idle) Energy: Power drawn by a system when it is powered on but not performing any meaningful computation. 2. Dynamic (Workload-Dependent) Energy: Additional energy consumed as a direct result of executing the application workload. For CMIP7, we are specifically requesting dynamic energy measurements, which reflect the computational cost of the climate simulation itself. To determine this, it is necessary to first quantify the static energy, either from known idle values or by running a dedicated idle job on the nodes used and then subtracting it from the total measured energy.