D1.5 - Portable test suite for shared software stack

Portable test suite for shared software stack MultiXscale Deliverable 1.5 Deliverable Type: Report Delivered in June, 2025 MultiXscale EuroHPC Centre of Excellence for Multiscale Modelling Acknowledgement Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093169. Disclaimer Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European High Performance Computing Joint Undertaking (JU). Neither the European Union nor the granting authority can be held responsible for them. MultiXscale Deliverable 1.5 Page ii Project and Deliverable Information Project Title MultiXscale: EuroHPC Centre of Excellence for Multiscale Modelling Project Ref. Grant Agreement 101093169 Project Website https://www.multixscale.eu EuroHPC Project Officer Dr. Matteo Mascagni Deliverable ID D1.5 Deliverable Nature Report Dissemination Level Public Contractual Date of Delivery Project Month 30 (30th June, 2025) Actual Date of Delivery 27th June, 2025 Description of Deliverable Final report on the portable test suite for the shared software stack Document Control Information Document Title: Portable test suite for shared software stack ID: D1.5 Version: As of June, 2025 Status: Accepted by Steering Committee Available at: https://www.multixscale.eu/deliverables Document history: Internal Project Management Link Review Review Status: Reviewed Authorship Written by: Caspar van Leeuwen (SURF) Contributors: Kenneth Hoste (UGent), Lara Peeters (UGent), Satish Kamath (SURF) Reviewed by: Kenneth Hoste (UGent), Satish Kamath (SURF) Approved by: Alan O’Cais (UB) Document Keywords Keywords: MultiXscale, HPC, software, applications , testing 27th June, 2025 Disclaimer: This deliverable has been prepared by the responsible Work Package of the Project in accordance with the Consortium Agreement and the Grant Agreement. It solely reflects the opinion of the parties to such agreements on a collective basis in the context of the Project and to the extent foreseen in such agreements. Copyright notices: This deliverable was co-ordinated by Caspar van Leeuwen1(SURF) on behalf of the MultiXscale consortium with contributions from Kenneth Hoste (UGent), Lara Peeters (UGent), Satish Kamath (SURF) . This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0 cb 1caspar[email protected] MultiXscale Deliverable 1.5 Page iii Contents Executive Summary 1 1 Introduction 2 1.1 Scope of the deliverable ................................................ 2 1.2 Target audience of the deliverable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Deliverable outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Partner contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Structure of the test suite 3 2.1 ReFrame test classes .................................................. 3 2.2 EESSI test suite logic .................................................. 3 2.3 Reframe configuration files .............................................. 3 2.4 Facilitating periodic runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 Portability 4 3.1 Overcoming the portability challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Configuring the EESSI test suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.3 EESSI Mixin class .................................................... 5 4 Other notable features 7 4.1 Declaring test memory requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 Memory usage reporting ................................................ 7 4.3 Process binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.4 Improved logging .................................................... 7 4.5 Flexible initialization .................................................. 8 4.6 More efficiently handling of staged files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.7 Improved support for hyperthreading systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5 Supported applications 9 6 Community building and sustainability 10 7 Conclusion and outlook 11 A The EESSI test suite GROMACS test 12 B The tutorial test: mpi4py 13 References 17 MultiXscale Deliverable 1.5 Page 1 Executive Summary The European Environment for Scientific Software Installations (EESSI) test suite consists of three main parts: EESSI test suite logic, the test classes themselves, and some auxilary files (configuration, scripts easily enabling periodic runs of the test suite, etc). One of the main focus points in developing the EESSI test suite was to ensure portability of tests. ReFrame tests traditionally contain a large amount of system-specific information. The paradigm followed by the EESSI test suite is that all system specific information is limited to the ReFrame configuration files that describe the system, so that the test classes themselves only contain generalizable information. As an example: instead of specifying a hardcoded number of tasks to run a test with (the traditional ReFrame approach), tests in the EESSI mixin class may declare that they ’run on a full node, with one task per core’. The EESSI Mixin class then reads the configuration for the system it is running on, and translates this to a concrete task count. While the implementation of the EESSI Mixin class was one of the key changes since the publication of D1.2, several other features were added as well that facilitate test development (e.g. memory usage reporting), improve test performance (e.g. process binding, improved support for hyperthreading systems) or improve usability (lower chance of out-of-memory errors since tests declare their memory requirement, improved logging, being able to run the test suite on a local module stack, more efficient staging). A total of 10 applications are now supported in the test suite. To get better coverage, community contributions are essential. Thus, substantial attention has been paid to aspects that may increase community uptake. This includes trying to make test development simpler, simplify running the test suite on a local software stack, and outreach in the form of a hands-on session. In summary, the EESSI test suite provides a portable test suite with key applications that are important to MultiXscale. It has solved the portability challenge through the implementation of the EESSI Mixin class, and portability was proven by running across a wide range of systems. MultiXscale Deliverable 1.5 Page 2 1 Introduction 1.1 Scope of the deliverable This deliverable describes the development of the EESSI test suite: a portable test suite created to test the software in the shared software stack. It also discusses how the test suite is employed in practice, and our efforts to attract community contributions. The work presented here was executed in the context of Task 1.3 - "Design and creation of a software test suite and facilitating Continuous Integration (CI) for software developers". 1.2 Target audience of the deliverable This deliverable is targeted at readers with experience in High Performance Computing (HPC) system support, particularly in the context of maintaining and testing software stacks (or specific applications) for HPC infrastructure. 1.3 Deliverable outline Section 2describes the (code) structure of the EESSI test suite. Section 3describes how we achieve portability, and discuss two important components that provide portability (the ReFrame configuration file, and the EESSI Mixin class). Section 4discusses other key features of the EESSI test suite. In section 5we list the applications that are currently support in the EESSI test suite. Section 6discusses what has been done to stimulate uptake of the EESSI test suite by the community in order to foster sustainability. 1.4 Partner contributions SURF and UGent contributed as planned to the work in this deliverable. Note that Task 1.3 had other partners (RIJKSUNIGRON, Bsc), but those were never envisioned to contribute to the EESSI test suite itself - they developed other components within Task 1.3. MultiXscale Deliverable 1.5 Page 3 2 Structure of the test suite In this section, we discuss the structure of the test suite. Understanding the structure of the test suite is helpful in understanding how the test suite works. The key directory and file structure of the test suite repository [1] is as follows: 1CI/< site >_<system_name>/ci_config . sh 2CI/run_reframe . sh 3CI/run_reframe_wrapper . sh 4config/< site >_<system_name>.py 5eessi / test s u i t e / t e sts / 6ees si / t es ts uite /common_config . py 7eessi / t es ts ui te / constants . py 8eessi / t es ts ui te / eessi_mixin . py 9eessi / t es ts ui te /hooks . py 10 ees si / t es t su ite / u t i l s . py 2.1 ReFrame test classes The core part of the test suite are the test definitions and resources under eessi/testsuite/tests/. Each test is essentially a class definition that inherits from a ReFrame test class. Some tests come with resources, e.g. input files, which are also stored in this subdirectory. Typically, tests for a single application are defined in a single python file, but multiple test classes may be defined for a single application. For example, the file eessi/testsuite/tests/apps/osu.py includes tests for the OSU Microbenchmarks both for point-to-point and collective communication, each of which is defined in a separate test class. 2.2 EESSI test suite logic The EESSI test suite is not a standard ReFrame test suite: it implements additional logic and configuration in order to achieve portability. This is achieved through eessi/testsuite/common_config.py,eessi/testsuite/constants.py,eessi/testsuite/ eessi_mixin.py,eessi/testsuite/hooks.py and eessi/testsuite/utils .py. 2.3 Reframe configuration files The config directory contains a number of configuration files for systems on which the EESSI test suite has been deployed in the context of the MultiXscale project. These serve both as examples for other users, and at the same time allows us to have version-controlled configuration files. 2.4 Facilitating periodic runs Everything under the CI directory is intended to make it easy to set up periodic (e.g. daily or weekly) runs on a system. The CI/<site>_<system_name>/ci_config.sh script allows one to set a configuration for these periodic runs. For example, these configuration items can be used to determine which version of the test suite to run, which version of ReFrame to use, which arguments to pass to ReFrame (e.g. for running a subset of the tests), etc. The CI/run_reframe.sh and run_reframe_wrapper.sh scripts are the main drivers of these periodic runs. Based on the configuration, they create installations of ReFrame, the test suite, and running everything with the desired arguments. It also sets defaults for all configuration items not explicitly defined in ci_config.sh. With this, setting up a periodic run as e.g. a cronjob can be as simple as: 1. Cloning the CI subdirectory of the repository; 2. Creating a ci_config.sh config file for the system (if none is present yet in the test-suite repository); 3. Adding a line like 0 0 ***EESSI_CI_SYSTEM_NAME=surf_snellius \$HOME/test−suite/CI/run_reframe_wrapper.sh to your crontab file. MultiXscale Deliverable 1.5 Page 4 3 Portability 3.1 Overcoming the portability challenge In order to create a portable test suite, it is of paramount importance that system-specific information and test-specific information is clearly separated. In the EESSI test suite, that is achieved by making sure that all system-specific information is provided through the ReFrame configuration file, while all the test-specific information is contained within the (ReFrame) test classes. This means that tests should be developed in such a way that they can take the information provided in the ReFrame configuration file, and do something sensible based on that. For example, if a test requires 200 GB of memory to run, but the ReFrame configuration file states that a given partition only provides 100 GB, the test should be skipped. Another example would be a program that uses pure MPI parallelism, where one typically wants to launch one task per available physical core. The test should then read the number of available cores from the ReFrame configuration file, and determine the number of tasks to be launched for the test accordingly. Note that this approach to achieving portability is sufficient because this is a test suite, not a benchmark suite. A test suite is meant to allow identification of (performance) regressions, whereas a benchmark is meant to show the absolute-best performance one can achieve for a given application on a given system. The latter typically requires manual tuning. For example, a task binding strategy that is optimal on system A is not necessarily optimal on system B. For the test suite, we just set abinding strategy that is expected to give reasonable performance on any system - but more importantly that is expected to result in reproducible performance: the more stable the performance, the smaller the performance regressions that one can identify. 3.2 Configuring the EESSI test suite While the EESSI test suite uses standard ReFrame configuration files, it does impose additional constraints. ReFrame allows certain fields to be defined as key-value pairs, i.e. free text. The EESSI test suite standardizes these key value pairs through constants defined in eessi/testsuite/constants.py. This allows us to assign meaning to those free text labels. Typically, the following sections that are specific to the EESSI test suite should be set in the ReFrame configuration file: 1’systems ’ : [ 2. . . 3’ partitions ’ : [ 4{ 5’prepare_cmds ’ : [ common_eessi_init ( ) ] , 6’ resources ’ : [ 7{ 8’name’ : ’memory’ , 9’ options ’ : [ ’− −mem={ si ze } ’ ] 10 } 11 ’ features ’ : [ 12 FEATURES.CPU, 13 ] + l i s t (SCALES. keys ( ) ) , 14 ’ extras ’ : { 15 EXTRAS.MEM_PER_NODE: 229376 # in MiB 16 } , 17 . . . 18 ] 19 } , 20 ] 21 ] , 22 ’ logging ’ : common_logging_config ( reframe_prefix ) , 23 ’ general ’ : [ 24 { 25 ’ remote_detect ’ : True , 26 **common_general_config( reframe_prefix ) 27 } 28 ] , • The common_eessi_init() is a common set of initialization commands that checks what EESSI environment is initialized on the node running the reframe command (typically a login node), and makes sure the same EESSI environment is initialized on the batch nodes. • The memory resource defines the flag that should be passed to the given resource schedule to ask for a certain amount of memory per node. This can then be used by test classes to try and request sufficient memory to run a test. MultiXscale Deliverable 1.5 Page 5 • The features list includes e.g. if a partition should run CPU tests (FEATURES.CPU), GPU tests (FEATURES.GPU) or both, and at what scales. A set of default test scales is defined that ranges from a single core to 16 full nodes, but on small systems one can pass a subset of this list to limit the maximum size of tests being run. • The EXTRAS.MEM_PER_NODE item defines the maximum amount of memory per node that a job can ask for on that partition. The test suite uses this information to skip tests that require more memory than that. • The remote_detect: True item tells ReFrame to automatically detect the CPU topology on the partition. ReFrame stores this information in a separate file, which then describes things like the number of cores per node, number of cores per socket, number of of numa domains, etc. All of the common_*functions are defined in eessi/testsuite/common_config.py and make sure that the general behavior of the system on different systems is similar, thus providing predictability in e.g. what items are logged, where, etc. With these few key items in the ReFrame configuration file, we can achieve test portability: we can ensure that tests only run if the required resources (GPUs, sufficient memory) are present, and we can ensure that a sensible number of tasks / thread for that system are launched (by using the detected CPU topology). Note that additional features and extras may be added in future test suite releases, as needed. E.g. for now, the shared software stack only has support for NVIDIA GPUs, and thus a general FEATURES.GPU is sufficiently specific. In the future, more specific features may be defined to provide more fine-grained control, e.g. ‘FEATURES.NVIDIA_GPU‘, ‘FEATURES. AMD_GPU‘, etc. 3.3 EESSI Mixin class In Deliverable D 1.2, a proof of concept for a portable test for GROningen MAChine for Chemical Simulation (GROMACS) was presented. Since then, tests for several other applications where added (Section 5). As more tests were added, it became clear there was a fair amount of duplication of logic between these tests. Thus, the code was refactored and the common logic between these tests classes that is needed to make the tests portable was separated into a so-called Mixin class. The result is that the test classes became more lightweight, and focussed on configuration rather than logic. E.g. a test can declare through a simple variable assignment that it requires a GPU. The logic in the EESSI Mixin class then makes sure this test is filtered out on partitions that do not offer any GPUs. For example, the implementation of the GROMACS test class in Appendix Ais now inheriting from the EESSI mixin class, and is substantially more compact that it was before (see Deliverable 1.2, Appendix A). An additional advantage of the EESSI Mixin class is that it makes writing the portable tests easier, for two reasons: 1. Writing a test class is now (mostly) done by just defining the required class properties (i.e. configuration). 2. The EESSI Mixin class has internal checks to make sure the inheriting class defines the correct properties at the correct stage of the ReFrame pipeline, and that the values assigned to those properties are valid. It gives verbose feedback if a certain keyword hasn’t been assigned in a timely fashion or has an invalid value. Thus, even without reading the documentation, a test developer is guided through the development process. For example, the EESSI Mixin class defines the following ReFrame pipeline hook that is run after the ’init’ stage of the ReFrame pipeline: 1@run_after ( ’ init ’ ) 2def EESSI_mixin_validate_init( self ) : 3"""Check that a l l v ariabl es that have to be set for subsequent hooks in the i n i t phase have been se t """ 4# L ist which variables we wi l l need/use in the run_after ( ’ init ’ ) hooks 5va r_ l i s t = [ ’ device_type ’ , ’ scale ’ , ’module_name’ , ’measure_memory_usage ’ ] 6for var in va r _ l i s t : 7i f not hasattr ( self , var ) : 8msg = "The variable ’%s ’ should be defined in any test class that inherits " % var 9msg += " from EESSI_Mixin before ( or in ) the i n i t phase , but i t wasn’ t " 10 raise ReframeFatalError (msg) 11 12 # Check that the value for these variables i s valid , 13 # i . e . exis t s in their respective dict from eessi . testsuite . constants 14 s e l f . EESSI_mixin_validate_item_in_list ( ’ device_type ’ , DEVICE_TYPES [ : ] ) 15 s e l f . EESSI_mixin_validate_item_in_list ( ’ scale ’ , SCALES . keys ( ) ) 16 s e l f . EESSI_mixin_validate_item_in_list ( ’ valid_systems ’ , [ [ ’ *’ ] ] ) 17 s e l f . EESSI_mixin_validate_item_in_list ( ’ valid_prog_environs ’ , [ [ ’ default ’ ] ] ) Essentially, this hook checks that for any test inheriting from EESSI mixin, the class attributes device_type,scale,valid_systems and valid_prog_environs have been defined, and their value is valid. If a test developer fails to assign one of these keywords, or assigns an invalid value, the EESSI Mixin class will raise a clear error. Subsequently, the EESSI Mixin class calls the following ReFrame pipeline hook: MultiXscale Deliverable 1.5 Page 6 1@run_after ( ’ init ’ ) 2def EESSI_mixin_run_after_init ( s e l f ) : 3"""Hooks to run a ft er i n i t phase """ 4 5# F i l t e r on which sca les are supported by the p art it ion s defined in the ReFrame configuration 6hooks . fi lt er _s up ported _s ca le s ( s e l f ) 7 8hooks . filter_valid_systems_by_device_type ( s el f , required_device_type= s e l f . device_type ) 9 10 hooks . set_modules ( s e l f ) 11 12 # Set scales as tags 13 hooks . set_tag_ sca le ( s e l f ) Each of the functions from the hooks namespace can now safely assume that the four aforementioned class attributes are set. To show how this simplifies the test development, consider the mpi4py tutorial test that is in the EESSI documentation (the final test from this tutorial is included in Appendix B). The only thing this class now needs to define for the init pipeline hook is: 1device_type = DEVICE_TYPES.CPU It does not need to call any of the hooks functions itself, nor does it need to define the other three properties (scale, valid_systems and valid_prog_environs): the EESSI Mixin class provides defaults - overwriting these is optional. Overall, the tutorial test is a clear example of how the EESSI Mixin class simplifies the test: the original test (before inheriting from the EESSI Mixin class) was 43 lines of code initially, and is reduced to only 21 lines of code (see Appendix B) after using the EESSI Mixin class. MultiXscale Deliverable 1.5 Page 13 B The tutorial test: mpi4py Here, we provide the Python code for the EESSI mpi4py test, which serves as a tutorial in the documentation. Below, we show two versions: first, the version without using the EESSI Mixin class, then, the version that inherits from the EESSI Mixin class. This code is included here to illustrate the relative simplicity of a test using the EESSI Mixin class. 1import reframe as rfm 2import reframe . u t i l i t y . sanity as sn 3 4# added only to make the l i nt e r happy 5from reframe . core . builtins import variable , parameter , run_after , performance_function , sanity_function 6 7from eessi . t e s tsuite import hooks 8from eessi . te s t s u i t e . constants import SCALES, COMPUTE_UNITS 9from ees si . t estsui t e . u t i l s import find_modules 10 11 12 # This python decorator indicates to ReFrame that this class defines a test 13 # Our class inherits from rfm . RunOnlyRegressionTest , since this test does not have a compilation stage 14 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RunOnlyRegressionTest 15 @rfm. simple_test 16 class EESSI_MPI4PY( rfm . RunOnlyRegressionTest ) : 17 # Programming environments are only relevant for te sts that compile something 18 # Since we are testing existing modules , we typically don’ t compile anything and simply define 19 # ’ default ’ as the valid programming environment 20 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . valid_prog_environs 21 valid_prog_environs = [ ’ default ’ ] 22 23 # Typically , we l i s t here the name of our cluster as i t is specified in our ReFrame configuration f i l e 24 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . valid_systems 25 valid_systems = [ ’ *’ ] 26 27 # ReFrame w i l l generate a test for each module 28 # NOTE: each parameter adds a new dimension to the parametrization space . 29 # (EG 4 parameters with (3 ,3 ,2 ,2) possible values wi l l result in 36 tests ) . 30 # Be mindful of how many parameters you add to avoid the number of te s ts generated being excessive . 31 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . parameter 32 module_name = parameter ( find_modules ( ’ mpi4py ’ ) ) 33 34 # ReFrame w i l l generate a test for each scale 35 scale = parameter (SCALES . keys ( ) ) 36 37 # Our script has two arguments , −−n_iter and −−n_warmup. By defining these as ReFrame variables , we can 38 # enable the end−user to overwrite their value on the command line when invoking ReFrame. 39 # Note that we don’ t typically expose ALL variables , especially i f a script has many − we expose 40 # only those that we think an end−user might want to overwrite 41 # Number of iter ation s to run (more iterations takes longer , but res ults in more accurate timing ) 42 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . variable 43 n_iterations = variable ( int , value=1000) 44 45 # Similar for the number of warmup iteratio ns 46 n_warmup = variable ( int , value=100) 47 48 # Define which executable to run 49 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . executable 50 executable = ’python3 ’ 51 52 # Define which options to pass to the executable 53 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . executable_opts 54 executable_opts = [ ’ mpi4py_reduce . py ’ , ’−− n_iter ’ , f ’ { n_iterations } ’ , ’−−n_warmup’ , f ’ {n_warmup} ’] 55 56 # Temporarily define postrun_cmds to make i t easy to find out memory usage 57 postrun_cmds = [ 58 # for cgroups v1 59 ’MAX_MEM_IN_BYTES=$( </ sys / fs /cgroup/memory/$( </proc/ s el f /cpuset ) / . . /memory. max_usage_in_bytes ) ’ , 60 # for cgroups v2 61 # ’MAX_MEM_IN_BYTES=$(</ sys / f s /cgroup/$( </proc/ se lf /cpuset ) / . . / . . / . . / memory. peak ) ’ , 62 ’echo "MAX_MEM_IN_BYTES=$MAX_MEM_IN_BYTES" ’ , 63 ’echo "MAX_MEM_IN_MIB=$ ( ($MAX_MEM_IN_BYTES/1048576) ) " ’ 64 ] MultiXscale Deliverable 1.5 Page 14 65 66 # Define a time limit for the scheduler running this test 67 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . time_limit 68 time_limit = ’5m00s’ 69 70 @run_after ( ’ init ’ ) 71 def set_modules ( s e l f ) : 72 hooks . set_modules ( s e l f ) 73 74 # Using t hi s decorator , we t e l l ReFrame to run t hi s AFTER the i n i t step of the t es t 75 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . run_after 76 # See https : // reframe−hpc . readthedocs . io /en/ stable / pipeline . html for a l l steps in the pipeline 77 # that reframe uses to execute tests . 78 @run_after ( ’ init ’ ) 79 def ru n_a fter _i nit ( s e l f ) : 80 hooks . set_tag_ sca le ( s e l f ) 81 82 @run_after ( ’ setup ’ ) 83 def set_num_tasks_per_node( self ) : 84 """ Setting number of tasks per node and cpus per task in this function . This function sets 85 num_tasks , num_tasks_per_node , num_cpus_per_task , and num_gpus_per_node , based on the current scale 86 and the current partition ’ s num_cpus, max_avail_gpus_per_node and num_nodes""" 87 hooks . assign_tasks_per_compute_unit ( self , COMPUTE_UNITS.CPU) 88 89 # This t est scales almost i ndefinit ely 90 # For tests that have limited scaling , make sure that te s t instances exceeding 91 # a predefined maximum task count are skipped using : 92 # max_tasks = 300 93 # s e l f . s k ip _i f ( s e l f . num_tasks > max_tasks , 94 # f ’ Skipping t e st : more than { max_tasks } tasks are requested ( { s e l f . num_tasks } ) ’ ) 95 96 # Make sure we request s u fficient memory from the scheduler 97 @run_after ( ’ setup ’ ) 98 def request_mem ( s e l f ) : 99 mem_required = s e l f . num_tasks_per_node *256 # request 256 MB per task per node 100 hooks . req_memory_per_node( self , app_mem_req=mem_required) 101 102 # Set binding strategy 103 @run_after ( ’ setup ’ ) 104 def set_binding ( s e l f ) : 105 hooks . set_compact_process_binding ( s e l f ) 106 107 # Now, we check i f the pattern ’Sum of a l l ranks : X’ with X the correct sum for the amount of ranks i s found 108 # in the standard output : 109 # https :// reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . b uilt ins . sanity_function 110 @sanity_function 111 def v al id at e ( s e l f ) : 112 # Sum of 0 , . . . , N−1 is (N *(N−1) / 2) 113 sum_of_ranks = round ( s e l f . num_tasks *( ( s e l f . num_tasks − 1) / 2) ) 114 # https :// reframe−hpc . readthedocs . io /en/ stable / deferrable_functions_reference . html#reframe . u t i l i t y . sanity . assert_found 115 return sn . assert_found ( r ’Sum of a l l ranks : %s ’ % sum_of_ranks , s e l f . stdout ) 116 117 # Now, we define a pattern to extract a number that r e fl ec t s the performance of this t est 118 # https :// reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . b uilt ins . performance_function 119 @performance_function ( ’ s ’ ) 120 def time ( s e l f ) : 121 # https :// reframe−hpc . readthedocs . io /en/ stable / deferrable_functions_reference . html#reframe . u t i l i t y . sanity . extractsingle 122 return sn . e xt rac ts ing le ( r ’^Time elapsed : \ s +(?P<perf >\S+) ’ , s e l f . stdout , ’ perf ’ , f l o at ) 123 124 @performance_function ( ’MiB’ ) 125 def max_mem_in_mib( s e l f ) : 126 return sn . e xt rac ts ing le ( r ’^MAX_MEM_IN_MIB=(?P<perf >\S+) ’ , s e l f . stdout , ’ perf ’ , i nt ) 1import reframe as rfm 2import reframe . u t i l i t y . sanity as sn 3 4# added only to make the l i nt e r happy 5from reframe . core . builtins import variable , parameter , performance_function , sanity_function 6 7# Import the EESSI_Mixin class so that we can inherit from i t 8from eessi . te s t s u i t e . eessi_mixin import EESSI_Mixin 9from eessi . te s t s u i t e . constants import COMPUTE_UNITS, DEVICE_TYPES 10 from ees si . t estsui t e . u t i l s import find_modules MultiXscale Deliverable 1.5 Page 15 11 12 13 # This python decorator indicates to ReFrame that this class defines a test 14 # Our class inherits from rfm . RunOnlyRegressionTest , since this test does not have a compilation stage 15 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RunOnlyRegressionTest 16 @rfm. simple_test 17 class EESSI_MPI4PY( rfm . RunOnlyRegressionTest , EESSI_Mixin ) : 18 19 # The device type makes sure this test only gets executed on systems/ partitions that can provide this device 20 device_type = DEVICE_TYPES.CPU 21 22 # One task is launched per compute unit . In this case , one task per ( physical ) CPU core 23 compute_unit = COMPUTE_UNITS.CPU 24 25 # ReFrame w i l l generate a test for each module that matches the regex ‘mpi4py‘ 26 # This means we impli citly assume that any module matching this name provides the required functionality 27 # to run this test 28 module_name = parameter ( find_modules ( ’ mpi4py ’ ) ) 29 30 # Our script has two arguments , −−n_iter and −−n_warmup. By defining these as ReFrame variables , we can 31 # enable the end−user to overwrite their value on the command line when invoking ReFrame. 32 # Note that we don’ t typically expose ALL variables , especially i f a script has many − we expose 33 # only those that we think an end−user might want to overwrite 34 # Number of iter ation s to run (more iterations takes longer , but res ults in more accurate timing ) 35 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . variable 36 n_iterations = variable ( int , value=1000) 37 38 # Similar for the number of warmup iteratio ns 39 n_warmup = variable ( int , value=100) 40 41 # Define which executable to run 42 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . executable 43 executable = ’python3 ’ 44 45 # Define which options to pass to the executable 46 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . executable_opts 47 executable_opts = [ ’ mpi4py_reduce . py ’ , ’−− n_iter ’ , f ’ { n_iterations } ’ , ’−−n_warmup’ , f ’ {n_warmup} ’] 48 49 # Define a time limit for the scheduler running this test 50 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . pipeline . RegressionTest . time_limit 51 time_limit = ’5m00s’ 52 53 # Define the benchmarks that are available in the test . 54 # In t his t es t ( ‘ EESSI_MPI4PY ‘ ) there is only one benchmark . I f there are more than one , 55 # define them using the ‘ parameter ( ) ‘ function . 56 bench_name = ’mpi4pi ’ 57 58 # Specify the benchmark to be tested in CI ( wil l be marked with a ‘CI ‘ tag ) . 59 bench_name_ci = ’mpi4pi ’ 60 61 # Define the f i l e s and/or dirs inside sourcesdir ( default=src ) that should be symlinked into the stage dir 62 readonly_files = [ ’ mpi4py_reduce . py ’ ] 63 64 # Define the class method that returns the required memory per node 65 def required_mem_per_node( self ) : 66 return s e l f . num_tasks_per_node *100 + 250 67 68 # Now, we check i f the pattern ’Sum of a l l ranks : X’ with X the correct sum for the amount of ranks i s found 69 # in the standard output : 70 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . sanity_function 71 @sanity_function 72 def v al id at e ( s e l f ) : 73 # Sum of 0 , . . . , N−1 is (N *(N−1) / 2) 74 sum_of_ranks = round ( s e l f . num_tasks *( ( s e l f . num_tasks − 1) / 2) ) 75 # https :/ / reframe−hpc . readthedocs . io /en/ stable / deferrable_functions_reference . html#reframe . u t i l i t y . sanity . assert_found 76 return sn . assert_found ( r ’Sum of a l l ranks : %s ’ % sum_of_ranks , s e l f . stdout ) 77 78 # Now, we define a pattern to extract a number that r e fl e ct s the performance of this t est 79 # https :/ / reframe−hpc . readthedocs . io /en/ stable / regression_test_api . html#reframe . core . buil tins . performance_function 80 @performance_function ( ’ s ’ ) MultiXscale Deliverable 1.5 Page 16 81 def time ( s e l f ) : 82 # https :/ / reframe−hpc . readthedocs . io /en/ stable / deferrable_functions_reference . html#reframe . u t i l i t y . sanity . extractsingle 83 return sn . e xt rac ts ing le ( r ’^Time elapsed : \ s +(?P<perf >\S+) ’ , s e l f . stdout , ’ perf ’ , f l oa t ) MultiXscale Deliverable 1.5 Page 17 References Acronyms used CI Continuous Integration CernVM-FS CernVM File System EESSI European Environment for Scientific Software Installations HPC High Performance Computing EuroHPC European High Performance Computing Joint Undertaking Software mentioned GROMACS GROningen MAChine for Chemical Simulation URLs referenced Page ii https://www.multixscale.eu ... https://www.multixscale.eu https://www.multixscale.eu/deliverables .. . https://www.multixscale.eu/deliverables Internal Project Management Link . .. https://github.com/multixscale/planning/issues/105 caspar[email protected] ... mailto:[email protected] http://creativecommons.org/licenses/by/4.0 ... http://creativecommons.org/licenses/by/4.0 Page 6 mpi4py tutorial test that is in the EESSI documentation ... https://www.eessi.io/docs/test-suite/writing-portable-tests/ #step-by-step-tutorial-for-writing-a-portable-reframe-test Citations [1] “EESSI test suite,” https://github.com/EESSI/test-suite. [2] “EESSI test suite: available tests,” https://www.eessi.io/docs/test-suite/available-tests/. [3] “EESSI test suite documentation,” https://www.eessi.io/docs/test-suite/. [4] “10th EasyBuild User Meeting,” https://easybuild.io/eum25/.